How To Extract Thousands Of News Articles In 10 Easy Steps

Web extraction using Python is extremely easy to do when you follow these 10 easy steps.

This blog post includes the first part: News articles data extraction using Python. We’ll make a script, which extracts the newest news articles from various newspapers as well as saves the text that would be fed in the model afterwards to get predictions in its category.

In case, we wish to extract different news articles from the website, the initial step is to understand how any website works.

We would follow an example for understanding this:

Whenever we insert the URL into a web browser (i.e. Firefox, Google Chrome, etc.) as well as access to that, what we observe is the grouping of three different technologies:

Note that all these are programming languages. They would permit us to make and manipulate all the design aspects of any webpage.

Let’s prove these concepts using an example. Whenever we visit a Politifact page, we will see these:

In case, we disable JavaScript, we won’t be able to utilize the pop-up any longer, as we are unable to see the video pop-up window:

In case, we delete the CSS elements from the web-page after getting it with Ctrl+F on the inspect window, we would see somewhat like that:

So, here, we will ask a question.

“If you wish to extract a webpage’s content using web scraping, where do you want to search?”

So, at that point, we hope that you are very clear about what type of source codes we require to extract. Yes, you are totally right, if you have thought about HTML.

Therefore, the last stage before using any web extraction methods is understanding a bit of HTML.

HTML is the language, which defines a webpage content as well as constitute of attributes and elements to extract data, you need to be familiar with examining those elements.

The element might be a paragraph, division, heading, anchor tags, and more.

An attribute might be that a heading is within bold letters.

The tags are characterized with the opening symbol as well as closing symbol

e.g.,

Output:

While, this will return content of the response using bytes.

III. Search for any particular substring of texts within response.

if "Politifact" in page.text:
         print("Yes, Scarpe it")

IV. Observe response’s Content-Types (observe if you get back JSON, XML, HTML, etc.)

print (page.headers.get("content-type", "unknown"))

Output:

Step-5: Delay the Request Time

Following the time modules, we could call sleep(2) function having values of 2 seconds. There, it delayed to send requests to the web-server within 2 seconds.

time.sleep(2)

The sleep() function interrupts execution of present thread for any given seconds.

Step 6: Extract Content using HTML

Now as you’ve done with your HTTP requests and got some new HTML content, you can parse it to extract different values you’re searching for.

A) With Regular Expressions

With Regular Expressions to look for the HTML content is not suggested at all.

Though, regular expressions are very useful for getting particular string patterns including prices, phone numbers, or email addresses.

Run any regular expression about the response texts to search for any particular string patterns:

import re  # put this at the top of the file
...
print(re.findall(r'\$[0-9,.]+', page.text))

Output:

Step-5: Delay the Request Time

Following the time modules, we could call sleep(2) function having values of 2 seconds. There, it delayed to send requests to the web-server within 2 seconds.

time.sleep(2)

The sleep() function interrupts execution of present thread for any given seconds.

Step 6: Extract Content using HTML

Now as you’ve done with your HTTP requests and got some new HTML content, you can parse it to extract different values you’re searching for.

A) With Regular Expressions

With Regular Expressions to look for the HTML content is not suggested at all.

Though, regular expressions are very useful for getting particular string patterns including prices, phone numbers, or email addresses.

Run any regular expression about the response texts to search for any particular string patterns:

import re  # put this at the top of the file
...
print(re.findall(r'\$[0-9,.]+', page.text))

Output:

B) With Object Soup from BeautifulSoup

BeautifulSoup is the Python library used to pull data out from XML and HTML files. This works with the favorite parser for providing idiomatic ways to navigate, search, and modify a parse tree. This generally saves programmers’ days of hours of work.

soup = BeautifulSoup(page.text, "html.parser")

The given-listed command would look for different tags e.g., <li> having particular attribute ‘o-listicle__item’

links=soup.find_all('li',attrs={'class':'o-listicle__item'})

To understand the give code, you have to inspect a webpage & follow along:

1. Go to the listed URL given

2. Press Ctrl+Shift+I for inspecting it.

3. That is how the ‘Inspect window’ would look like:

Press Ctrl+Shift+C to choose an element given in a page for inspecting it or going to leftmost arrow given in a header of an Inspect window.

4) To get the given specific elements & attributes in the inspect window

Initially, try to go in each section of a webpage to see the changes on the inspect window, you would easily get the idea about how webpages work as well as which element are what and what specific attributes are contributing in the webpage.

When completed with the given step, we assume that you could understand the workings of given element<li> as well as its attribute.

As we needed a news section of any particular article, we go to the article section through choosing the inspect element alternative in an inspect window, this will highlight the article section on a web-page as well as its HTML resource on the Inspect Window. Voila!

Did you locate the similar tag on the machine?

However, we don’t require any text rather than date, therefore I utilize indexing. Though, you may clean your attributes later using a few regex combinations. The ‘footer’ is a component, which contained the necessary text.

Date = j.find('div',attrs={'class':'m-statement__body'}).find('footer').text[-14:-1].strip()

Here, we have done the whole thing same as before excluding get() that is scraping content of the passed attribute (i.e., title)

Source = j.find('div', attrs={'class':'m-statement__author'}).find('a').get('title').strip()

On this site, we do find articles already attached with the labels however, the text isn’t retrievable as it is restricted in the image. For this type of particular task, you may utilize get() to save particular text efficiently. Here, we are passing ‘alt’ like an attribute for get() that has the Label text.

Label = j.find('div', attrs ={'class':'m-statement__content'}).find('img',attrs={'class':'c-image__original'}).get('alt').strip()

In the given code lines, we have put different concepts together as well as tried to draw data for five attributes of our Dataset.

for j in links:
        Statement = j.find("div",attrs={'class':'m-statement__quote'}).text.strip()
        Link=st.find('a')['href'].strip()
        Date = j.find('div',attrs={'class':'m-statement__body'}).find('footer').text[-14:-1].strip()
        Source = j.find('div', attrs={'class':'m-statement__author'}).find('a').get('title').strip()
        Label = j.find('div', attrs ={'class':'m-statement__content'}).find('img',attrs={'class':'c-image__original'}).get('alt').strip()
        frame.append([Statement,Link,Date,Source,Label])
upperframe.extend(frame)

Step-8: Make Dataset

Append all attribute values to the empty lists ‘frame’ for every article

frame.append([Statement,Link,Date,Source,Label])

After that, extend the list to the empty listing ‘upperframe’ for every page.

upperframe.extend(frame)

Step-9: Visualize Datasets

If you want to visualize data on Jupiter, you could utilize pandas DataFrame for doing so.

data=pd.DataFrame(upperframe, columns=['Statement','Link','Date','Source','Label'])
data.head()

Step-10: Make CSV files & Save it in Your PC

A) Open & Write to file

The given command will assist you to write the CSV file as well as save that to machine in a same directory where the Python file is saved.

filename="NEWS.csv"
    f=open(filename,"w")
    headers="Statement,Link,Date, Source, Label\n"
    f.write(headers)
    ....
        f.write(Statement.replace(",","^")+","+Link+",
"+Date.replace(",","^")+","+Source.replace(",","^")+","+Label.replace(",","^")+"\n")

The line would write every attribute to the file by replacing all ‘,’ with ‘^’.

f.write(Statement.replace(",","^")+","+Link+","+Date.replace(",","^")+","+Source.replace(",","^")+","+Label.replace(",","^")+"\n")

Therefore, whenever you run the file on a command shell, this will make the CSV file within a .py file directory.

While opening that, you might observe weird data in case, you don’t utilize strip() while extracting. Therefore, check that without applying the strip() as well as in case, you don’t exchange ‘^’ with ‘,’, this will look weird.

Therefore, replace that using these easy steps:

Open the excel file or .csv file

Press Ctrl+H (One pop-up window would open asking find what and replace with)

You can check in the given image, how it extracts vague data for Statement attribute.

Therefore, rather than spending hours to clean the data manually, we would recommend writing some additional lines of code given in the main technique.

Now, you are completed with that!

IMPORTANT: If you have tried to copy and paste our source code to extract different websites as well as run that, it’s quite possible that this will show an error as every webpage’s layout is quite different and to do that, you require to make the changes accordingly.

import urllib.request,sys,time
from bs4 import BeautifulSoup
import requests
import pandas as pd

pagesToGet= 1

upperframe=[]  
for page in range(1,pagesToGet+1):
    print('processing page :', page)
    url = 'https://www.politifact.com/factchecks/list/?page='+str(page)
    print(url)