Therefore, the first thing required is to ensure that you have installed Python 3 and if not, just install Python 3 as well as get that installed before the proceedings.
After that, just install BeautifulSoup using
pip3 install beautifulsoup4
We will need libraries’ requests, soupsieve, and LXML to fetch scrape, break that down into XML, as well as utilize CSS selectors. After that, install them with.
pip3 install requests soupsieve lxml
Once get installed, open the editor as well as type in.
# -*- coding: utf-8 -*- from bs4 import BeautifulSoup import requests
Then, let’s go to a Meetup event listing page as well as examine the data we have.
It will look like this:
Now, coming back to our codes, let’s try and find data through pretending that we are the browser like this.
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.meetup.com/find/events/?allMeetups=true&radius=25&userFreeform=New York, NY&mcId=c10001&mcName=New York, NY'
response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
Then save it as the scrapeMeetup.py.
In case, you run that.
python3 scrapeMeetup.py
You would see the entire HTML page.
Then, let’s utilize CSS selectors for getting the data you want… To perform it, let’s open Chrome and go to inspect tool.
Here, we have noticed that all individual products data are available within the class called ‘event-listing’. We could scrape this using a CSS selector called ‘.event-listing.’ With ease. The code will look like this:
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.meetup.com/find/events/?allMeetups=true&radius=25&userFreeform=New York, NY&mcId=c10001&mcName=New York, NY'
response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
for item in soup.select('.event-listing'):
try:
print('----------------------------------------')
print(item)
except Exception as e:
#raise e
print('')
It will print the content in all the elements, which hold the products’ data.
Now, we can pick the classes within these rows, which have the required data. We notify that a title is within a
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import requests
headers = {'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9'}
url = 'https://www.meetup.com/find/events/?allMeetups=true&radius=25&userFreeform=New York, NY&mcId=c10001&mcName=New York, NY'
response=requests.get(url,headers=headers)
soup=BeautifulSoup(response.content,'lxml')
for item in soup.select('.event-listing'):
try:
print('----------------------------------------')
#print(item)
print(item.select('[itemProp=name]')[0].get_text())
print(item.select('[itemProp=name]')[1].get_text())
print(item.select('.omnCamp')[0].get_text().strip().replace('\n', ' '))
print(item.select('.attendee-count')[0].get_text().strip().replace('\n', ' '))
except Exception as e:
#raise e
print('')
In case you run that, it would print out the information.
And that’s it!! We have got all of them.
In case, you wish to utilize that in production as well as wish to scale thousands of links, you would find that you will have the IP blocked immediately by Meetup. With this scenario, through rotating proxy services for rotating IPs is a must. You may utilize services like Proxies API for routing your calls using millions of inhabited proxies.
In case, you wish to extract the crawling speeds as well as don’t wish to set up the infrastructure, you may utilize Cloud base crawler from X-Byte Enterprise Crawling to easily scrape thousands of URLs with higher speeds from our crawlers.
✯ Alpesh Khunt ✯
