How does this web scraper work & Why won't it work on workspaces

Hi Marie Lu, I'll go through the code line by line as you asked for :-)

So the first three lines are import commands. The first one import urllib2 import the whole urllib2 package, which is capable of sending requests and receiving responses via http protocols. It is the same command as in the third line for the python regular expression package re.

import re

In the second line this script imports the class BeautifulSoup from the *bs4 package, which is for scraping documents for example based on ther xhtml structure.

from bs4 import BeautifulSoup

Before we move on, you probably receive the error no module named urllib 2 because this package is not installed and therefore cannot be imported. To install the package you would need to use the Python package manager pip or pip3 and install the urllib2 package via the command line with the command pip install urllib2 in your terminal (not in the Python code) Before that you can check which packages are installed with pip list

Now, let's move on:

With opener = urllib2.build_opener() you instantiate the url opener of the urllib2 package. This means you will be able to use its functions like opening a request to an url, which will be done later.

The following line just adds a user agent to your instance of the url opener, so a webpage can process your request.

opener.addheaders = [('User_agent', 'Mozilla 5.0')]

Here you assign your desired url as a string to the variable url. From my point you do not need the curved brackets.

url = ('https://en.wikipedia.org/wiki/List_of_American_comedy_films')

In the next line you provide your url to your instance of the url opener and you read the response. This response is assigned to the variable ourUrl.

ourUrl = opener.open(url).read()

What you want now is to be able to parse your response, why you need to instantiate the BeautifulSoup. This is done in the next line together with our response in the variable ourUrl

Soup = BeautifulSoup(ourUrl)

The actual parsing happens now. In here, I think, we have a little typo as you try to use soup.findAll instead of

Soup.findAll like it was defined in the line above.

But what happens here is, that you would like to go through all links in our response and extract all anchor texts of links which match the regular expression ^/wiki/ and prints the anchor text.

for link in soup.findAll('a',altrs = {'href':re.compile("^/wiki/")}): print (link.text)

In the body line we search for the text Origin and the next td element Documentation

The rest of the code just writes the text of your body variable to the file wikipedia.txt

I hope that helps, just reply for any further questions Cheers Urs