Heads up! To view this whole video, sign in with your Courses account or enroll in your free 7-day trial. Sign In Enroll
Well done!
You have completed Scraping Data From the Web!
You have completed Scraping Data From the Web!
Preview
Let's further explore how to crawl the web.
Related Discussions
Have questions about this video? Start a discussion with the community and Treehouse staff.
Sign upRelated Discussions
Have questions about this video? Start a discussion with the community and Treehouse staff.
Sign up
This is great if you know all the URLs
of the pages you want to scrape.
0:00
What happens though,
0:01
when you want to start following links
that are included on the page itself?
0:02
Scrapy has some helpful methods for
handling these situations,
0:05
with link extractor, and
crawl spider classes.
0:09
A word of caution here, before we crawl
down this path, we need to be aware of
0:13
the overwhelming amount of data and
sites that are connected on the web.
0:17
Writing a spider that gets and follows
all the links on each followed web page,
0:21
can lead to a program that never ends.
0:26
Also, with the idea that any given site is
only six clicks away from any other site,
0:29
You need to sign up for Treehouse in order to download course files.
Sign upYou need to sign up for Treehouse in order to set up Workspace
Sign up