Welcome to the Treehouse Community
The Treehouse Community is a meeting place for developers, designers, and programmers of all backgrounds and skill levels to get support. Collaborate here on code errors or bugs that you need feedback on, or asking for an extra set of eyes on your latest project. Join thousands of Treehouse students and alumni in the community today. (Note: Only Treehouse students can comment or ask questions, but non-students are welcome to browse our conversations.)
Looking to learn something new?
Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and a supportive community. Start your free trial today.
Building a scrapper using same principles as Node.js Basics
I'm trying to get to work a scrapper but I'm confused on how should I use a Loop to send different queries to the main url. For example the url is this
http://website.com/categories/.json? and I need to scrape through
<numbers> is provided by each page in the JSON format as
My question is: How can I use the
data.next attribute inside a Loop to keep scrapping the website's pages? And what is the proper way to code this using the same principles as this course, like modules, forEach functions, etc?
I would put the scraping into a function that checks for the
next property in
data, and if it finds it, it calls itself again (recursive function), and if not, it just returns/ends.
You'll have to decide what to do with the
data that is returned each time the function runs. Best case would be to return/yield a promise and do something with the result somewhere else (output to file, output to console, etc).