Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python

Posted by Mayur Pande
Mayur Pande
Courses Plus Student 11,711 Points

Need some help on thought process for python

So my work have finally asked me to help with some programming (woo hoo!).

But they have asked me to help with building a python scraper using beautiful soup (I am used to JS).

I have no code so far, but this is my thought process.

So does anyone have any ideas on how I can go about the following;

1) Scrape certain websites for specific article links using class or id names and check whether I have the article link.

I think I can solve this by using beautiful soup after reading the docs.

2) After obtaining this info, check whether the article link (and other info for article i.e, title, date, country, is relevant) is listed in either CSV document or MySQL db preferably MySQL db if it isn't add it to the db.

This is the part I am getting stuck on, I know some MySQL but am a bit confused as how to go about it. I guess I can compare by dates.

3) Create a site with the data from the db put into a table, only catch here is I should add a checkbox to each row of the table saying "is relevant" if it is relevant it stays in the db and table of the site. If it isn't delete is from the db and table.

I am also slightly getting confused about this, would I add the checkbox value to the db somehow? Or would I add it when the site is created.

Sorry if this is confusing, please let me know if I can try and put it into more layman's terms.

1 Answer

Iain Simmons
MOD
Iain Simmons
Treehouse Moderator 32,305 Points
Iain Simmons
Iain Simmons
Treehouse Moderator 32,305 Points

So, there are a bunch of courses here on Treehouse, but here are some that might more directly address the challenges you're facing:

There's also Django, but it might be a bit overkill for what you're doing. Though it does a lot of the setup for you, as far as databases, etc and provides an excellent, customisable admin interface.

Let us know how you go, or if there's any more specific question you have.