Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python

Posted by Rodrigo Muñoz
Rodrigo Muñoz
Courses Plus Student 20,171 Points

Is it possible to create a web scrapping website

I have a simple algorithm that loads the Google stocks price from Yahoo Finance and saves it into a .csv file, but I'm trying to create a website that is constantly loading the price and displays it on the front page. Is it possible to create this? How can I manage to webscrap and display at the frontend at the same time?

Here is what I've done:

from urllib import request

stock_url = 'http://real-chart.finance.yahoo.com/table.csv?s=GOOG&d=3&e=15&f=2015&g=d&a=2&b=27&c=2014&ignore=.csv'

def download_stock_data(csv_url):
    response = request.urlopen(csv_url)
    csv = response.read()
    csv_str = str(csv)
    lines = csv_str.split("\\n")
    dest_url = r'goog.csv'
    fx = open(dest_url, "w")
    for line in lines:
        fx.write(line + "\n")
    fx.close()

download_stock_data(stock_url)
print(__name__)

2 Answers

Ricky Catron
Ricky Catron
13,023 Points

An API like Erik mentions would be the best solution. I believe what is usually done in this case is you build a view which scrapes the data and displays the information in a format called JSON. Then you would get the data from that page using JavaScript and display it on the page you wanted.

This would be pretty easy in flask, reuse most of the code you have but instead of using a CSV return the data using flask builtin JSON.

Goodluck! --Ricky

Rodrigo Muñoz
Rodrigo Muñoz
Courses Plus Student 20,171 Points

Thanks you both guys. I've actually concluded that I need to use the requests package to load the URL and then try to figure out how to display the content on the page.

Erik Nilsen
Erik Nilsen
20,433 Points
Erik Nilsen
Erik Nilsen
20,433 Points

While I am not familiar with the python syntax, I would use that api to generate the HTML you need. It should be no different than the .csv you are already creating. In these situations Google is your best friend.