Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python Scraping Data From the Web Introducing Data Scraping Being a Good Citizen

Posted by Mark Chesney
Mark Chesney
11,747 Points

robots.txt and web scraping

Hi. Regarding robots.txt, I read examples in Wikipedia. I'm wondering are there coding examples to learn how to get the gist of these?

1 Answer

frankgenova
.a{fill-rule:evenodd;}techdegree
frankgenova
Python Web Development Techdegree Student 15,616 Points

This site looks like it might have some useful information. I have not reviewed it so I'm only providing it because it might be useful:

https://moz.com/learn/seo/robotstxt

or maybe try:

https://developers.google.com/search/reference/robots_txt