Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community!

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Python Scraping Data From the Web Introducing Data Scraping Review Introduction to Data Scraping


Many sites use a .txt file to limit where scraping bots can go on the site itselF

1 Answer

Maxwell Newberry
Maxwell Newberry
7,693 Points

Robots.txt - For information on this, see 2:05 on Being a Good Citizen.