Welcome to the Treehouse Community

The Treehouse Community is a meeting place for developers, designers, and programmers of all backgrounds and skill levels to get support. Collaborate here on code errors or bugs that you need feedback on, or asking for an extra set of eyes on your latest project. Join thousands of Treehouse students and alumni in the community today. (Note: Only Treehouse students can comment or ask questions, but non-students are welcome to browse our conversations.)

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and a supportive community. Start your free trial today.

Python Scraping Data From the Web A World Full of Spiders Installing Scrapy

How do you create a scrapy project?

I'm using pycharm. I have scrapy installed on my project I go to the terminal. I type "scrapy startproject AraneaSpider"

I'm fairly new to using pycharm, I was following the directions in this course. Also I'm using windows.

Then I get this error.

  File "C:\Users\ID611\PycharmProjects\web_scraping\venv\Scripts\scrapy.exe\__main__.py", line 5, in <module>
ModuleNotFoundError: No module named 'scrapy.cmdline'; 'scrapy' is not a package
Eric M
Eric M
11,516 Points

Hi Nathan,

What happens if you type pip show scrapy in your venv terminal?

3 Answers

Eric M
Eric M
11,516 Points

Although Scrapy is showing as installed it's likely something has gone wrong with the install. Perhaps with one of Scrapy's dependencies or path.

If you've only installed via the video's instructions you should only have Scrapy as a project install. You can verify this by running pip show scrapy from a command line outside of Pycharm's venv. Such as a regular powershell terminal. It will be blank.

This means you can try creating a brand new PyCharm project and installing Scrapy again there (the same way as from the video). Be sure to wait for the green install successful message as appears in the video (though it may take a bit longer than in the video).

(venv) C:\Users\ID611\PycharmProjects\web_scraping>pip show scrapy Name: Scrapy Version: 1.5.1 Summary: A high-level Web Crawling and Web Scraping framework Home-page: https://scrapy.org Author: Scrapy developers Author-email: None License: BSD Location: c:\users\id611\pycharmprojects\web_scraping\venv\lib\site-packages Requires: pyOpenSSL, cssselect, queuelib, PyDispatcher, service-identity, parsel, six, w3lib, lxml, Twisted Required-by: You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Thank you for the fix, I just created a new project and ran the same commands.

Eric M
Eric M
11,516 Points

No worries Nathan, glad it worked!