Python Scraping Data From the Web A World Full of Spiders Installing Scrapy

Nathan English
Nathan English
2,644 Points

How do you create a scrapy project?

I'm using pycharm. I have scrapy installed on my project I go to the terminal. I type "scrapy startproject AraneaSpider"

I'm fairly new to using pycharm, I was following the directions in this course. Also I'm using windows.

Then I get this error.

  File "C:\Users\ID611\PycharmProjects\web_scraping\venv\Scripts\scrapy.exe\__main__.py", line 5, in <module>
ModuleNotFoundError: No module named 'scrapy.cmdline'; 'scrapy' is not a package
Eric McKibbin
Eric McKibbin
Treehouse Moderator 11,445 Points

Hi Nathan,

What happens if you type pip show scrapy in your venv terminal?

3 Answers

Eric McKibbin
MOD
Eric McKibbin
Treehouse Moderator 11,445 Points

Although Scrapy is showing as installed it's likely something has gone wrong with the install. Perhaps with one of Scrapy's dependencies or path.

If you've only installed via the video's instructions you should only have Scrapy as a project install. You can verify this by running pip show scrapy from a command line outside of Pycharm's venv. Such as a regular powershell terminal. It will be blank.

This means you can try creating a brand new PyCharm project and installing Scrapy again there (the same way as from the video). Be sure to wait for the green install successful message as appears in the video (though it may take a bit longer than in the video).

Nathan English
Nathan English
2,644 Points

(venv) C:\Users\ID611\PycharmProjects\web_scraping>pip show scrapy Name: Scrapy Version: 1.5.1 Summary: A high-level Web Crawling and Web Scraping framework Home-page: https://scrapy.org Author: Scrapy developers Author-email: None License: BSD Location: c:\users\id611\pycharmprojects\web_scraping\venv\lib\site-packages Requires: pyOpenSSL, cssselect, queuelib, PyDispatcher, service-identity, parsel, six, w3lib, lxml, Twisted Required-by: You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Nathan English
Nathan English
2,644 Points

Thank you for the fix, I just created a new project and ran the same commands.