Python Scraping Data From the Web A World Full of Spiders Installing Scrapy

Nathan English
Nathan English
2,644 Points

Problem starting a scrapy project

i'm in the terminal, and I run "scrapy startproject arenaSpider " But then I get

File "C:\Users\ID611\PycharmProjects\web_scraping\venv\Scripts\scrapy.exe_main_.py", line 5, in <module> ModuleNotFoundError: No module named 'scrapy.cmdline'; 'scrapy' is not a package

1 Answer

Alex Koumparos
MOD
Alex Koumparos
Python Web Development Treehouse Moderator 33,475 Points

Hi Nathan,

How did you install scrapy? I'm guessing that you followed the video and installed it from inside PyCharm. I don't use PyCharm or Windows, so I can't replicate that process on my end.

I'm going to suggest, if you haven't tried this already, to completely uninstall the version of scrapy that you have installed. Then try following the official scrapy instructions for Windows users:

Windows

Though it’s possible to install Scrapy on Windows using pip, we recommend you to install Anaconda or Miniconda and use the package from the conda-forge channel, which will avoid most installation issues.

Once you’ve installed Anaconda or Miniconda, install Scrapy with:

conda install -c conda-forge scrapy

If that doesn't work, the problem may be due to having two versions of scrapy installed somewhere on your system and in conflict. Older versions had a different path to cmdline than newer versions, so a version mismatch could explain why the version of scrapy being executed isn't finding the cmdline module.

There have been some people on SO with a similar problem to you (mostly on Unix systems, where the commands used for diagnosis and repair won't be exactly the same as on Windows), like: scrapy error ImportError: No module named scrapy.cmdline. In your case you would use where scrapy instead of which scrapy which should tell you whether the version of scrapy that appears first in your PATH is the same one installed in your virtual environment.

Good luck!

Alex