Welcome to the Treehouse Community

The Treehouse Community is a meeting place for developers, designers, and programmers of all backgrounds and skill levels to get support. Collaborate here on code errors or bugs that you need feedback on, or asking for an extra set of eyes on your latest project. Join thousands of Treehouse students and alumni in the community today. (Note: Only Treehouse students can comment or ask questions, but non-students are welcome to browse our conversations.)

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and a supportive community. Start your free trial today.

Python Scraping Data From the Web A World Full of Spiders Installing Scrapy

Problem starting a scrapy project

i'm in the terminal, and I run "scrapy startproject arenaSpider " But then I get

File "C:\Users\ID611\PycharmProjects\web_scraping\venv\Scripts\scrapy.exe_main_.py", line 5, in <module> ModuleNotFoundError: No module named 'scrapy.cmdline'; 'scrapy' is not a package

1 Answer

Alex Koumparos
seal-mask
.a{fill-rule:evenodd;}techdegree
Alex Koumparos
Python Development Techdegree Student 36,862 Points

Hi Nathan,

How did you install scrapy? I'm guessing that you followed the video and installed it from inside PyCharm. I don't use PyCharm or Windows, so I can't replicate that process on my end.

I'm going to suggest, if you haven't tried this already, to completely uninstall the version of scrapy that you have installed. Then try following the official scrapy instructions for Windows users:

Windows

Though it’s possible to install Scrapy on Windows using pip, we recommend you to install Anaconda or Miniconda and use the package from the conda-forge channel, which will avoid most installation issues.

Once you’ve installed Anaconda or Miniconda, install Scrapy with:

conda install -c conda-forge scrapy

If that doesn't work, the problem may be due to having two versions of scrapy installed somewhere on your system and in conflict. Older versions had a different path to cmdline than newer versions, so a version mismatch could explain why the version of scrapy being executed isn't finding the cmdline module.

There have been some people on SO with a similar problem to you (mostly on Unix systems, where the commands used for diagnosis and repair won't be exactly the same as on Windows), like: scrapy error ImportError: No module named scrapy.cmdline. In your case you would use where scrapy instead of which scrapy which should tell you whether the version of scrapy that appears first in your PATH is the same one installed in your virtual environment.

Good luck!

Alex