Installing Scrapy package within Python Environment

Question

This is my very first attempt with both Python and Scrapy

I have installed Scrapy in a virtual environment called scrapy-env using the following commands:

pip install virtualenv
virtualenv `scrapy-env`
cd `scrapy-env`
pip install scrapy

Now I am trying to write my first Python file, which is a spider, I have created a file called spider1.py under the following path:

~/scrapy-env/spider_practice/spider_practice/spiders/spider1.py

I am using pycharm editor, which is complaining about not having Scrapy package installed:

If I click on Install Package Scrapy then I get the following message:

I am not sure why scrapy is not imported at the first place, I thought I have created an environment and installed scrapy? How should I resove this?

Answer 1

I think you wanted to install scrapy inside a virtual environment.

In python you can create virtual environment venv which is very handy:

Usage of virtualenv is deprecated.

To create virtual environment:

python3 -m venv scrapy_env

To activate virtual environment:

source scapy_env/bin/activate

To install scrapy:

pip install scrapy

Open IDE and set scrapy_env/bin/python3 or required python interpreter.

Answer 2

Use pipenv for creating virtual environment

may be you are running on lower then 3.5 version Scrapy runs on Python 3.5 or above

install pipenv with pip3

pip3 install pipenv

install scrapy with pip3

pip3 install Scrapy

create virtual environment with python "whichever_python_version_you_have"

pipenv install --python 'python_version'

to create virtual environment

pipenv run

now check in your folder's pip file. in this file you can see which python version you are using