Running scrapy with PyCharm - Debug works but Run does not work

Question

I met a very strange issue, running Scrapy with PyCharm:
With the exact same configuration, Debug works, but Run with PyCharm does NOT work.

• Windows 10
• PyCharm 2016.3.3
• Scrapy 1.3.3
• Python 3.6.0

Configuration: PyCharm Edit Configuration Page

When I attempt to debug scrapy, it works perfectly. When I attempt to run scrapy, I get below error:

C:\Users\baib2\AppData\Local\Programs\Python\Python36\python.exe 
C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py crawl scenelist_spider
Traceback (most recent call last):
  File "C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py", line 8, in <module   
    import scrapy
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\__init__.py", line 27, in <module   
    from . import _monkeypatches
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\_monkeypatches.py", line 20, in <module   
    import twisted.persisted.styles  # NOQA
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\persisted\styles.py", line 21, in <module   
    from twisted.python.compat import _PY3, _PYPY
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\__init__.py", line 11, in <module   
    from .compat import unicode
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\compat.py", line 612, in <module   
    from http import cookiejar as cookielib
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\Lib\site-packages\scrapy\http\__init__.py", line 8, in <module   
    from scrapy.http.headers import Headers
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\__init__.py", line 10, in <module   
    from scrapy.http.request import Request
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\request\__init__.py", line 8, in <module   
    from w3lib.url import safe_url_string
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\w3lib\url.py", line 17, in <module   
    from six.moves.urllib.request import pathname2url, url2pathname
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 160, in _resolve
    module = _import_module(self.mod)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 82, in _import_module
    __import__(name)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 88, in <module   
    import http.client
ModuleNotFoundError: No module named 'http.client'

Process finished with exit code 1

No need to attach any code because even running the command:

python3.exe C:\\Users\\baib2\\AppData\\Local\\Programs\\Python\\Python36\\Lib\\site-packages\\scrapy\\cmdline.py

will produce the same error.

I've checked my sys.path , comparing with run, debug has 1 more path:

'C:\\Program Files (x86)\\JetBrains\\PyCharm 2016.3.3\\helpers\\pydev'

And I really don't think this shall make any difference.

Hope someone can take a look, thanks!

Answer 1

菜单运行>编辑配置启用复选框“以后显示命令行”这对我有帮助

Answer 2

如果您使用的是新版本的PyCharm， Show Command line afterward不再可用，则Run with Python console可解决此问题。

Answer 3

I've also spent some time tracking a similar problem.

My issue was that I have a local module named http which of course does not submodule called client . Another similar issue is described here .

The solution is to change Working directory for this test invocation via Edit configurations .