Scrapy（Python）与cron一起运行时抛出ImportError

Question

我正在用cron运行一个抓痒的蜘蛛，但是它抛出一个ImportError异常：

Traceback (most recent call last):
  File "/Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py", line 2, in <module>
    import scrapy
  File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line 12, in <module>
    from scrapy.http.request.rpc import XmlRpcRequest
  File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py", line 7, in <module>
    from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

奇怪的是，当我运行由cron运行的脚本时，它运行良好。

cron设置为

*   *   *   *   *   sh /Users/som/sh/hm_scraping.sh

脚本是

#!/bin/bash
python /Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py

我正在按如下所述使用CrawlerProcess类： http ://doc.scrapy.org/en/latest/topics/practices.html

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(HmSpider)
process.start()

================================================
编辑

基于MuhammadTahir和lapinkoira的评论，我直接在终端中测试了以下内容：

/usr/bin/python /Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py

和

sudo -u som /usr/bin/python /Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py

第一个运行良好，但是当我使用sudo（我也没有设置用户的情况下运行）时，它会返回相同的问题。 也许cron在后台使用sudo。

有任何想法吗？？

谢谢！

Answer 1

我会尝试两者之一：

1-首先激活环境：

source /path/of/your/venv/bin/activate && /path/of/your/venv/bin/python /Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py

2-或不激活环境（可能不起作用）：

/path/of/your/venv/bin/python /Users/som/scrapy_testing/scrapy_testing/spiders/hm_spiders.py