[英]Python Watchdog spawn multiprocessing
我正在将文件(以非常快的速度)写入特定目录。 我想监视目录中是否有新文件,然后生成一个运行外部脚本的进程。 现在,我收到一个酸洗错误(即使我使用的是Pathos)
Can't pickle <type 'Struct'>: it's not found as __builtin__.Struct
我需要解决酸洗错误的帮助,这可能会使我不得不重新考虑自己在做什么,这很好。
这是我到目前为止的内容:
#!/usr/bin/python
import os
import sys
import argparse
import json
import time
import os
from datetime import datetime
#Test for Pathos
from pathos.multiprocessing import ProcessingPool as Pool
from multiprocessing import cpu_count
from subprocess import check_output
import ConfigParser
import logging
#WatchDog
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler, FileSystemMovedEvent
from CodernityDB.database import Database
###
# CONFIGURATION
###
CONFIG GOES HERE BUT REMOVED
###
# Custom handler for Python Watchdog
# When spawned, it will spawn a new worker into the pool
###
class MyHandler(FileSystemEventHandler):
def __init__(self):
self.db = Database("/var/db/test.db")
try:
self.db.open()
except Exception, e:
print str(e)
self.db.create()
def on_created(self, event):
#print event.src_path
try:
pool.map(doIt, (self.db, event.src_path,))
except Exception, e:
print str(e)
def codernityIt(db, json_):
try:
print json_
db.insert(json_)
except Exception, e:
print str(e)
logging.error(str(e))
def doIt(db, file_):
try:
codernityIt(db, json.loads(check_output(['python', '/external/script.py', file_])))
except Exception, e:
print str(e)
logging.error(str(e))
if __name__ == '__main__':
###
# Pool specific Settings
###
pool = Pool(processes=cpu_count())
event_handler = MyHandler()
###
# Watchdog specific settings
###
observer = Observer()
observer.schedule(event_handler, path=watchPath, recursive=True)
observer.start()
###
# This While True loop listens for Keyboard interrupts and will gracefully exit the program if found
###
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.unschedule_all()
observer.stop()
db.close()
#observer.join()
你为什么不尝试变声? 它可能会帮助您: https : //pypi.python.org/pypi/inotify
我很确定它会失败,因为dill
( pathos
使用的是dill
)不知道如何腌制Struct
。 这是我认为dill
目前无法处理的少数几个对象之一,因为未定义它在命名空间中的位置……请参阅: https : //github.com/python/typeshed/issues/24 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.