Django Haystack内存错误

Question

所以我用Django的草垛有弹性的搜索，而现在我试图rebuild_index或update_index为55149页的文件和我得到一个Memory Error 。 我认为这是因为它们文件太多，但是我仍然可以如何处理呢？ 请不要，我希望索引约200.000个文件。

python manage.py rebuild_index

WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
All documents removed.
Indexing 55149 processs
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/management/commands/rebuild_index.py", line 16, in handle
    call_command('update_index', **options)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
    return klass.execute(*args, **defaults)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 195, in handle
    return super(Command, self).handle(*items, **options)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/django/core/management/base.py", line 385, in handle
    label_output = self.handle_label(label, **options)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 221, in handle_label
    self.update_backend(label, using)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 267, in update_backend
    do_update(backend, index, qs, start, end, total, self.verbosity)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 89, in do_update
    backend.update(index, current_qs)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 183, in update
    self.conn.bulk_index(self.index_name, 'modelresult', prepped_docs, id_field=ID)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 96, in decorate
    return func(*args, query_params=query_params, **kwargs)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 388, in bulk_index
    query_params=query_params)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 238, in send_request
    **({'data': request_body} if body else {}))
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/sessions.py", line 425, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/adapters.py", line 330, in send
    timeout=timeout
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
    body=body, headers=headers)
  File "/home/vagrant/blook/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 812, in _send_output
    msg += message_body
MemoryError

Answer 1

使用--batch-size=XXX尝试较小的批处理大小

默认值为1000，因此请尝试一些较小的操作，然后逐步提高。

Answer 2

我可以通过增加VM中的内存（之前只有384MB）并释放一些空间来解决此问题。

增加设置中的TIMEOUT可能也有帮助