Elasticsearch update document in python

Question

I am trying to update document in elasticsearch python,

from elasticsearch import Elasticsearch
from datetime import datetime
from scapy.all import *
import json
# es = Elasticsearch()
es = Elasticsearch(['http://localhost:9200'])

doc = getDoc("1")
    print doc
    if doc != None:
      doc['_op_type'] = 'update'
      doc['_source']['macList'].append('new')
      helpers.bulk(es, doc, stats_only=False)

But this is not working. Can someone please tell me what I am doing wrong ? thanks

This is the error:

Traceback (most recent call last):
  File "./req.py", line 48, in <module>
    helpers.bulk(es, doc, stats_only=False)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 188, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 160, in streaming_bulk
    for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 89, in _process_bulk_chunk
    raise e
elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: index is missing;2: type is missing;3: index is missing;4: type is missing;5: index is missing;6: type is missing;7: index is missing;8: type is missing;9: index is missing;10: type is missing;11: index is missing;12: type is missing;13: index is missing;14: type is missing;')

Update I have tryed this : es.update(index='macs', doc_type='users', id="1", body=doc) and I am having this error: Traceback (most recent call last):

  File "./req.py", line 50, in <module>
    es.update(index='macs', doc_type='users', id="1", body=doc)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 460, in update
    doc_type, id, '_update'), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 329, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 108, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: script or doc is missing;').

It is so easy with postman but I really don't understand why it is so had with Elasticsearch (python) to update doc. Someone has an other idea to help ?

Answer 1

Since the second argument in helpers.bulk() method is should be an iterable , try enclosing it in a list.

Also looking at your code, it seems you are expecting only one document from the getDoc() function. So it would be better if you use simple es.update() method instead of using helpers.bulk()

Answer 2

This solution did work for me.

doc= ess.get(...) 
# modify my doc and ...   
coll = Elasticsearch()
coll.update(index='stories-test',doc_type='news',id=hit.meta.id,
                    body= {"doc": doc['_source']} )

Answer 3

(In case someone lands up here with a similar problem,) I was getting the first error mentioned above:

elasticsearch.exceptions.RequestError: TransportError(400, 
'action_request_validation_exception', 'Validation Failed: 
1: index is missing;2: type is missing;

and the following fixed it for me:

obj.to_dict(include_meta=True)

When converting the obj to dict, I added include_meta=True and it worked for me.