使用Python将数据帧索引到Elasticsearch中

Question

I'm trying to indexing some pandas dataframe into ElasticSearch. I have some troubles while parsing the json that I'm generating. I think that my problem is coming from the mapping. Please below find my code.

import logging
from pprint import pprint
from elasticsearch import Elasticsearch
import pandas as pd

def create_index(es_object, index_name):
    created = False
    # index settings
    settings = {
        "settings": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        },
        "mappings": {
            "danger": {
                "dynamic": "strict",
                "properties": {
                    "name": {
                       "type": "text"
                    },
                    "first_name": {
                        "type": "text"
                    },
                    "age": {
                        "type": "integer"
                    },
                    "city": {
                        "type": "text"
                    },
                    "sex": {
                        "type": "text",
                    },
                }
            }
        }
    }

    try:
        if not es_object.indices.exists(index_name):
            #Ignore 400means to ignore "Index Already Exist" error
            es_object.indices.create(index=index_name, ignore=400,     
body=settings)
            print('Created Index')
        created = True
    except Exception as ex:
        print(str(ex))
    finally:
        return created


def store_record(elastic_object, index_name, record):
    is_stored = True
    try:
        outcome = elastic_object.index(index=index_name,doc_type='danger', body=record)
        print(outcome)
    except Exception as ex:
        print('Error in indexing data')


data = [['Hook', 'James','90', 'Austin','M'],['Sparrow','Jack','15', 'Paris', 'M'],['Kent','Clark','13', 'NYC', 'M'],['Montana','Hannah','28','Las Vegas', 'F'] ]
df = pd.DataFrame(data,columns=['name', 'first_name', 'age', 'city', 'sex'])
result = df.to_json(orient='records')
result = result[1:-1]
es = Elasticsearch()
if es is not None:
        if create_index(es, 'cracra'):
            out = store_record(es, 'cracra', result)
            print('Data indexed successfully')

I got the following error

POST http://localhost:9200/cracra/danger [status:400 request:0.016s]

Error in indexing data
RequestError(400, 'mapper_parsing_exception', 'failed to parse')
Data indexed successfully

I don't know where it is coming from. If anyone may help me to solve this, I would be grateful.

Thanks a lot !

Answer 1

Try to remove extra commas from your mappings:

"mappings": {
  "danger": {
    "dynamic": "strict",
    "properties": {
      "name": {
        "type": "text"
      },
      first_name": {
        "type": "text"
      },
      "age": {
        "type": "integer"
      },
      "city": {
        "type": "text"
      },
      "sex": {
        "type": "text", <-- here
      }, <-- and here
    }
  }
}

UPDATE

It seems that the index is created successfully and the problem is in data indexing. As Nishant Saini noted you probably are trying to index several documents at a time. It can be done using Bulk API . Here is the example of correct request that indexes two documents:

POST cracra/danger/_bulk
{"index": {"_id": 1}}
{"name": "Hook", "first_name": "James", "age": "90", "city": "Austin", "sex": "M"}
{"index": {"_id": 2}}
{"name": "Sparrow", "first_name": "Jack", "age": "15", "city": "Paris", "sex": "M"}

Every document in the request body must appear in the new line with some meta information before it. In this case metainfo contains only id that must be assigned to the document.

You can either make this query by hand or use Elasticsearch Helpers for Python that can take care of adding correct metainfo.