How to fix RequestError for python Elasticsearch bulk?

Question

I'm trying to read data from a website and save to Elasticsearch. But I got the following error. I think its because of my doc which isn't correct. I followed the bulk elastic documentation. Still I'm struggling how to solve this. Can someone help me?

My error:

RequestError: RequestError(400, 'action_request_validation_exception', 'Validation Failed: 1: type is missing;2: type is missing;')

My code is:

import os
import requests
import time
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
import json
es = Elasticsearch('http://ip:port',timeout=600)
while (True):
 df = requests.get("https://my url").json()  

 del df["positions"][1]
 print(df)
 def gendata(df):
    for word in df:
        yield {
        "_index": "chapter",
        "doc": {"info": {
                          "satnam":{"type":"text"},
                          "satid":{"type":"integer"},
                          "transactionscount":{"type":"integer"}},
                "positions": {"satlatitude":{"type":"float"},
                          "satlongitude":{"type":"float"},
                          "sataltitude":{"type":"float"},
                          "azimuth":{"type":"float"},
                          "elevation":{"type":"float"},
                          "ra": {"type":"float"},
                           "dec": {"type":"float"},
                           "timestamp":{"type":"datetime"} }}
    }
 bulk(es,gendata(df))
 time.sleep(10)`

My Json file from website is:

{"info":{"satname":"SPACE STATION","satid":00000,"transactionscount":0}, "positions":{"satlatitude":-50.00807313,"satlongitude":-37.47024176,"sataltitude":435.04,"azimuth":178.02,"elevation":-47.22,"ra":132.83647708,"dec":-72.05784906,"timestamp":1589984178}

Answer 1

Main thing: You are doing a bulk() for your mappings. You are supposed to do it for your document objects.

So this is how your code should get refactored:

• Create the index 'chapter'.

• Create the mappings for 'chapter' index however you expect it to have the documents.

• Then start indexing documents to the index.

And another thing: you don't need to declare mappings in json style - better approach to use elasticsearch-dsl => https://elasticsearch-dsl.readthedocs.io/en/latest/

So you would be defining a class like so (please see details in above reference):

class Chapter(Document):
    satnam=Text()
    satid=Integer()
    transactionscount=Integer()
    .
    .
    .


    class Index:
        name = 'chapter'

This class is mainly for giving mapping-details/index-name/analyzers etc.

Once you call Chapter.init() - the index 'chapter' will be created.

And then comes your code for bulk indexing the documents into the 'chapter' index which you just created using init().

Answer 2

I believe your bulk insert elastic query format is wrong the expected query syntax would be

{"index": {"_index": "chapter"}}
{"info": {
                          "satnam":{"type":"text"},
                          "satid":{"type":"integer"},
                          "transactionscount":{"type":"integer"}},
                "positions": {"satlatitude":{"type":"float"},
                          "satlongitude":{"type":"float"},
                          "sataltitude":{"type":"float"},
                          "azimuth":{"type":"float"},
                          "elevation":{"type":"float"},
                          "ra": {"type":"float"},
                           "dec": {"type":"float"},
                           "timestamp":{"type":"datetime"} }}

reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html