使用 python 将 JSON 发送到带有时间戳字段的 ElasticSearch 索引

Question

我有一个脚本可以调用我的 API 并提取股票数据。 我想将来自 API 的响应保存为 JSON 文件，然后使用响应中的“t”字段作为日期字段/时间戳创建 ES 索引。

我可以看到 ES 集群中的数据，但它没有被索引，并且“t”字段显示为错误的类型。 长而不是日期。

我不确定如何最好地索引results因为这对我来说很关键，因为我真的想将其显示为时间序列。

收藏家.py

def collect():
    import json
    key = ""
    url = "https://api"
    payload={}
    headers = {}
    response = requests.request("GET", url, headers=headers, data=payload)
    j = response.json()
    print (j['results'][1])
    data = response.text
    for i in j['results']: 
        print (i)
    with open('data.json', 'w', encoding='utf-8') as f:
        json.dump(j, f, ensure_ascii=False, indent=4)
    import os, sys
    from elasticsearch import Elasticsearch
    directory = '/home/'
    res = requests.get('http://localhost:9200')    
    print (res.content)
    es = Elasticsearch([{'host': 'localhost', 'port': '9200'}])
    i = 1
    for filename in os.listdir(directory):
        if filename.endswith(".json"):
         f = open(filename)
         docket_content = f.read()
         # Send the data into es
         es.index(index='core', ignore=400, doc_type='docket', 
         id=i, body=json.loads(docket_content))
         i = i + 1
    return render_template('show.html', data=data)

JSON 响应和 data.json 的内容

{
  "ticker": "AAPL",
  "queryCount": 10,
  "resultsCount": 10,
  "adjusted": true,
  "results": [
    {
      "v": 1668,
      "vw": 151.8826,
      "a": 151.8826,
      "o": 152,
      "c": 151.92,
      "h": 152,
      "l": 151.75,
      "t": 1636012800000,
      "n": 70
    },
    {
      "v": 1467,
      "vw": 151.9323,
      "a": 151.9059,
      "o": 151.95,
      "c": 151.96,
      "h": 151.96,
      "l": 151.89,
      "t": 1636012860000,
      "n": 60
    },
    {
      "v": 1096,
      "vw": 151.9585,
      "a": 151.9195,
      "o": 151.96,
      "c": 151.94,
      "h": 151.96,
      "l": 151.94,
      "t": 1636012920000,
      "n": 64
    },
    {
      "v": 1303,
      "vw": 151.7871,
      "a": 151.8889,
      "o": 151.77,
      "c": 151.73,
      "h": 151.77,
      "l": 151.73,
      "t": 1636013040000,
      "n": 70
    },
    {
      "v": 847,
      "vw": 151.8279,
      "a": 151.8811,
      "o": 151.87,
      "c": 151.8,
      "h": 151.87,
      "l": 151.8,
      "t": 1636013100000,
      "n": 37
    },
    {
      "v": 451,
      "vw": 151.8722,
      "a": 151.8737,
      "o": 151.87,
      "c": 151.87,
      "h": 151.87,
      "l": 151.87,
      "t": 1636013280000,
      "n": 17
    },
    {
      "v": 5347,
      "vw": 151.8021,
      "a": 151.8446,
      "o": 151.82,
      "c": 151.8,
      "h": 151.82,
      "l": 151.8,
      "t": 1636013400000,
      "n": 55
    },
    {
      "v": 2834,
      "vw": 151.7431,
      "a": 151.8266,
      "o": 151.74,
      "c": 151.73,
      "h": 151.74,
      "l": 151.73,
      "t": 1636013460000,
      "n": 58
    },
    {
      "v": 615,
      "vw": 151.7193,
      "a": 151.8226,
      "o": 151.73,
      "c": 151.68,
      "h": 151.73,
      "l": 151.68,
      "t": 1636013520000,
      "n": 22
    },
    {
      "v": 876,
      "vw": 151.717,
      "a": 151.8173,
      "o": 151.71,
      "c": 151.73,
      "h": 151.73,
      "l": 151.71,
      "t": 1636013580000,
      "n": 33
    }
  ],
  "status": "OK",
  "request_id": "2354523452356236",
  "count": 10
}

弹性指数

core    
mappings    
properties  
a   
type    "float"
c   
type    "float"
h   
type    "float"
l   
type    "float"
n   
type    "long"
o   
type    "float"
t   
type    "long"
v   
type    "long"
vw  
type    "float"

Answer 1

你真的应该在索引之前定义你的映射，要么使用模板，要么使用集合映射发布索引。 否则 Elasticsearch 将尝试为每个字段选择最佳数据类型，但它并不总是正确的，如您所见

https://elasticsearch-py.readthedocs.io/en/v7.15.1/api.html?highlight=mapping#elasticsearch.client.IndicesClient.put_mapping可能就是你想要的