使用批量 api elasticsearch kibana 更改 csv 列的字段格式

Question

I want to change the type of one of the columns of my.csv file that I import via bulk api in elastic search in python. The column contains dates but is imported as a string (however, when I upload the file manually in kibana, it takes it in date format).

es = Elasticsearch()
    with open('user.csv') as f:
        reader = csv.DictReader(f)
        helpers.bulk(es, reader, index='user', doc_type='my-type')

I already tried mapping but it doesn't work:

mapping = {
  "mappings": {
    "my-type": {
      "properties": {
        "('affiliation',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('banned',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('bracket',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('country',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('created',)": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('email',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('hidden',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('id',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('name',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('oauth_id',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('password',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('promotion',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('school',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('secret',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('speciality',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('type',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('verified',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "('website',)": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

    es.indices.create(index='user', ignore=400, body=mapping)
    with open('user.csv') as f:
        reader = csv.DictReader(f)
        helpers.bulk(es, reader, index='user', doc_type='csv')

Do you have any ideas or solutions? Thanks a lot !

Answer 1

The doc types need to be consistent in order for the correct mapping to be applied. Your first vs second call:

helpers.bulk(es, reader, index='user', doc_type= 'my-type' )

helpers.bulk(es, reader, index='user', doc_type= 'csv' )

If your mapping configures 'my-type' , reference it as such in all subsequent function calls.

But more importantly, reading from a CSV doesn't guarantee any original column types -- most of them will be read in as strings, As such. it's recommended to pre-process your docs' attributes to guarantee they'll be treated correctly -- ie, dates, numbers, booleans. etc.

In the function generateBulkPayload below you can parse/modify select values right before they're inserted into ES:

import csv
from elasticsearch import Elasticsearch
from elasticsearch import helpers

es = Elasticsearch()

index_name = "user"
doc_type = "my-type"

mapping = {
    "mappings": {
        "my-type": {
            "created": {
                "type": "date",
                "format": "epoch_millis"  # assuming you're dealing with millisecond timestamps
            }
        }
    }
}

es.indices.create(index=index_name, ignore=400, body=mapping)


def generateBulkPayload(csv_reader):
    for row in csv_reader:
        # handle your parsing here
 
        # overwriting the `created` attribute
        row.update(dict(created=int(row.get('created'))))

        yield row


with open('user.csv') as f:
    reader = csv.DictReader(f)
    helpers.bulk(es,
                 generateBulkPayload(reader),
                 index=index_name,
                 doc_type=doc_type)

Answer 2

This code compiles well but the date format is still not recognized by elasticsearch. What to do so that elasticsearch recognizes it?

def generateBulkPayload(csv_reader):
    for row in csv_reader:
        created=row.get("('created',)") # Base format : 2021-03-04 13:56:16.663801
        datetime = parser.parse(created)
        epoch= datetime.timestamp()

        row.update(dict(created=int(epoch)))

        yield row