Logstash翻译插件问题解析csv

Question

I'm trying to parse all the columns of a CSV file (except the first one, obviously). The plugin only get the second column as result of the filter. All the other column are ignored.

It should be possible, according to this sentence getting from the documentation :

When using a CSV dictionary, multiple values in the translation must be extracted with another filter eg Dissect or KV. Note that the fallback is a string so on no match the fallback setting needs to formatted so that a filter can extract the multiple values to the correct fields.

Here is my logstash code :

translate {
    field => "idBatch"
    dictionary_path => "D:\idBatch-description.csv"
    refresh_interval => 500
    destination => "donneesDictionnaireExterne"
    # Données par défaut en l'absence de correspondance
    fallback => "Aucune correspondance trouvée,10000"
    add_tag => [ "import_CSV_ok"]
}

# Mapper des données du dictionnaire externe
dissect {
    mapping => {
        "donneesDictionnaireExterne" => "%{descriptionBatch},%{maxDuration}"
        # EXEMPLE pour GAR01B0 : Batch d'injection Archive;86408
    }
}

Here is a sample of my CSV file :

"GDA08A0_SupPdc","Batch de Suppression de PDC","9999"
"GDI01A0_Parsing","Moteur de parsing des etats internes","9999"

Does anyone know why it doesn't work ?

Answer 1

The translate filter will ignore everything after the second column, you will need to change the format of your dictionary.

Your dictionary needs to be something like this.

"GDA08A0_SupPdc","Batch de Suppression de PDC;9999"
"GDI01A0_Parsing","Moteur de parsing des etats internes;9999"

Then your dissect filter will be like this one:

dissect {
    mapping => {
        "donneesDictionnaireExterne" => "%{descriptionBatch};%{maxDuration}"
    }
}

You can also use a mutate filter to remove the donneesDictionnaireExterne field.

mutate {
    remove_field => ["donneesDictionnaireExterne"]
}

Finally the output for your example is:

{
    "descriptionBatch" => "Batch de Suppression de PDC",
         "maxDuration" => "9999",
            "@version" => "1",
          "@timestamp" => 2019-04-02T02:10:45.107Z,
             "idBatch" => "GDA08A0_SupPdc",
             "message" => "{ \"idBatch\":\"GDA08A0_SupPdc\"}",
                "tags" => [
        [0] "import_CSV_ok"
    ],
                "host" => "hostname"
}
{
    "descriptionBatch" => "Moteur de parsing des etats internes",
         "maxDuration" => "9999",
            "@version" => "1",
          "@timestamp" => 2019-04-02T02:10:45.109Z,
             "idBatch" => "GDI01A0_Parsing",
             "message" => "{ \"idBatch\":\"GDI01A0_Parsing\"}",
                "tags" => [
        [0] "import_CSV_ok"
    ],
                "host" => "hostname"
}