如何格式化德鲁伊语中的TSV文件

Question

I am trying to load in a TSV in druid using this ingestion speck:

MOST UPDATED SPEC BELOW:

{                                                                                                                                                                                               
"type" : "index",
"spec" : {
    "ioConfig" : {
        "type" : "index",
        "inputSpec" : {
            "type": "local",
            "baseDir": "quickstart",
            "filter": "test_data.json"
        }
    },
    "dataSchema" : {
        "dataSource" : "local",
        "granularitySpec" : {
            "type" : "uniform",
            "segmentGranularity" : "hour",
            "queryGranularity" : "none",
            "intervals" : ["2016-07-18/2016-07-22"]
        },
        "parser" : {
            "type" : "string",
            "parseSpec" : {
                "format" : "json",
                "dimensionsSpec" : {
                    "dimensions" : ["name", "email", "age"]
                },
                "timestampSpec" : {
                    "format" : "yyyy-MM-dd HH:mm:ss",
                     "column" : "date"
                }
            }
        },
        "metricsSpec" : [
            {
                "name" : "count",
                "type" : "count"
            },
            {
              "type" : "doubleSum",
              "name" : "age",
              "fieldName" : "age"
            }
        ]
    }
}

}

If my schema looks like this:

Schema: name    email    age

And actual dataset looks like this:

name    email    age    Bob    Jones    23    Billy    Jones    45

Is this how the columns should be formatted^^ in the above dataset for a TSV? Like name email age should be first (the columns) and then the actual data. I am confused how Druid will know how to map the columns to the actual dataset in TSV format.

Answer 1

TSV stands for tab separated format, so it looks the same as csv but you will use tabs instead of commas eg

Name<TAB>Age<TAB>Address
Paul<TAB>23<TAB>1115 W Franklin
Bessy the Cow<TAB>5<TAB>Big Farm Way
Zeke<TAB>45<TAB>W Main St

you will use frist line as header to define your column names - so you can use "name", "age" or "email" in dimensions in your spec file

as for the gmt and utc, they are basically the same

There is no time difference between Greenwich Mean Time and Coordinated Universal Time

first one is time zone, the other one is a time standard

btw don`t forget to include a column with some time value in your tsv file!!

so eg if you will have tsv file that looks like:

"name"  "position"  "office"    "age"   "start_date"    "salary"
"Airi Satou"    "Accountant"    "Tokyo" "33"    "2016-07-16T19:20:30+01:00" "162700"
"Angelica Ramos"    "Chief Executive Officer (CEO)" "London"    "47"    "2016-07-16T19:20:30+01:00" "1200000"

your spec file should look like this:

{
    "spec" : {
        "ioConfig" : {
            "inputSpec" : {
                "type": "local",
                "baseDir": "path_to_folder",
                "filter": "name_of_the_file(s)"
            }
        },
        "dataSchema" : {
            "dataSource" : "local",
            "granularitySpec" : {
                "type" : "uniform",
                "segmentGranularity" : "hour",
                "queryGranularity" : "none",
                "intervals" : ["2016-07-01/2016-07-28"]
            },
            "parser" : {
                "type" : "string",
                "parseSpec" : {
                    "format" : "tsv",
                    "dimensionsSpec" : {
                        "dimensions" : [
                            "position",
                            "age",
                            "office"
                        ]
                    },
                    "timestampSpec" : {
                        "format" : "auto",
                         "column" : "start_date"
                    }
                }
            },
            "metricsSpec" : [
                {
                    "name" : "count",
                    "type" : "count"
                },
                {
                    "name" : "sum_sallary",
                    "type" : "longSum",
                    "fieldName" : "salary"
                }
            ]
        }
    }
}