無法獲取logstash在彈性搜索中創建/填充索引

Question

這是我正在使用的：

CentOS 6.5 x64
Logstash 1.5.0
Elasticsearch 1.5.2
Marvel

所有新安裝的。

這是我的logstash配置文件：

input {

file {

    path => "/home/datafeeds/public_html/myfeeds/datafeeds/completed/*.csv"
    }

}


filter {

csv {


    columns => ["z_md5","z_product_name_slug","z_country_code","z_product_name","z_expiry_timestamp","z_expiry_human","z_product_name_gb","z_description_gb","z_product_name_and_description_gb_md5","z_product_name_fr","z_description_fr","z_product_name_and_description_fr_md5","z_product_name_de","z_description_de","z_product_name_and_description_de_md5","z_product_name_it","z_description_it","z_product_name_and_description_it_md5","z_product_name_es","z_description_es","z_product_name_and_description_es_md5","feed_ID","feed_product_ID","feed_merchant_product_id","feed_merchant_category","feed_deep_link","feed_merchant_image_url","feed_search_price","feed_description","feed_product_name","feed_merchant_deep_link","feed_image_url","feed_merchant_name","feed_merchant_id","feed_category_name","feed_category_id","feed_delivery_cost","feed_currency","feed_store_price","feed_display_price","feed_data_feed_id","feed_rrp_price","feed_specifications","feed_condition","feed_promotional_text","feed_warranty","feed_merchant_thumb_url","feed_thumb_url","feed_brand_name","feed_brand_id","feed_delivery_time","feed_valid_from","feed_valid_to","feed_web_offer","feed_pre_order","feed_in_stock","feed_stock_quantity","feed_is_for_sale","feed_product_type","feed_commission_group","feed_upc","feed_ean","feed_mpn","feed_isbn","feed_model_number","feed_parent_product_id","feed_language","feed_last_updated","feed_dimensions","feed_colour","feed_keyword","feed_custom_1","feed_custom_2","feed_custom_3","feed_custom_4","feed_custom_5","feed_saving","feed_delivery_weight","feed_delivery_restrictions","feed_reviews","feed_average_rating","feed_number_stars","feed_number_available","feed_rating","feed_alternate_image","feed_large_image","feed_basket_link"]
    separator => ","

}

}



output {

elasticsearch {

    action => "index"
    host => "localhost"
    index => "products_test"
    workers => 1
}
}

我已經仔細檢查過，並在其中找到一個csv文件：

/home/datafeeds/public_html/myfeeds/datafeeds/completed/

我已將文件CHMOD 777確定為可讀性。

我已經測試過我的配置文件，以確保一切正常

$ sudo /opt/logstash/bin/logstash --configtest -t -f /etc/logstash/conf.d/products.conf
$ Configuration OK

彈性搜索和logstash正在運行，但是沒有數據傳遞給elasticsearch或創建的索引。

我嘗試手動創建索引，但未填充任何數據

我嘗試重新啟動服務器，多次停止/重新啟動logstash /彈性搜索。

這是以詳細模式啟動logstash的輸出：

sudo /opt/logstash/bin/logstash --verbose -f /etc/logstash/conf.d/products.conf
Using version 0.1.x input plugin 'file'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x codec plugin 'plain'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x filter plugin 'csv'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Using version 0.1.x output plugin 'elasticsearch'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info}
Registering file input {:path=>["/home/datafeeds/public_html/shop.z/datafeeds/completed/feed_mydata_data.csv"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/root/.sincedb_fffe878b6c3eb92e792a8cceb1f1f170", :path= ["/home/datafeeds/public_html/shop.z/datafeeds/completed/feed_mydata_data.csv"], :level=>:info}
Jun 06, 2015 1:35:08 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-datafeeds-server.server.com-5042-9782] version[1.5.1],   pid[5042], build[5e38401/2015-04-09T13:41:35Z]
Jun 06, 2015 1:35:08 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-datafeeds-server.server.com-5042-9782] initializing ...
Jun 06, 2015 1:35:08 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [logstash-datafeeds-server.server.com-5042-9782] loaded [], sites []
Jun 06, 2015 1:35:10 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-datafeeds-server.server.com-5042-9782] initialized
Jun 06, 2015 1:35:10 PM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-datafeeds-server.server.com-5042-9782] starting ...
Jun 06, 2015 1:35:10 PM org.elasticsearch.transport.TransportService doStart
INFO: [logstash-datafeeds-server.server.com-5042-9782] bound_address  {inet[/0:0:0:0:0:0:0:0:9302]}, publish_address {inet[/<ip>:9302]}
Jun 06, 2015 1:35:10 PM org.elasticsearch.discovery.DiscoveryService doStart
INFO: [logstash-datafeeds-server.server.com-5042-9782]  elasticsearch/dvhITquLQBef1eA0pJVuFw
Jun 06, 2015 1:35:13 PM org.elasticsearch.cluster.service.InternalClusterService$UpdateTask run
INFO: [logstash-datafeeds-server.server.com-5042-9782] detected_master [Devil-Slayer][GeNlEqTkQu-UlSOzS7mStw][datafeeds-server.server.com]  [inet[/<ip>:9301]], added {[Devil-Slayer][GeNlEqTkQu-UlSOzS7mStw] [datafeeds-server.server.com][inet[/<ip>:9301]],[logstash-datafeeds-  server.server.com-4935-9782][LtNpHWk6SMGJLDRCXGQKFg][datafeeds- server.server.com][inet[/<ip>:9300]]{data=false, client=true},},  reason: zen-disco-receive(from master [[Devil-Slayer][GeNlEqTkQu-UlSOzS7mStw] [datafeeds-server.server.com][inet[/<ip>:9301]]])
Jun 06, 2015 1:35:13 PM org.elasticsearch.node.internal.InternalNode start
 INFO: [logstash-datafeeds-server.server.com-5042-9782] started
 Automatic template management enabled {:manage_template=>"true", :level=>:info}
 Using mapping template {:template=>{"template"=>"logstash-*", "settings"=> {"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=> {"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=># <Java::JavaUtil::ArrayList:0x6fe472ee>, "properties"=>{"@version"=> {"type"=>"string", "index"=>"not_analyzed"}, "geoip"=>{"type"=>"object",  "dynamic"=>true, "properties"=>{"location"=>{"type"=>"geo_point"}}}}}}},  :level=>:info}
New Elasticsearch output {:cluster=>nil, :host=>["127.0.0.1"], :port=>"9300-9305", :embedded=>false, :protocol=>"node", :level=>:info}
Pipeline started {:level=>:info}
Logstash startup completed

如何獲取logstash填充elasticsearch？

我錯過了什么嗎？

Answer 1

當logstash新發現文件時，它從文件末尾開始。 如果要從頭開始，則需要添加start_position ：

file {
  path => "/home/datafeeds/public_html/myfeeds/datafeeds/completed/*.csv"
  start_position => "beginning"
}