将weburl的内容索引到elasticsearch / kibana中

Question

I have scrapped 500+ links/sublinks of a website using beautiful soup+python,now I am looking forward to index all the contents/text of this url in elasticsearch,is there any tool that can help me indexing directly with elastic search/kibana stack.

please help me with pointers,i tried searching on google and found logstash,but seems it works for single url.

Answer 1

For reference on Logstash please see: https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

Otherwise, an example of putting your crawler output into a file, with a line per url, you could have the logstash config below, in this example, logstash will read one line as being a message and send it to the elastic servers on host1 and host2.

input {
    file {
        path => "/an/absolute/path" #The path has to be absolute
        start_position => beginning
     }
}

output {
    elasticsearch{
        hosts => ["host1:port1", "host2:port2"] #most of the time the host being the DNS name (localhost as the most basic one), the port is 9200
        index => "my_crawler_urls"
        workers => 4 #to define depending on your available resources/expected performance
    }
}

Now of course, you might want to do some filter, post-treatment of the output of your crawler, and for that Logstash gives you the possibility with codecs and/or filters