Index documentation url with ELK stack

Question

I want to load below documentation using logstash to do indexing it with elastic search:

https://www.ibm.com/support/knowledgecenter/SSEPGG_10.1.0/com.ibm.db2.luw.common.doc/doc/t0020018.html

Currently,I am able to index all the logs with elasticsearch,but I am not sure how to index documentation with a url.

Is this scenario is valid with logstash,if yes,please help me on this.

Answer 1

You could give it a go by using the http poller plugin which was released recently, within your logstash config itself. The input could look something like this:

input {
  http_poller {
    urls => {
      myurl=> "https://www.ibm.com/support/knowledgecenter/SSEPGG_10.1.0/com.ibm.db2.luw.common.doc/doc/t0020018.html"
      }
    }
    request_timeout => 60
    interval => 60
    codec => "json"  <--- to decode the results to json
  }
}

The above is just a sample so that you could reproduce. Hope this helps!