在Ubuntu上作为服务运行的logstash-input-heroku（logstash 5.2.1）

Question

I'm trying to run logstash on an aws ec2 instance running ubuntu 16.04 using systemd. I've installed heroku toolbelt on the machine. Running the pipeline normally (via bin/logstash.bat) works fine and events are ingested (however after a few minutes a get a "Request timed out" error and the pipeline stops, which is a separate question).

But when I try to run the service on systemd I get errors, not sure if the two types of errors are related. The first is an SSL error:

Error: no cipher match (OpenSSL::SSL::SSLError)

[2017-02-15T13:08:44,037][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin. Plugin: "xxxxxx", codec=>"^%{TIMESTAMP_ISO8601} %{WORD}\\[\\w+(\\.\\d+)?\\]:(\\s{3,}| \\})", what=>"previous", id=>"032c3b317ae49982945ec7e8fbf11224be98f237-3", enable_metric=>true, negate=>false, charset=>"UTF-8", multiline_tag=>"multiline", max_lines=>500, max_bytes=>10485760>, id=>"032c3b317ae49982945ec7e8fbf11224be98f237-4", enable_metric=>true>

The second is that the heroku toolbelt seems to be prompting for credentials:

Feb 15 13:08:43 ip-10-0-1-216 logstash[4402]: Enter your Heroku credentials.

Feb 15 13:08:43 ip-10-0-1-216 logstash[4402]: Email: Password (typing will be hidden):

My logstash configuration:

input {
    heroku {
        app => "xxx-1"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})"
            what => "previous"
        }
    }
    heroku {
        app => "xxx-2"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})"
            what => "previous"
        }
    }
    heroku {
        app => "xxx-3"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})"
            what => "previous"
        }
    }
    heroku {
        app => "xxx-4"
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})"
            what => "previous"
        }
    }
}

filter {
  grok {
    break_on_match => true
    patterns_dir => ["./grok_patterns"]
    match => { "message" => [
        "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku_source}\[%{DYNO:dyno}\]: %{LEVEL:level}: HTTP %{OPT_NOT_SPACE_COMMA:organization}, %{OPT_NOT_COMMA:user}, %{OPT_NOT_COMMA:device}, %{WORD:method} %{ENDPOINT:endpoint}%{QUERY:query} \[%{INT:responseCode:int}\].*? \(p%{INT:nodeProcess:int}\) \(%{INT:responseTime:int}ms\).*$",
        "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku}\[%{WORD:component}\]: at=\w+ method=%{WORD:method} path=\"%{ENDPOINT:endpoint}\??%{QUERY:query}\" .*?fwd=\"%{IP:site_ip}\" dyno=%{DYNO:dyno} .*?service=%{INT:responseTime:int}ms status=%{INT:responseCode:int} bytes=%{INT:sizeBytes:int}.*?$",
        "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku_source}\[%{DYNO:dyno}\]: (?<data>.*)"
    ] }
    add_field => { "endpoint_template" => "%{endpoint}" }
  }
  mutate {
    gsub => ["endpoint_template", "[0-9a-f]{24}", "ID"]
    add_field => { "type" => "heroku" }
  }
  if ![heroku_source] {
    geoip {
        source => "site_ip"
    }
    mutate {
        add_field => { "heroku_source" => "heroku" }
    }
  }
}

output {
    elasticsearch {
        hosts => [ "aws-es-endpoint:443" ]
        ssl => true        
    }
}

(I'm sure it could be improved)

I've tried running the service as root but the result is the same. Just to clarify, this works:

/usr/share/logstash/bin/logstash --path.settings /etc/logstash/

While this does not:

sudo systemctl start logstash

This is a clean install of logstash 5.2.1 following the procedures on elastic . Systemd is also run according to their procedures , so that it executes the same command as I execute manually. cat logstash.service output:

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

(result is the same when I comment out the user and group above)

Answer 1

Just for the record, turns out the problem was that systemd was not seeing the heroku credentials. I asked a similar question on AskUbuntu . The solution was to add the ubuntu home directory to the logstash service so it would access the credentials. Accomplished by editing the logstash.service file in /etc/systemd/system and setting Environment="Home=/home/ubuntu" there.

Example of final logstash.service file:

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=root
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
Environment="HOME=/home/ubuntu"
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target