logs to splunk are getting truncated

Question

I am using fluentd to forward my Kubernetes pod logs to splunk but in splunk I am not able to see full length of pod log as they getting truncated. For example we have a single line log length of 74286 chars, but splunk shows only 16385 chars. what can I do to overcome this issue ?

This way I have configured in fluentd configmap.

<match **>
      @id splunk
      @type splunk-hec
      @log_level info
      server "#{ENV['FLUENT_SPLUNK_HOST']}"
      protocol https
      verify false
      host "#{ENV['CLUSTER_NAME']}_#{ENV['NODE_NAME']}"
      token "#{ENV['FLUENT_SPLUNK_TOKEN']}"
      index "#{ENV['SPLUNK_INDEX']}"
      buffer_type memory
      buffer_queue_limit 256
      buffer_chunk_limit 8m
      batch_size_limit 8000000
      flush_interval 1s
    </match>

Answer 1

By default, Splunk is supposed to truncate at 10,000 characters. You can change that in your props.conf file.

[mysourcetype]
TRUNCATE = 75000

This would be in addition to the rest of the "magic" 6 settings: TIME_PREFIX , TIME_FORMAT , MAX_TIMESTAMP_LOOKAHEAD , SHOULD_LINEMERGE , and LINE_BREAKER .