When Cloudwatch Logs data is sent into kinesis data stream, what is its encoding format

Question

I'm trying to write a Go program, to download data from aws kinesis data stream. I read that kinesis data stream encode the data with base64 , so I need first decode with base64 . However, I can't figure out what encoding was used on the data as it is passed, from cloudwatch logs to kinesis data stream.

I'm trying the different decoding method but none works. My unprocessed byte array downloaded from kinesis data stream is as the following:

[31 139 8 0 0 0 0 0 0 0 53 206 65 11 130 64 16 134 225 191 178 204 89 130 178 34 246 22 97 30 178 130 12 58 68 196 166 147 14 233 174 236 140 69 68 255 61 204 58 190 204 7 243 188 160 70 102 83 224 254 217 32 104 88 108 55 251 221 54 57 175 163 52 157 199 17 4 224 30 22 125 119 169 92 155 63 140 100 101 226 10 134 0 42 87 196 222 181 13 104 232 43 21 143 166 238 147 219 11 103 158 26 33 103 151 84 9 122 6 125 60 125 119 209 29 173 116 249 2 202 251 185 80 141 44 166 110 64 15 167 227 201 48 28 79 166 225 108 20 6 127 94 7 56 36 234 199 83 63 158 86 139 18 179 27 217 66 149 104 42 41 149 187 170 28 89 200 154 238 179 90 145 69 38 86 252 165 13 224 125 122 127 0 234 141 66 79 242 0 0 0]

Can someone give me some tips how to process this piece of data?

Answer 1

You can use a subscription filter with Kinesis, Lambda, or Kinesis Data Firehose. Logs that are sent to a receiving service through a subscription filter are base64 encoded and compressed with the gzip format.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html