I have some containers deployed in ECS Fargate, that send the logs to Cloudwatch logs. Now, I want to send also the logs to a custom Elasticsearch instance (not Amazon Elasticsearch Service). I have read some info about firelens , but it is not clear for me if the logs will be sent also to Cloudwatch logs. Also, all the documentation seems to refer to Amazon Elasticsearch Service, not an own Elasticsearch instance. Do you have any recommendation/snippet of how to do that? Additionally, If I wanted to send the logs to a third, or fourth destination, what would be the approach to do this? Of course, in the cheapest possible way, avoiding Firehose or Lambda tricks.
Adding an answear because I don't have the reputation points to comment. Anyway, I suggest you review this AWS Blog post on custom firelens configuration to send ecs fargate logs to multiple destinations.
First you'll need to build a custom firelens image and make it available on ECR:
FROM amazon/aws-for-fluent-bit:stable
ADD logDestinations.conf /logDestinations.conf
The custom image should have a.conf file with your log destinations... Here's an example:
[OUTPUT]
Name cloudwatch
Match YourContainerName*
region us-east-1
log_group_name app-container
log_stream_prefix fluent-bit
auto_create_group true
[OUTPUT]
Name datadog
Match *
Host http-intake.logs.datadoghq.com
TLS on
compress gzip
apikey <DATADOG_API_KEY>
dd_service <APPLICATION_SERVICE>
dd_source <SOURCE>
dd_message_key log
dd_tags env:dev,<TAG_KEY>:<TAG_VALUE>
You will add Firelens as a new container in your TaskDefinition:
ContainerDefinitions:
... [Other Containers in your task. E.g.: Your application, datadog agent, etc]
- Name: log-router
Essential: true
Image: amazon/customImage:latest
Cpu: 100
Memory: 256
FirelensConfiguration:
Type: fluentbit
Options:
enable-ecs-log-metadata: true
config-file-type: file
config-file-value: "/logDestinations.conf"
I tested this configuration, it works well. The only thing I had to worry was setting up my image in a separate pipeline, since it's not possible to get the logDestinations.conf file from S3 when using ECS on Fargate.
Reference: https://aws.amazon.com/pt/premiumsupport/knowledge-center/ecs-container-log-destinations-fargate/
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.