更新多个EC2实例上的日志级别

Question

My app runs on multiple EC2 instances to ensure high availability. The default log level is INFO for the app. But sometimes for debugging purposes, I want to update the log level to DEBUG . The request to update the log level passes through the ElasticLoadBalancer which delegates the request to any one of the multiple EC2 instances. The log level for the app running on that instance is updated but apps on the other instances will still log at level INFO . I want all the apps to log at DEBUG level.

I am using Spring, SLF4J and Logback.

If I somehow make the log level information to be centralized, and the request will update the level on the centralized location, but still someone has to intimate apps on all instances about the change as app will never be requesting the log level.

Answer 1

If you want an AWS solution you can utilize sns.

Once your app gets instantiated, register its endpoint (using it's private ip) to an sns topic for a http notification. Thus instead of changing your LOG level through the load balancer you can issue a sns message and the message shall be sent to the endpoints registered.

Keep in mind to deregister the http endpoint from sns,once the app gets terminated.

Answer 2

You might want to take a look at Zookeeper :

ZooKeeper is a centralized service for maintaining configuration information , naming, providing distributed synchronization, and providing group services.

It's quite easy to setup and start small. The app running on your EC2 nodes just needs to implement a "listener/watcher" interface. This will notify your app when some configuration changed (eg. you decided you want to set the global log level to DEBUG ).

Based on this configuration-change, all of your nodes will update the local log-level without you having to come up with all kinds of ELB-bypassing manual REST-calls to tell each node to update - exactly what zookeeper is solving:

Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

---

When this works for you, you can add additional configuration to the zookeeper if needed, limiting the amount of configuration you need to package in the deployed apps or copied alongside them.

Answer 3

Amazons Remote Management (Run Command) allows you to run commands on your instances. You just need a simple script to change the loglevel.

But it is not easy to set it up and set grant all the needed IAM rights :

Answer 4

There are tags for an instance. Some tags exist by default and you can create your own tags. So, if we add a tag which identifies all those instances on which app is currently running, we can very easily fetch all those instances' IP addresses.

DescribeInstancesRequest request = new DescribeInstancesRequest();

Filter filter1 = new Filter("tag:Environment", Collections.singletonList("Sandbox"));
Filter filter2 = new Filter("tag:Application", Collections.singletonList("xxxxx"));
Filter filter3 = new Filter("tag:Platform", Collections.singletonList("xxxx"));

InstanceProfileCredentialsProvider mInstanceProfileCredentialsProvider =
                new InstanceProfileCredentialsProvider();
AWSCredentials credentials = mInstanceProfileCredentialsProvider.getCredentials();

AmazonEC2 ec2Client = new AmazonEC2Client(credentials);
List<String> privateIps = new ArrayList<>();

ec2Client.describeInstances(request.withFilters(filter1, filter2, filter3)).getReservations().forEach(
                reservation -> reservation
                        .getInstances()
                        .forEach(instance -> privateIps.add(instance.getPrivateIpAddress())));

for (String privateIp : privateIps) {
     hitTheInstance(privateIp);
}

Here, I have used 3 tags to filter out the instances.