With Elastic Beanstalk, can I determine programmatically if I'm on the leader node?

Question

I have some housekeeping tasks within an Elastic Beanstalk Java application running on Tomcat, and I need to run them every so often. I want these tasks run only on the leader node (or, more correctly, on a single node, but the leader seems like an obvious choice).

I was looking at running cron jobs within Elastic Beanstalk, but it feels like this should be more straightforward than what I've come up with. Ideally, I'd like one of these two options within my web app:

1. Some way of testing within the current JRE whether or not this server is the leader node
2. Some some way to hit a specific URL (wget?) to trigger the task, but also restrict that URL to requests from localhost.

Suggestions?

Answer 1

It is not possible, by design ( leaders are only assigned during deployment , and not needed on other contexts). However, you can tweak and use the EC2 Metadata for this exact purpose.

Here's an working example about how to achieve this result ( original source ). Once you call getLeader, it will find - or assign - an instance to be set as a leader:

package br.com.ingenieux.resource;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

import org.apache.commons.io.IOUtils;

import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.model.CreateTagsRequest;
import com.amazonaws.services.ec2.model.DeleteTagsRequest;
import com.amazonaws.services.ec2.model.DescribeInstancesRequest;
import com.amazonaws.services.ec2.model.Filter;
import com.amazonaws.services.ec2.model.Instance;
import com.amazonaws.services.ec2.model.Reservation;
import com.amazonaws.services.ec2.model.Tag;
import com.amazonaws.services.elasticbeanstalk.AWSElasticBeanstalk;
import com.amazonaws.services.elasticbeanstalk.model.DescribeEnvironmentsRequest;

@Path("/admin/leader")
public class LeaderResource extends BaseResource {
    @Inject
    AmazonEC2 amazonEC2;

    @Inject
    AWSElasticBeanstalk elasticBeanstalk;

    @GET
    public String getLeader() throws Exception {
        /*
         * Avoid running if we're not in AWS after all
         */
        try {
            IOUtils.toString(new URL(
                    "http://169.254.169.254/latest/meta-data/instance-id")
                    .openStream());
        } catch (Exception exc) {
            return "i-FFFFFFFF/localhost";
        }

        String environmentName = getMyEnvironmentName();

        List<Instance> environmentInstances = getInstances(
                "tag:elasticbeanstalk:environment-name", environmentName,
                "tag:leader", "true");

        if (environmentInstances.isEmpty()) {
            environmentInstances = getInstances(
                    "tag:elasticbeanstalk:environment-name", environmentName);

            Collections.shuffle(environmentInstances);

            if (environmentInstances.size() > 1)
                environmentInstances.removeAll(environmentInstances.subList(1,
                        environmentInstances.size()));

            amazonEC2.createTags(new CreateTagsRequest().withResources(
                    environmentInstances.get(0).getInstanceId()).withTags(
                    new Tag("leader", "true")));
        } else if (environmentInstances.size() > 1) {
            DeleteTagsRequest deleteTagsRequest = new DeleteTagsRequest().withTags(new Tag().withKey("leader").withValue("true"));

            for (Instance i : environmentInstances.subList(1,
                        environmentInstances.size())) {
                deleteTagsRequest.getResources().add(i.getInstanceId());
            }

            amazonEC2.deleteTags(deleteTagsRequest);
        }

        return environmentInstances.get(0).getInstanceId() + "/" + environmentInstances.get(0).getPublicIpAddress();
    }

    @GET
    @Produces("text/plain")
    @Path("am-i-a-leader")
    public boolean isLeader() {
        /*
         * Avoid running if we're not in AWS after all
         */
        String myInstanceId = null;
        String environmentName = null;
        try {
            myInstanceId = IOUtils.toString(new URL(
                    "http://169.254.169.254/latest/meta-data/instance-id")
                    .openStream());

            environmentName = getMyEnvironmentName();
        } catch (Exception exc) {
            return false;
        }

        List<Instance> environmentInstances = getInstances(
                "tag:elasticbeanstalk:environment-name", environmentName,
                "tag:leader", "true", "instance-id", myInstanceId);

        return (1 == environmentInstances.size());
    }

    protected String getMyEnvironmentHost(String environmentName) {
        return elasticBeanstalk
                .describeEnvironments(
                        new DescribeEnvironmentsRequest()
                                .withEnvironmentNames(environmentName))
                .getEnvironments().get(0).getCNAME();
    }

    private String getMyEnvironmentName() throws IOException,
            MalformedURLException {
        String instanceId = IOUtils.toString(new URL(
                "http://169.254.169.254/latest/meta-data/instance-id"));

        /*
         * Grab the current environment name
         */
        DescribeInstancesRequest request = new DescribeInstancesRequest()
                .withInstanceIds(instanceId)
                .withFilters(
                        new Filter("instance-state-name").withValues("running"));

        for (Reservation r : amazonEC2.describeInstances(request)
                .getReservations()) {
            for (Instance i : r.getInstances()) {
                for (Tag t : i.getTags()) {
                    if ("elasticbeanstalk:environment-name".equals(t.getKey())) {
                        return t.getValue();
                    }
                }
            }
        }

        return null;
    }

    public List<Instance> getInstances(String... args) {
        Collection<Filter> filters = new ArrayList<Filter>();

        filters.add(new Filter("instance-state-name").withValues("running"));

        for (int i = 0; i < args.length; i += 2) {
            String key = args[i];
            String value = args[1 + i];

            filters.add(new Filter(key).withValues(value));
        }

        DescribeInstancesRequest req = new DescribeInstancesRequest()
                .withFilters(filters);
        List<Instance> result = new ArrayList<Instance>();

        for (Reservation r : amazonEC2.describeInstances(req).getReservations())
            result.addAll(r.getInstances());

        return result;
    }
}

Answer 2

You can keep a secret URL (a long URL is un-guessable, almost as safe as a password), hit this URL from somewhere. On this you can execute the task.

One problem however is that if the task takes too long, then during that time your server capacity will be limited. Another approach would be for the URL hit to post a message to the AWS SQS. The another EC2 can have a code which waits on SQS and execute the task. You can also look into http://aws.amazon.com/swf/

Answer 3

Another approach if you're running on the Linux-type EC2 instance:

1. Write a shell script that does (or triggers) your periodic task
2. Leveraging the .ebextensions feature to customize your Elastic Beanstalk instance , create a container command that specifies the parameter leader_only: true -- this command will only run on an instance that is designated the leader in your Auto Scaling group
3. Have your container command copy your shell script into /etc/cron.hourly (or daily or whatever).

The result will be that your "leader" EC2 instance will have a cron job running hourly (or daily or whatever) to do your periodic task and the other instances in your Auto Scaling group will not.