How do I retrieve a started Ignite instance when a website restart occurs in IIS?

Question

I use Apache Ignite .NET EntityFramework second level caching in my ASP.NET MVC 5 website. Everything work as expected on my development machine, but things get complicated when I try to publish the website using Web Deploy on the production IIS.

I always get a class org.apache.ignite.IgniteException: Ignite instance with this name has already been started: GridName when I publish or restart the website on IIS. The error is pretty clear, Ignite is already running.

The only way I found to get away this error is restarting the application pool of the website. However, I don't want to do this every time I publish the website in production (which happens several times a week).

I tried stopping the Ignite instance in the global.asax Dispose() or Application_End(), but the thing is that the AppDomain takes many seconds to stop. Ignite therefore has the time to try starting itself before it is stopped and causes the error mentioned above.

I also tried to call Ignition.TryGetIgnite() to retrieve the running instance instead of trying to start it, but it always return null. Looking at the source code of this function in the Apache Ignite Github repository, I see that the Ignition object simply keeps a static list of nodes in memory and perform operations on this list. Since the AppDomain restarted at this point, the list is empty but the Ignite node is still running in the JVM.

Is there a way to retrieve the Ignite instance or to stop it reliably, so I don't need to restart the application pool each time?

Answer 1

This is a known problem: AppDomain is stopped, but JVM keeps running (because process is not stopped), so Java part of Ignite nodes is still there.

Workaround is to stop all Ignite nodes with Ignition.StopAll in Application_End event like this in Global.asax.cs :

    protected void Application_Start()
    {
        ...

        using (new Mutex(true, "ignite_" + Process.GetCurrentProcess().Id))
        {
            var ignite = Ignition.TryGetIgnite() ?? Ignition.Start();
        }
    }

    protected void Application_End()
    {
        using (new Mutex(true, "ignite_" + Process.GetCurrentProcess().Id))
        {
            Ignition.StopAll(true);
        }
    }

Mutex is necessary here because old domain stopping overlaps with new domain starting. Process id is included in mutex name to ensure process-wide exclusive lock, but not machine-wide.

---

Another workaround , probably more robust and clean:

1) Stop all nodes in AppDomain.DomainUnload:

AppDomain.CurrentDomain.DomainUnload += (sender, args) => Ignition.StopAll(true);

2) Use different IgniteConfiguration.GridName every time:

Ignition.Start(new IgniteConfiguration { GridName = Guid.NewGuid().ToString() });

This way existing nodes won't prevent you from starting a new one. Eventually, old nodes will be stopped and cleaned up from memory.

Documentation: https://apacheignite-net.readme.io/docs/deployment#section-aspnet-deployment

Answer 2

Not exactly what I was looking for but I solved the problem for the moment.

What I do is in Application_End , I call Ignition.StopAll() as suggested by Pavel, but since Application_End sometimes takes a long time to be called, the new AppDomain loaded for the website has time to start and receive requests before Ignite is stopped in the other AppDomain.

To go around this, I start Ignite using the following code (which could be improved):

//Try to get already launched Ignite Instance
IIgnite ignite = Ignition.TryGetIgnite(igniteName);

while (ignite == null)
{
    try
    {
        //Try to start Ignite
        ignite = Ignition.Start(igniteConfig);
    }
    catch (Exception) //If failing to start Ignite, wait a bit for the previous AppDomain to stop the Ignite running instance...
    {
        HttpRequest request = null;
        try
        {
            request = HttpContext.Current.Request;
        }
        catch { }

        //Check if there is a request coming from the same machine, if yes, cancel it.
        if (request == null || !(request.IsLocal || request.UserHostName.EndsWith("myhostname.com"))
            Thread.Sleep(1000);
        else
            throw new HttpException(500, "Server error. Server rebooting.");
    }
}

//Use ignite variable here...

Note that in the code above, I put an additionnal check (which may not be needed by everyone), to check if there is a current request originating from the same machine. If you want to know why, read below, otherwise just take the code and remove the last if else part and simply keep the Thread.Sleep() .

If request originates from the same machine, an HttpException is thrown to cancel the request, otherwise, it is conserved. The reason I put this code is that if I publish the website on IIS, I don't want consumers to see an error, but I don't mind if they wait a bit for the previous AppDomain to stop. But if I receive a request from the same machine (or website), I want the request to abort, since it probably originates from code of the old AppDomain and will therefore delay the Application_End and Ignition.StopAll() from the old AppDomain.

Hopefully some future Apache Ignite .NET development could make the instance closing easier and more reliable than this workaround.