Azure WebJob code not updated via deployment slot swap

Question

I have an Azure website with several deployment slots. I use Visual Studio to deploy to a Staging slot and then swap this into Production.

My Visual Studio solution contains one project for my web application and another for a continuous WebJob. I used (Right-click project) > Add > Existing Project as Azure WebJob to configure the WebJob to be deployed to the site along with the web application.

My problem is that the WebJob sometimes continues to run old code after a swap . Sometimes it runs the new code on one invocation and then switches back to the old code on the next invocation. I can't seem to figure out a rhyme or reason to this.

This morning, in order to try to find the cause of this, I did the following experiment:

1. Delete WebJob in Staging.
2. Delete WebJob in Production.
3. Deploy web app to Staging.
4. Verify that WebJob is running new code in Staging.
5. Swap Staging into Production.
6. Verify that WebJob no longer appears in Staging and that it now appears in Production.

What I found was that, after step 4, all the WebJob functions started afresh in Staging and were all running the new code. But after I swapped Staging into Production, there were no new invocations of the WebJob in Production, and when I manually invoked it, the old code ran!

What on earth is going on here? How can I totally clear out the old code and ensure the new WebApp code will run in Production?

I did see another thread saying that checking the Exclude files from the App_Data folder box in publishing settings could cause this issue, but I do not have it checked.

I feel like I'm taking crazy pills here. Any pointers would be sincerely appreciated.

---

Update 1: As described in my comment below, I have given up and switched to Azure Functions, but I just thought of something that might help anyone else who is encountering this. I recall that at least once or twice, I deployed my WebJob by manually uploading a zip file as opposed to deploying via Visual Studio. I'm starting to wonder if perhaps, under the covers, there is a different code path in use that could give rise to this behavior. Perhaps that will be a useful pointer for someone.

---

Update 2: I've figured out what happened. I'm describing this in the hopes that it might help someone in the future.

We have several deployment slots: Staging, Production, and a few others. One such slot is called Replica. Its purpose is to be a replica of Production (same code, same configuration settings). The idea is that, for those weird one-off problems that only repro in Production, you can upload a debug build to the Replica slot and attach a debugger without disrupting Production.

This is useful once in a while, but we didn't use it very frequently. The result was that the code running in the Replica slot, including our WebJob code, gradually got further and further out of date compared to what was running in Production. However, meanwhile, the Replica WebJobs kept running the old code, and since the Replica slot has all the same configuration settings as Production, it was operating on the same data and writing to the same logs as our Production WebJobs. So when I would go look at my Production WebJob logs, I would see output that I knew should no longer be there, and I couldn't figure out why. The rogue code was actually running in another slot that I had forgotten about.

So, mystery solved. This probably means we didn't need to cut everything over to Azure Functions. But I'm still glad that we did. I like them better. I like how we have the option to run them in the same app service as our site or a separate one, and the logging system is better. It feels more modern overall.

Anyway, I hope this helps someone out there.

Answer 1

Usually it is not possible run old code after swap successfully and restart your web app. So you can check the content under site->wwwroot in kudu: https://YourWebApp.scm.azurewebsites.net/DebugConsole to make sure your webjob is correct.