Compact Framework 3.5 application crashing intermittently on CE7 / Motorola MC32N0

Question

We have a C# application using Compact Framework 3.5, running on a range on "mobile computers" running Windows CE 5.0 and Windows Mobile.

The application works well on these devices, including Motorola/Symbol MC3190, running Windows CE 5.0.

Now, this MC3190 has become end of life and is replaced by Motorola/Symbol MC3200 (also MC32N0). This device runs Windows CE 7.0 and has Compact Framework 3.5 pre-installed.

Our application is intermittently and mysteriously crashing on this device.

I can reproduce the problem somewhat consistently by opening and closing a lot of forms in our application. These actions aren't causing any network traffic. Sometimes I get a crash within 30 seconds, sometimes it takes 10 minutes of random actions.

At some point, the application will freeze completely. Pressing the Windows CE start button will draw the start menu over the frozen application, but pressing it again won't hide the start menu.

It looks like there is some kind of infinite loop going on within the toolkit, but I can't find out what it is. Here is a list of things I tried:

• I have made many changes to the source base (mostly changes to how forms are shown and disposed)

• I've tried to crash the application while attached to the VS debugger and then "break all", but this will just stall until I hard reboot the device.

• I've tried using the .NET Compact Framework Remote Performance monitor, which shows me interesting numbers about garbage collection etc, but no memory leak and no information to help me pinpoint the freeze.

• I've tried using the "CLR Profiler", but this errors out on my device with message "Unknown error while connecting: RemoteInstall: unsupported instruction set".

• I've tried contacting Motorola support and even sent them my source code, but they just demand to know "which API is causing the crash" (which I was hoping they could tell me)

Questions:

• Has anyone seen crashes like these on CF applications running on MC32N0?

• Does anyone have the CLR Profiler working on MC32N0?

• Does anyone have an idea how to get a kind of thread dump on a frozen CF application? (sort of like jstack for the JVM)

• Does anyone have any other idea how to go about debugging this?

Any help is much appreciated!

Update

The device has crashed on me a few times with a visual crash artifact, like so:

http://i.stack.imgur.com/HAxt8.jpg

Crash symptoms generally look like this:

• Happily use application, picking orders without a care in the world
• Suddenly, application no longer processes keyboard input
• Interestingly, at this point I am often able to focus a different TextBox and the cursor will blink in there for a little while
• After a few more interactions, the system completely locks up, and the Windows start button becomes unresponsive as well
• A few times - 3 of the about 100 of these crashes I've seen - the device will produce a crash pattern like in the image above

Note: Our supplier had delivered two MC3200 devices which both produce the crashes, so I find it hard to believe that it's a hardware issue.

Does anyone recognize these "symptoms"? Any thoughts?

Answer 1

We are experiencing the same "lock up" issue with our application on the MC32N0.

Zebra does have tools that will give you a better picture of what is going on. eMscript is one tool you can use that will run in the background and do some process and resource monitoring. RTLog (which is on the device) can give you some information but is only useful with drastic failures like uncaught exceptions. On our device we can kick off an RTLog dump with FUNC-F9, the device will give a beep if successful and the dump files will be in the root directory on the device.

Ok, after many hours of debugging/testing/rewriting code what we eventually found in an obscure post was that the lockup issue was caused by garbage collection. We seemed to lock up on form transitions so what we did prior to navigation was to add two lines of code in the Uninitialize() method of our base form.

        GC.Collect();
        GC.WaitForPendingFinalizers();

Fortunately all of our forms were derived from a base form so we only had to include it in the one location.

I can no longer find the post or I would reference it here. It was a Microsoft post which described the issue as random lockups when the finalizers were run but they also said it was corrected.

We've been running without lockup issues since.

Hope this helps someone...

Answer 2

The answer by Symbol/Motorola/Zebra (SMZ) is 'bad'. They should have tools to run a debug monitor in the background for intermitent error analysis.

What you can do is enbale Error Reporting and then see if there a kernel dump files created when the issue shows. But then you need help again by SMZ as the kdmp files have to be used together with the debug symbols of the Operating System Image/Firmware Build. The adresses and register values inside a kernel dump lead then to the function/API in the Build that has an issue.

If there is no kernel dump, then there is no 'native' exception that causes the dead-lock of your app. If available I would install CF35 on top of the existing one on the device. There are also PowerTools that enable extended debugging: https://www.microsoft.com/en-us/download/details.aspx?id=47266

There is also a number of updates to Windows Embedded Compact 7 (it seems no longer being called simply Windows CE7). These updates have to be adopted by the OEM (here Zebra Motorola Solutions). Inside https://www.microsoft.com/en-us/download/details.aspx?id=50356 is a html with known isssues called "Windows Embedded Compact 7 Monthly Update November 2015". Possibly you can avoid using the known issues.

For Error Reporting see https://msdn.microsoft.com/en-us/library/jj584896%28v=winembedded.70%29.aspx

Summary:

• Try to get a kernel logging tool by the OEM

• Try Error Reporting switched on

• Try PowerTools for WEC7

• Try updating the device's firmware

• Try an updated CF35 (if available)

• Avoid known faulty functions listed in the WEC7 update list. Except the OEM did include all updates in there OS build for the device (they can do or not at any time).

• Switch to another vendor/device

Answer 3

I've worked with devices like this (and the MC32N0) for many years and you've done everything correctly. The only thing that I normally also do is try and keep reducing a copy of your application in functionality until you end up with something much smaller that exhibits the fault. And then try and create a new test app that allows you to reproduce the issue easier.

Hopefully once you know what it is you can workaround or fix it. Or send it to Zebra. I'm currently porting a .Net C# app from the MC3190, MC2180 and am having issues with the MC32N0. I've noticed that calling GC.Collect(); GC.WaitForPendingFinalizers(); seems to be crashing the device. Its used after displaying big images and on older devices was needed to reclaim memory. Which is the exact opposite of what one of the other posts is reporting.

I've also had a problem with the EMDK sending out scanner cancelled events when the CE 6 devices don't. It was causing issues as my app mistook them as scan events and doing weird things. I'm now ignoring them which seems to be OK.

For your issue my guess is its memory related so look for big things like images or for GC code. Maybe check the program memory in the memory applet in control panel hasn't been accidentally set too small.