Automatically detect non-deterministic behaviour in Python

Question

This may be impossible, but I am just wondering if there are any tools to help detect non-deterministic behaviour when I run a Python script. Some fancy options in a debugger perhaps? I guess I am imagining that theoretically it might be possible to compare the stack instruction-by-instruction or something between two subsequent runs of the same code, and thus pick out where any divergence begins.

I realise that a lot is going on under the hood though, so that this might be far too difficult to ask of a debugger or any tool...

Essentially my problem is that I have a test failing occasionally, almost certainly because somewhere the code relies accidentally on the ordering of output from iterating over a dictionary, or some such thing where the ordering isn't actually guaranteed. I'd just like a tool to help me locate the source of this sort of problem :).

The following question is similar, but there was not much suggestion of how to really deal with this in an automated or general way: Testing for non-deterministic behavior of python function

Answer 1

I'm not aware of a way to do this automatically, but what I would recommend doing is starting a debugger when a test fails, then running it automatically (overnight?) until you get a failure. You can then examine the variables and see if anything stands out. If you're using pytest, running with the --pdb flag will start a debugger on failure.

You might also consider using Hypothesis to run generative test cases.

You might also consider running the tests over and over, collecting the output of each run (success or failure). When you have a representative sample, compare the two, particularly the ordering of tests that were run!