Hypothesis equivalent of QuickCheck frequency generator?

Question

As a learning project I am translating some Haskell code (which I'm unfamiliar with) into Python (which I know well)...

The Haskell library I'm translating has tests which make use of QuickCheck property-based testing. On the Python side I am using Hypothesis as the property-based testing library.

The Haskell tests make use of a helper function which looks like this:

mkIndent' :: String -> Int -> Gen String
mkIndent' val size = concat <$> sequence [indent, sym, trailing]
  where
    whitespace_char = elements " \t"
    trailing = listOf whitespace_char
    indent = frequency [(5, vectorOf size whitespace_char), (1, listOf whitespace_char)]
    sym = return val

My question is specifically about the frequency generator in this helper. http://hackage.haskell.org/package/QuickCheck-2.12.6.1/docs/Test-QuickCheck-Gen.html#v:frequency

I understand it to mean that most of the time it will return vectorOf whitespace_char with the expected size , but 1 in 5 times it will return listOf whitespace_char which could be any length including zero.

In the context of the library, an indent which does not respect the size would model bad input data for the function under test. So I see the point of occasionally producing such an input.

What I currently don't understand is why the 5:1 ratio in favour of valid inputs? I would have expected the property-based test framework to generate various valid and invalid inputs. For now I assume that this is sort of like an optimisation, so that it doesn't spend most of its time generating invalid examples?

The second part of my question is how to translate this into Hypothesis. AFAICT Hypothesis does not have any equivalent of the frequency generator.

I am wondering whether I should attempt to build a frequency strategy myself from existing Hypothesis strategies, or if the idiom itself is not worth translating and I should just let the framework generate valid & invalid examples alike?

What I have currently is:

from hypothesis import strategies as st

@st.composite
def make_indent_(draw, val, size):
    """
    Indent `val` by `size` using either space or tab.
    Will sometimes randomly ignore `size` param.
    """
    whitespace_char = st.text(' \t', min_size=1, max_size=1)
    trailing = draw(st.text(draw(whitespace_char)))
    indent = draw(st.one_of(
        st.text(draw(whitespace_char), min_size=size, max_size=size),
        st.text(draw(whitespace_char)),
    ))
    return ''.join([indent, val, trailing])

If I generate a few examples in a shell this seems to be doing exactly what I think it should.

But this is my first use of Hypothesis or property-based testing and I am wondering if I am losing something vital by replacing the frequency distribution with a simple one_of ?

Answer 1

As far as I can see, you've correctly understood the purpose of using frequency here. It is used to allow the occasional mis-sized indent instead of either (1) only generating correctly sized indents which would never test bad indent sizes; or (2) generating randomly sized indents which would test bad indents over and over again but only generate a fraction of cases with good indents to test other aspects of the code.

Now, the 5:1 ratio of good to (potentially) bad indent sizes is probably quite arbitrary, and it's hard to know if 1:1 or 10:1 would have been better choices without seeing the details of what's being tested.

Luckily though, with respect to porting this to hypothesis , the answer to Have a Strategy that does not uniformly choose between different strategies includes a deleted comment:

Hypothesis doesn't actually support user-specific probabilities - we start with a uniform distribution, but bias it based on coverage from observed inputs. [...] – Zac Hatfield-Dodds Apr 15 '18 at 3:43

This suggests that the "hypothesis" package automatically adjusts weights when using one_of to increase coverage, meaning that it may automatically up-weight the case with correct size in your make_indent_ implementation, making it a sort of automatic version of frequency .