Property Based Testing with Hypothesis

June 25th 2019

Buenos Aires, Argentina

Who Am I?

Augusto Kielbowicz

SEP 2018

$\partial$ CIB Rates Derivatives

Sooo ... What is property based testing?

A simple example

"The sum of a list of integers is greater than the largest element in the list"
def test_sum_greater_than_max_small_numbers():
    xs = [1, 2, 3]
    assert sum( xs ) > max( xs )

def test_sum_greater_than_max_big_numbers():
    xs = [1000, 2000, 3000]
    assert sum( xs ) > max( xs )

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

import pytest

@pytest.mark.parameterize('xs',[
    [1, 2, 3], [1000, 2000, 3000]
])
def test_sum_greater_than_max(xs):
    assert sum( xs ) > max( xs )

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

from hypothesis import given
from hypothesis.strategies import lists, integers 

@given(lists(integers()))
def test_sum_greater_than_max(xs):
    assert sum( xs ) > max( xs )

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

from hypothesis import given
from hypothesis.strategies import lists, integers 

@given(lists(integers()))
def test_sum_greater_than_max(xs):
    assert sum( xs ) > max( xs )

Falsifying example: test_sum_greater_than_max(xs=[])
Traceback (most recent call last):
  ...
    assert sum( xs ) > max( xs )
ValueError: max() arg is an empty sequence

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

from hypothesis import given
from hypothesis.strategies import lists, integers 

@given(lists(integers(), min_size=1))
def test_sum_greater_than_max(xs):
    assert sum( xs ) > max( xs )

Traceback (most recent call last):
    ...
    assert sum( xs ) > max( xs )
AssertionError: assert 0 > 0
 +  where 0 = sum([0])
 +  and   0 = max([0])
----- Hypothesis --------
Falsifying example: test_sum_greater_than_max(xs=[0])

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

from hypothesis import given
from hypothesis.strategies import lists, integers 

@given(lists(integers(), min_size=1))
def test_sum_greater_than_max(xs):
    assert sum( xs ) >= max( xs )

Traceback (most recent call last):
   ...
    assert sum( xs ) >= max( xs )
AssertionError: assert -1 >= 0
 +  where -1 = sum([0, -1])
 +  and   0 = max([0, -1])
----- Hypothesis ----------
Falsifying example: test_sum_greater_than_max(xs=[0, -1])

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

from hypothesis import given
from hypothesis.strategies import lists, integers 

@given(lists(integers(min_value=0), min_size=1))
def test_sum_greater_than_max(xs):
    assert sum( xs ) >= max( xs )

.                                                                                                   [100%]

=========== 1 passed in 0.19 seconds ========

( example extracted from Zac Hatfield-Dodds PyConAu 2018 slides )

  • Define properties rather than specific scenarios
  • Give the test the input and check that properties hold
  • *Automatically generate random inputs

Hypothesis

https://hypothesis.works/

Generators

In [2]:
from hypothesis import strategies
# from hypothesis.strategies import ...

Numeric

>> strategies.integers().example() 
-20719
>> strategies.floats().example()
2.00001
>> strategies.decimals().example()
Decimal('NaN')
>> strategies.complex_numbers().example()
(5.835754834383092e+16-1.9j)

Collections

lists( integers() ), tuples( booleans() ), 
dictionaries( text(), floats() ), 
sets( characters() )

More complex data types

emails, functions, datetimes, timedeltas, nothing, just ....

Specific strategies

from_regex, from_types, sample_from, one_of ...

Compound strategies

builds, composite, defer, recursive ...

External libraries

from hypothesis.extra.numpy import arrays
from hypothesis.extra.pandas import data_frames, columns
from hypothesis.extra.django import from_model

Shrinking

@given(lists(integers(), min_size=1))
def test_sum_greater_than_max(xs):
    assert sum( xs ) >= max( xs )
...
Falsifying example: test_sum_greater_than_max(xs=[0, -1])

[-999,100,8] X
[-999,100]   X
[-999,0]     X
[0,0]        ✓ 
[-1,0]       X

''The core of properties is coming up with rules about a program that should always remain true.''

                                                                Fred Hebert

Common properties patterns

Smoke test

@given(lists(integers()))
def test_smoke_max( xs ):
    max(xs)

No assertions in the test!

Encode/Decode

assert text == json.loads(json.dumps(text))

Invariants

assert len( xs ) == len( reversed( xs ) )

Idempotence

assert set( xs ) == set( set( xs ) )

Test Oracle

assert awesome_new_function(x) == old_slow_function(x)

assert fancy_algorithm(x) == brute_force(x)

assert eat_cookies(x, threads=10) == eat_cookies(x, threads=1)

Rule-based stateful testing

from hypothesis.stateful import RuleBasedStateMachine
  • Preconditions
  • Actions
  • Postconditions
class DatabaseComparison(RuleBasedStateMachine):
    ...
    keys = Bundle('keys')
    values = Bundle('values')

    @rule(target=keys, k=st.binary())
    def add_key(self, k):
        ...
    @rule(target=values, v=st.binary())
    def add_value(self, v):
        ...
    @rule(k=keys, v=values)
    def save(self, k, v):
        ...
    @rule(k=keys, v=values)
    def delete(self, k, v):
        ...
    @rule(k=keys)
    def values_agree(self, k):
        ...

Example from Hypothesis Docs 

AssertionError: assert set() == {b''}

------------ Hypothesis ------------

state = DatabaseComparison()
var1 = state.add_key(k=b'')
var2 = state.add_value(v=var1)
state.save(k=var1, v=var2)
state.delete(k=var1, v=var2)
state.values_agree(k=var1)
state.teardown()

Example from Hypothesis Docs

On summary

Example base testing Property based testing
Focus on low level detail Focus on high level requirements
Tedious to test Properties define behaviour
Lots of repetition Randomly generated input
Painful to mantain Failure case minimisation

Some useful references of Property-based tests

Experiences with QuickCheck: Testing the Hard Stuff and Staying Sane

https://propertesting.com/

https://fsharpforfunandprofit.com/pbt/

Tomasz Kowal - Introduction to stateful property based testing - ElixirConf EU 2019 (video)

Escape from auto-manual testing with Hypothesis! - PyCon US 2019 (Tutorial)

Don't write tests, generate them!

                 John Huges, QuickCheck author.