Property Based Testing con Hypothesis

28 de Agosto de 2019

Buenos Aires, Argentina

Y vo', quién sos?

Sasha

Entonces ... Qué es testing basado en propiedades?

Un ejemplo bien simple

"La suma de una lista de numeros enteros es mayor al elemento más grande de la lista"

def test_suma_mas_grande_que_maximo_con_numeros_pequenios():
    xs = [1, 2, 3]
    assert sum( xs ) > max( xs )
    
def test_suma_mas_grande_que_maximo_con_numeros_grandes():
    xs = [1000, 2000, 3000]
    assert sum( xs ) > max( xs )

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

import pytest

@pytest.mark.parameterize('xs',[
    [1, 2, 3], [1000, 2000, 3000]
])
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) > max( xs )

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

@given(lists(integers()))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) > max( xs )

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

@given(lists(integers()))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) > max( xs )

Falsifying example: test_suma_mas_grande_que_maximo(xs=[])
Traceback (most recent call last):
  ...
    assert sum( xs ) > max( xs )
ValueError: max() arg is an empty sequence

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

@given(lists(integers(), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) > max( xs )

Traceback (most recent call last):
    ...
    assert sum( xs ) > max( xs )
AssertionError: assert 0 > 0
 +  where 0 = sum([0])
 +  and   0 = max([0])
----- Hypothesis --------
Falsifying example: test_suma_mas_grande_que_maximo(xs=[0])

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

@given(lists(integers(), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) >= max( xs )

Traceback (most recent call last):
   ...
    assert sum( xs ) >= max( xs )
AssertionError: assert -1 >= 0
 +  where -1 = sum([0, -1])
 +  and   0 = max([0, -1])
----- Hypothesis ----------
Falsifying example: test_suma_mas_grande_que_maximo(xs=[0, -1])

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

@given(lists(integers(min_value=0), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) >= max( xs )

.                                             [100%]

=========== 1 passed in 0.19 seconds ========

( ejemplo extraído de diapositivas de Zac Hatfield-Dodds PyConAu 2018 )

• Definir propiedades en vez de escenarios específicos

• Dale un input al test y verifica que las propiedades se preservan

• *Generar automáticamente inputs aleatorios

Hypothesis

https://hypothesis.works/

@given(lists(integers(min_value=0), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) >= max( xs )

from hypothesis import given
from hypothesis.strategies import lists, integers

@given(lists(integers(min_value=0), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) >= max( xs )

Generadores ( Generators )

from hypothesis import strategies
# from hypothesis.strategies import ...

Numéricos

>> strategies.integers().example() 
-20719
>> strategies.floats().example()
2.00001
>> strategies.decimals().example()
Decimal('NaN')
>> strategies.complex_numbers().example()
(5.835754834383092e+16-1.9j)

Colecciones

lists( integers() ), tuples( booleans() ), 
dictionaries( text(), floats() ), 
sets( characters() )

Tipos de datos más complejos

emails, functions, datetimes, timedeltas, nothing, just ....

Estrategias específicas

from_regex, from_types, sample_from, one_of ...

Estrategias compuestas

builds, composite, defer, recursive ...

Librerías externas

from hypothesis.extra.numpy import arrays
from hypothesis.extra.pandas import data_frames, columns
from hypothesis.extra.django import from_model

Reducción ( Shrinking )

@given(lists(integers(), min_size=1))
def test_suma_mas_grande_que_maximo(xs):
    assert sum( xs ) >= max( xs )
...
Falsifying example: test_suma_mas_grande_que_maximo(xs=[0, -1])

[-999,100,8] X
[-999,100]   X
[-999,0]     X
[0,0]        ✓ 
[-1,0]       X

"El núcleo de las propiedades es obtener reglas sobre un programa, que siempre tienen que mantenerse verdaderas."

''The core of properties is coming up with rules about a program that should always remain true.''

Fred Hebert

Patrones comunes sobre propiedades

Smoke test

@given(lists(integers()))
def test_smoke_max( xs ):
    max(xs)

No hay asserts en el test!

Codificar/Decodificar (Encode/Decode)

assert text == json.loads(json.dumps(text))

Invariantes

assert len( xs ) == len( reversed( xs ) )

Idempotencia

assert set( xs ) == set( set( xs ) )

Oráculo ( Test Oracle )

assert nueva_funcion_increible(x) == vieja_funcion_lenta(x)

assert algoritmo_sexy(x) == fuerza_bruta(x)

assert descarga_torrent(x, threads=10) == descarga_torrent(x, threads=1)

Testeo de estado basado en reglas (Rule-based stateful testing)

from hypothesis.stateful import RuleBasedStateMachine

• Pre-condiciones
• Acciones
• Post-condiciones

class DatabaseComparison(RuleBasedStateMachine):
    ...
    keys = Bundle('keys')
    values = Bundle('values')

    @rule(target=keys, k=st.binary())
    def add_key(self, k):
        ...
    @rule(target=values, v=st.binary())
    def add_value(self, v):
        ...
    @rule(k=keys, v=values)
    def save(self, k, v):
        ...
    @rule(k=keys, v=values)
    def delete(self, k, v):
        ...
    @rule(k=keys)
    def values_agree(self, k):
        ...

Example from Hypothesis Docs

AssertionError: assert set() == {b''}

------------ Hypothesis ------------

state = DatabaseComparison()
var1 = state.add_key(k=b'')
var2 = state.add_value(v=var1)
state.save(k=var1, v=var2)
state.delete(k=var1, v=var2)
state.values_agree(k=var1)
state.teardown()

Example from Hypothesis Docs

En resumen

Testing basado en ejemplos	Testing basado en propiedades
Foco en detalles de bajo nivel	Foco en requerimientos de alto nivel
Tedioso de testear	Las propiedades definen comportamiento
Mucha repetición	Input aleatorio generado automáticamente
Molesto de mantener	Minimiza los casos de falla

Un par de referencias útiles para aprender sobre Property-based tests

Experiences with QuickCheck: Testing the Hard Stuff and Staying Sane

https://propertesting.com/

https://fsharpforfunandprofit.com/pbt/

Tomasz Kowal - Introduction to stateful property based testing - ElixirConf EU 2019 (video)

Escape from auto-manual testing with Hypothesis! - PyCon US 2019 (Tutorial)

"No escriban tests, genérenlos"

"Don't write tests, generate them!"

                 John Huges, QuickCheck author.