We have more to go yet in our short series on prioritization, but I wanted to take a short interlude to share a few notes on testing with you. This is a collection of test strategies that are incredibly helpful in working with existing and even new Django sites.
 

Boundary Value Testing and Snapshot Testing

I learned recently that some test strategies we’d been using actually had names. The first is boundary value testing[0]. This is a strategy for testing a function - we’ll use function here as the representative but clearly you can test more than this - by testing it at the boundaries of expected values. Another way of putting it, by testing for unexpected values in the function range at known changes in the function domain.

Okay, let’s consider a Django example.

You want to test a Django form. It validates that someone is at least 18 years of age by checking their birthday. You want to ensure that the form is valid when their birthdate indicates that they are 18 or older, and that the form is invalid when they are less than 18. Need you check every single date? No. You need only check the values where the expected outcome changes and ensure those values are correct.

There are many cases where this is the simplest and preferred method, e.g. a form with only a single field. In many such cases though my choice instead of boundary value testing would be property testing, allowing for a full range of values, even multiple values in combination, and testing the results of their multiplicative combinations.[1]

The other term is snapshot testing[2][3]. Some readers may be familiar with snapshot tests from testing React components. I was not! It’s a method of regression testing in which the [rendered] output from a function is captured and used to test against subsequent runs. 

It’s usually used for UI testing, in the strict sense of testing the interface, but given that the UI is rendering deeper computation it can also be used for checking for regressions in the entire application stack. This makes it a useful tool when approaching an app with minimal to no tests, but also for serving as a critical warning for changes in complicated systems.
 

The test pyramid

The test pyramid[4] is now a pretty well established concept. It divides tests into three categories, (1) unit, (2) integration, and (3) UI, in a pyramid from unit at the bottom to UI at the top. Unit tests are faster and cheaper, UI tests are slower and more costly, and integration tests in between.

But this isn’t universally accepted. There are software developers, including test practitioners, who suggest that nowadays unit tests are not just cheap but worthless. Developers like to focus on unit tests because they’re easy to write and check off the box for testing. Moreover TDD is near impossible to follow without unit tests. And yet…

So the new theory is that if all you really need are good integration tests - the more integrated the better - and ignore unit tests.

I’ll be honest, if a project has to have only unit tests or only integration tests, it’s better to have only the latter. But there’s a small problem here if unit tests are not included or considered completely disposable. 

Unit tests - and lower level integration tests, because it’s not always clear where the boundary is - make it easier to test how changes to small parts fit together. And _this _is extremely useful when making changes to legacy or any kind of existing project.

The big integration tests will tell you that _the thing is broken, something you’ve changed is breaking the thing! _But it will give you little insight into _how _it’s broken. At this point you can make changes and keep running tests to see if it works, or you can rely on unit tests to guide the individual changes at a lower level before needing to lean on the big integration tests.

If you imagine assembling various pipes to connect to a faucet, the final test is that water comes out the end when it’s supposed to, it doesn’t when it’s not supposed to, and it doesn’t leak anywhere in the middle. Now if you do find that it’s not working, what’s your course of action? Keep making changes until the end result is correct? Or can you can you make small intermediate changes and then verify that these do in fact work, even if its only in their narrow scope?

The key point is that the length of the feedback loop isn’t just measured based on time of test execution. It’s also on quality of information, and if all you’re getting is a red light for total product instead of a pointer to an unexpected result midway, you can have the fastest test suite in the world but your test feedback loop is still going to be slow. Keep your unit tests.

Assertively yours,
Ben

[0] Via Lynoure Braakman: https://www.guru99.com/equivalence-partitioning-boundary-value-analysis.html
[1] Property based testing: https://fsharpforfunandprofit.com/posts/property-based-testing/
[2] Snapshot testing: https://semaphoreci.com/community/tutorials/snapshot-testing-react-components-with-jest
[3] Thanks to David Colgan for providing the name “snapshot testing”
[4] Test pyramid: https://martinfowler.com/bliki/TestPyramid.html

More like this

• Prioritizing technical debt vs. feature development in your Django app (This Old Pony #55)
• Triaging and squashing bugs in existing Django apps (This Old Pony #53)