Property-based testing is one of the powerful technique to unit test the code. Unlike the example-based testing (where we Arrange a set of example inputs to Assert the system under test), Property based testing enable us to focus on what we wanted to test and liberate us from doing mundane work on coming up with sample inputs.
Recently, I got a chance to use Property based testing in an open source library that I am developing to use Suave in Azure Functions. I felt very productive and it was a such a joy to write the unit tests.
In this blog post, you are going to learn how Property-based testing has helped me and why you need to consider using(or learning) it
Use Case
Let me start with the use case that I wanted to test. The library that I am working is an adapter between two HTTP protocol abstractions, System.Net.Http and Suave.Http
The first task is to map HttpStatusCode of System.Net to the HttpCode of Suave.Http and the next task is doing the vice-versa.
HttpStatusCode is an enum type and HttpCode is a discriminated union type. Refer this blog post to know more about this difference.
Identifying the Property
The first step is Property-based testing is identifying the property. In other words, it forces to think about the relationship between the input and the output.
I believe this thinking will make a huge difference in the quality of the software that we deliver.
In the first task that we are about to implement, the relationship is the integer representation of the HTTP status code is both HttpStatusCode and HttpCode.
Programmatically, this can be asserted like
// boolean condition
// In F# (=) operator represents the equality checking
LanguagePrimitives.EnumToValue httpStatusCode = httpCode.codeThe EnumToValue returns the integer value associated with the enum, which is nothing but the integer representation of the HTTP status code it represents. The
codeis a member of HttpCode that represents the same integer.
If this property holds true for given set of inputs, then we can assert that the function that does the transformation is working correctly.
Implementing the mapping function httpStatusCode
The initial implementation that I had in my mind was
let NetStatusCode = function
| HttpCode.HTTP_200 -> HttpStatusCode.OK
| HttpCode.HTTP_201 -> HttpStatusCode.Created
| HttpCode.HTTP_400 -> HttpStatusCode.BadRequest
| HttpCode.HTTP_404 -> HttpStatusCode.NotFound
| HttpCode.HTTP_202 -> HttpStatusCode.Accepted
| // ...If I haven't choosen to use Property based testing, I might have ended up with this line by line mapping for all the HTTP status codes. As a matter of fact, in my last blog post on using Suave in Azure Functions, I've used this same approach.
While thinking regarding properties to assert the transformation, for the first time I came to know about the Language Primitives module in F# and the EnumToValue function.
There should be an another function EnumOfValue right?
Yes!
Let's use this in the httpStatusCode function implementation
let httpStatusCode (httpCode : HttpCode) : HttpStatusCode =
LanguagePrimitives.EnumOfValue httpCode.codeShort and Sweet!
Property-based testing made my day and helped me to save a lot of keystrokes!
Writing Our First Property based Unit Test
In F# we can use the FsCheck library to write the Property based unit tests. For this implementation, we will be using FsCheck.Xunit to run Property tests in XUnit
open Suave.Http
open System.Net
open FsCheck
open FsCheck.Xunit
[<Property>]
let ``httpStatusCode maps Suave's HttpCode to System.Net's HttpStatusCode ``
(httpCode : HttpCode) =
let httpStatusCode = Response.httpStatusCode httpCode
LanguagePrimitives.EnumToValue httpStatusCode = httpCode.codeThe magic here is the Property attribute. It makes our life easier by auto-populating the parameter httpCode with the all the possible values and runs the unit test against each value.
As a convention, we need to return boolean by validating the property that we wanted to assert.
There is no Arrange step to setup the input parameter and FsCheck does that for you with 100% test coverage 😃
Mapping HttpCode to HttpStatusCode
The next task in the use case is doing the reverse, mapping HttpStatusCode to HttpCode.
Let's start with the test first.
The integer representation of HTTP status code property that we used on the previous task holds true for this mapping also.
[<Property>]
let ``suaveHttpCode maps System.Net's HttpStatusCode to Suave's HttpCode if exists``
(httpStatusCode : HttpStatusCode) =
let httpCode = suaveHttpCode httpStatusCode
let code = LanguagePrimitives.EnumToValue httpStatusCode
Option.isSome httpCode && httpCode.Value.code = codeThe suaveHttpCode function returns an Option type as Suave.Http.HttpCode's tryParse function parses the integer HTTP status code and returns the corresponding HttpCode if the parsing succeeds.
The implementation of suaveHttpCode function is
let suaveHttpCode (httpStatusCode : HttpStatusCode) =
let code =
LanguagePrimitives.EnumToValue httpStatusCode
|> HttpCode.tryParse
match code with
| Choice1Of2 httpCode -> Some httpCode
| _ -> NoneNow if we run the unit tests, You will be getting the following error (if you are using Suave NuGet package version <= 1.1.3)
Suave.Azure.Functions.Tests.suaveHttpCode maps System.Net's HttpStatusCode to Suave's HttpCode [FAIL]
FsCheck.Xunit.PropertyFailedException :
Falsifiable, after 7 tests (0 shrinks) (StdGen (1629668181,296211181)):
Original:
Unused
The reason for this failure is Suave (up to v1.1.3) doesn't have the HTTP status code 306 (UnUsed). The unit test that we wrote was exercised by the FsCheck for the all possible values of the HttpStatusCode enum till it encountered this failure case.
Let's patch our unit test to fix this failure.
let httpCode = Response.suaveHttpCode httpStatusCode
let code = LanguagePrimitives.EnumToValue httpStatusCode
let unSupportedSuaveHttpCodes = [306]
if List.contains code unSupportedSuaveHttpCodes then
Option.isNone httpCode
else
Option.isSome httpCode && httpCode.Value.code = codeIf you run the unit test after this, you will get an another failure
Suave.Azure.Functions.Tests.suaveHttpCode maps System.Net's HttpStatusCode to Suave's HttpCode [FAIL]
FsCheck.Xunit.PropertyFailedException :
Falsifiable, after 10 tests (0 shrinks) (StdGen (507658935,296211184)):
Original:
UpgradeRequired
Again, The reason is HTTP Status Code 426 (UpgradeRequired) is not supported is Suave at this point of writing. Let's patch this too by adding this to the list unSupportedSuaveHttpCodes.
// ...
let unSupportedSuaveHttpCodes = [306;426]
// ...After this FsCheck is happy and we too.
If I haven't used Property based testing, for sure, I might have missed these two unsupported HTTP status codes.
The Spirit of Open Source
While looking at Suave.Http codebase to fix the unit test failures, this is what I saw in the code
This is the beauty of open source, and I am glad to do it
Summary
Writing tests first forces you to think about the problem you're solving. Writing property-based tests forces you to think way harder. — Jessica Kerr (@jessitron)
