ReferenceTests.jl Documentation

@withcolor ex

Make sure that ex is evaluated while Base.have_color is set to true. The original value of Base.have_color will be restored afterwards.

This macro is particularily useful for CI, where it is not unusual that julia is executed without the --color=yes argument by default.

@withcolor print_with_color(:green, "foo")

@io2str ex

Search and replace the placeholder ::IO with a newly allocated Base.IOBuffer which is afterwards converted to a string and returned.

This macro allows a user to conveniently test text-based printing functionality that require an IO argument. A particularly common example of such would be a custom Base.show method.

Examples

julia> using ReferenceTests

julia> @io2str print(::IO, "Hello World")
"Hello World"

julia> @io2str show(IOContext(::IO, :limit=>true, :displaysize=>(10,10)), "text/plain", ones(5))
"5-element Array{Float64,1}:
 1.0
 1.0
 1.0
 1.0
 1.0"

@test_reference filename expr [by] [kw...]

Tests that the expression expr with reference filename using equality test strategy by.

The pipeline of test_reference is:

1. preprocess expr
2. read and preprocess filename via FileIO.jl
3. compare the results using by
4. if test fails in an interactive session (e.g, include(test/runtests.jl)), an interactive dialog will be trigered.

Arguments:

• filename::String: relative path to the file that contains the macro invocation.
• expr: the actual content used to compare.
• by: the equality test function. By default it is isequal if not explicitly stated.
• format: Force reading the file using a specific format

Types

The file-extension of filename, as well as the type of the result of evaluating expr, determine how contents are processed and compared. The contents is treated as:

• Images when expr is an image type, i.e., AbstractArray{<:Colorant};
• SHA256 when filename endswith *.sha256;
• Any file-type which FileIO.jl handles and with the proper backend loaded;
• Text as a fallback.

Any FileIO.jl handled filetype

Any file-types which can be read by FileIO.jl can be used. Note that most Julia values can be stored using packages such as, e.g., BSON.jl or JLD and can thus be used in reference-tests.

Images

Images are compared approximately using a different by to ignore most encoding and decoding errors. The default function is generated from psnr_equality.

The file can be either common image files (e.g., *.png), which are handled by FileIO, or text-coded *.txt files, which is handled by ImageInTerminal. Text-coded image has less storage requirements and also allows to view the reference file in a simple terminal using cat.

SHA256

The hash of the expr and content of filename are compared.

Fallback

Simply test the equality of expr and the contents of filename without any preprocessing. Note that reading filename will return a String.

Examples

# compare text-file against string representation of a value
@test_reference "stringtest1.txt" collect(1:20)

# test number with absolute tolerance 10
@test_reference "references/string3.txt" 1338 by=(ref, x)->isapprox(ref, x; atol=10)

# store a floating point array ar in BSON-file and compare it back.
# Note that a Dict is needed and the custom comparison function.
using BSON
comp(d1, d2) = keys(d1)==keys(d2) &&
    all([ v1≈v2 for (v1,v2) in zip(values(d1), values(d2))])
@test_reference "reftest-files/X.bson" Dict(:ar=>[1, pi, 4.5]) by=comp

# store as string using ImageInTerminal with encoding size (5,10)
using TestImages
@test_reference "camera.txt" testimage("cameraman") size=(5,10)

# using folders in the relative path is allowed
@test_reference "references/camera.png" testimage("cameraman")

# Images can also be stored as hash. Note however that this
# can only check for equality (no tolerance possible)
@test_reference "references/camera.sha256" testimage("cameraman")

# test images with custom Peak Signal-to-Noise Ratio (psnr) threshold
@test_reference "references/camera.png" testimage("cameraman") by=psnr_equality(20)

psnr_equality(threshold=25) -> f

Generates an equality comparison function in terms of Peak Signal-to-Noise Ratio (PSNR).

The return function f accepts two images reference and actual as its inputs; f(reference, actual) == true if psnr(reference, actual) >= threshold.

This function is useful for image comparison; higher PSNR means higher image similarity. Hence if you want to loose the equality test creterion, you can use a smaller threshold value, e.g.,

@test_reference "references/camera.png" testimage("cameraman") by=psnr_equality(20)