# ReTest.jl

ReTest is a testing framework for Julia allowing:

1. Defining tests in source files, whose execution is deferred and triggered on demand.

This is useful when one likes to have definitions of methods and corresponding tests close to each other. This is also useful for code which is not (yet) organized as a package, and where one doesn't want to maintain a separate set of files for tests.

2. Filtering run testsets with a Regex, which is matched against the descriptions of testsets.

This is useful for running only part of the test suite of a package. For example, if you made a change related to addition, and included "addition" in the description of the corresponding testsets, you can easily run only these tests.

Note that a pull request exists in the Julia repository to implement regex-filtering for Test.@testset.

A couple more features are also enabled, like shuffling the order in which the testsets are run, or running testsets in parallel (via Distributed).

ReTest is mostly backward-compatible with Test, i.e. minimal change to test files is necessary in order to switch to ReTest; it's often even possible to use ReTest features without changing a line, e.g. on Julia's Base/stdlib tests), cf. Working with test files which use Test.

ReTest is still at an early stage of development. If you like to use it for your packages, it's recommended to keep your tests compatible with Test, so that they can be run through both frameworks (e.g. using ReTest interactively and Test in C.I.).

## Usage

The exported ReTest.@testset macro can be used as a direct replacement for Test.@testset (with limitations, see below), and retest() has to be called for the tests to be executed. See retest's docstrings for more details. Moreover, ReTest re-exports (almost) all exported symbols from Test, so there should not be any need to import Test together with ReTest.

When using @testset "inline", i.e. within the source-code of a package, one can use the InlineTest package instead of ReTest, which only defines the strict minimum and also exports @testset, and therefore loads faster (even if ReTest itself loads fast, it can be desirable to have an even lighter dependency). But ReTest still has to be loaded (as a "test" dependency) in order to call retest.

Finally, for convenience, ReTest.@testset also implicitly defines a runtests function within the enclosing module (and within all recursive parent modules), say M, such that M.runtests(...) is equivalent to calling retest(M, ...).

## Quick start

Both ReTest and InlineTest are registered packages and can be installed the usual way. Let's create a package MyPackage, which is initially a single file located in a directory known to LOAD_PATH. We want to test its greet function, by writing tests directly in the same file; this is a use-case for InlineTest, which loads faster than ReTest:

# MyPackage.jl file

module MyPackage
using InlineTest

greet() = "Hello World!"

@testset "greet" begin
@test greet() == "Hello World!"
end

end # module

Now, in a Julia session, we load MyPackage and ReTest (needed to actually run the tests):

julia> using MyPackage, ReTest

julia> MyPackage.runtests()
Pass
greet    |      1

Suppose now that we organize MyPackage as a standard package, with a proper "runtests.jl" file. We can still keep testsets within "MyPackage.jl", while adding more thorough tests in the "test" folder, which can contain two files, "runtests.jl" and "MyPackageTests.jl":

# MyPackage/test/runtests.jl file

using ReTest, MyPackage
include("MyPackageTests.jl")

# when including this file (e.g. with Pkg.test), all the tests
# in both modules will be run:

retest(MyPackage, MyPackageTests)
# MyPackage/test/MyPackageTests.jl file

module MyPackageTests
using MyPackage, ReTest

@testset "more greet" begin
@testset "concatenation" begin
@test MyPackage.greet()^2 == "Hello World!Hello World!"
end
end

@testset "stuff" begin
@test true
end

end # module

We can now load tests either via using MyPackageTests, if LOAD_PATH is configured appropriately, or via include, and run whichever tests we want:

julia> include("test/MyPackageTests.jl");

julia> using ReTest # to use the retest function

julia> retest(dry=true, verbose=2) # just list tests, showing nested ones
MyPackage
1| greet

Main.MyPackageTests
1| more greet
2|   concatenation
3| stuff

julia> retest("greet", verbose=2) # run only tests related to greet()
Pass
MyPackage:
greet              |      1

Main.MyPackageTests:
more greet         |      1
concatenation    |      1

Overall              |      2

julia> MyPackageTests.runtests(3) # run only testset with ID 3 in MyPackageTests
Pass
3| stuff              |      1

Here it is for basic usage!

## API

### Defining tests

InlineTest.@testsetMacro
@testset args...

Similar to Test.@testset args..., but the contained tests are not run immediately, and are instead stored for later execution, triggered by retest() or runtests().

Besides the @testset body (last argument) and a description string, arguments of @testset can be:

• the verbose option, with a literal boolean value (e.g. verbose=true)
• a literal symbol, serving as a label which can be used for testset filtering (see retest's docstring for details). All nested testsets inherit such labels.

Invocations of @testset can be nested, but qualified invocations of ReTest.@testset can't.

A @testset can contain a nested Test.@testset, or call a function which defines a Test.@testset: in this case, the Test.@testset will be run whenever the parent testset is run, but retest won't know about it: it won't be taken into account during the filtering phase, and won't be printed in dry mode.

Internally, @testset expressions are converted to an equivalent of Test.@testset at execution time.

source
InlineTest.@testset_macroMacro
@testset_macro @mac

Declare @mac as a macro which must be expanded statically by retest so that contained @testsets can be discovered.

Consider this pattern with Test which factors out testsets in a function:

using Test

function test_iseven(x)
@testset "iseven $x" begin @test iseven(x) end end @testset "test$x" for x=2:2:4
test_iseven(x)
end

This doesn't translate directly with ReTest, as the call to test_iseven will be performed at run-time, and will end up declaring a new @testset "iseven $x" at toplevel (this is a problem similar to having include inside testsets). So on the first run of retest(), no @test is run, and on the second one, it fails because x is not defined at global scope. The alternative is to turn test_iseven into a macro and declare it with @testset_macro: using ReTest macro test_iseven(x) quote @testset "iseven$($x)" begin @test iseven($x)
end
end
end

@testset_macro @test_iseven

@testset "test $x" for x=2:2:4 @test_iseven(x) end Then, running retest("iseven", verbose=2) gives:  Pass test 2 | 1 iseven 2 | 1 test 4 | 1 iseven 4 | 1 Main | 2 source ### Running tests InlineTest.retestFunction retest(mod..., pattern...; dry::Bool=false, stats::Bool=false, verbose::Real=true, [id::Bool], shuffle::Bool=false, recursive::Bool=true, static::Union{Bool,Nothing}=nothing, dup::Bool=false, load::Bool=false, seed::Union{Integer,Bool}=false, marks::Bool=true, tag=[], spin::Bool=true) Run tests declared with @testset blocks, within modules mod if specified, or within all currently loaded modules otherwise. Filtering patterns can be specified to run only a subset of the tests. Keywords • If dry is true, don't actually run the tests, just print the descriptions of the testsets which would (presumably) run. • If stats is true, print some time/memory statistics for each testset. • If specified, verbose must be an integer or Inf indicating the nesting level of testsets whose results must be printed (this is equivalent to adding the verbose=true annotation to corresponding testsets); the default behavior (true or 1) corresponds to printing the result of top-level testsets. • If id is true, a unique (per module) integer ID is printed next to each testset, which can be used for filtering. The default value of id depends on other options. • If shuffle is true, shuffle the order in which top-level testsets within a given module are run, as well as the list of passed modules. • If recursive is true, the tests for all the recursive submodules of the passed modules mod are also run. • The static keyword controls testsets filtering: if true, only testsets which are known to match "statically" the passed patterns, i.e. at filtering time, are run. See docstring of interpolated for more details. • If dup is true, multiple toplevel testsets can have the same description. If false, only the last testset of a "duplicate group" is kept. The default is false in order to encourage having unique descriptions (useful for filtering) but also and mostly to play well with Revise. This keyword applies only to newly added testsets since the last run. • When load is true, for each package module Mod which is selected, retest attempts to also select a corresponding Main.ModTests module with the same pattern specification, unless such module is already explicitly passed as an argument. If this test module doesn't already exist, retest attempts first to include into Main the corresponding test file "test/ModTests.jl" which is assumed, if it exists, to define one or more test modules (typically ModTests); these new test modules are associated to Mod (they inherit its pattern specification as above), and are cached and used again on subsequent invocations. • If seed is provided, it is used to seed the global RNG before running the tests. As a special case, if seed === false (the default), no seeding is performed, and if seed === true, a seed is chosen randomly. • When marks and dry are true, "check marks" are printed next to testsets which passed or failed in previous runs, as well as labels. • The tag keyword allows to tag a testset with labels, encoded as symbols. When tag is a list of symbols, tag all matching testsets with these. When tag is a symbol, tag all matching testsets with it. Instead of a symbol :sym, it's possible to instead pass not(:sym) in order to remove the :sym label from matching testsets. Currently, tag has an effect only if dry is true. • When spin is true, the description of the testset being currently executed is shown (if there is only one), as well as a "spinner". This is disabled when all the available threads/workers are used to run tests (i.e. typically Threads.nthreads() should be greater than 1 for spin to take effect). Note also that this feature slows down a bit the execution of tests. The default values of these keywords can be overriden by defining a dictionary or named tuple within Main called __retest_defaults__, whose keys are symbols. E.g. __retest_defaults__ = (verbose=Inf, spin=false). Filtering It's possible to filter run testsets by specifying one or multiple patterns. A testset is guaranteed to run only if it "matches" all passed patterns (conjunction). Even if a testset is run, its nested testsets might not run if they don't match the patterns. Moreover if a testset is run, its enclosing testset, if any, also has to run (although not necessarily exhaustively, i.e. other nested testsets might be filtered out). A pattern can be a string, a Regex, an integer, a symbol, an array or a tuple. For a testset to "match" an array, it must match at least one of its elements (disjunction). To match a tuple, it must match all of its elements (conjunction). To match an integer, its ID must be equal to this integer (cf. the id keyword). To match a symbol, it must be tagged with that symbol (label). A pattern can also be the "negation" of a pattern, via the not function, which allows to exclude testsets from being run. As a special case, the negation of an integer can be expressed as its arithmetic negation, e.g. not(3) is equivalent to -3. Patterns can also be created via reachable, interpolated and depth. Regex filtering The "subject" of a testset is the concatenation of the subject of its parent @testset, if any, with "/$description" where description is the testset's description. For example:

@testset "a" begin # subject is "/a"
@testset "b" begin # subject is "/a/b"
end
@testset "c$i" for i=1:2 # subjects are "/a/c1" and "/a/c2" end end When pattern is a Regex, a testset is guaranteed to run only when its subject matches pattern. Moreover, even if a testset matches (e.g. "/a" above with pattern == r"a$"), its nested testsets might be filtered out if they don't also match (e.g. "a/b" doesn't match pattern).

If a passed pattern is a string, then it is wrapped in a Regex with the "case-insensitive" flag, and must match literally the subjects. This means for example that "a|b" will match a subject like "a|b" or "A|B", but not like "a" (only in Julia versions >= 1.3; in older versions, the regex is simply created as Regex(pattern, "i")).

As a special case, if a string pattern starts with the '-' character, it's interpreted as the negation of the pattern corresponding to the string with '-' chopped off, e.g. "-abc" is equivalent to not("abc"). Unless the string starts with two '-' characters, in which case the first '-' is chopped off, e.g. "--abc" will match subjects such as "123-abc". To negate such a pattern, just use not, e.g. not("--abc").

Per-module patterns

In addition to modules or patterns, positional arguments of retest can also be a pair of the form mod => pattern: then pattern is used to filter only testsets from mod; if other "standalone" patterns (not attached to a module) are specified, they also conjunctively apply to mod. For example, a call like retest(mod1 => 1:3, mod2, "x") is equivalent to retest(mod1 => (1:3, "x"), mod2 => "x"). If recursive is true, pattern is also applied to all recursive submodules sub of mod; if sub is also specified as sub => subpat, the patterns are merged, i.e. this is equivalent to specifying sub => (pattern, subpat).

Note

This function executes each (top-level) @testset block using eval within the module in which it was written (e.g. mod, when specified).

source
ReTest.watchFunction
ReTest.watch(args...; kwargs...)

Run retest(args...; kwargs...) repeatedly each time Revise detects file updates. Revise must be loaded beforehand in your Julia session.

Warning

This experimental function is not tested and is currently very basic.

source

Cf. Working with test files which use Test for hijack and hijack_base.

ReTest.loadFunction
ReTest.load(testpath::AbstractString;
parentmodule::Module=Main, [revise::Bool])

Include file testpath into parentmodule. If revise is true, Revise, which must be loaded beforehand in your Julia session, is used to track all recursively included files (in particular testsets). The revise keyword defaults to true when Revise is loaded and VERSION >= v"1.5", and to false otherwise.

The point of using this function is when revise is true and in particular when files are included recursively. If revise is false, this is equivalent to parentmodule.include(testpath), and if there are no recursively included files, this should be equivalent to Revise.includet(testpath), provided parentmodule == Main and all @testsets defined in testpath are in a module defining __revise_mode__ = :eval.

Julia 1.5

This function requires at least Julia 1.5 when revise is true.

source
ReTest.load(Mod::Module, testfile::AbstractString="ModTests.jl";
parentmodule::Module=Main, [revise::Bool])

Given a package Mod, include into parentmodule the corresponding tests from file testfile, which is assumed to be located in the "test" directory of the package. It is expected that a unique new test module is created within parentmodule after the inclusion, which is then returned. Otherwise, the list of all newly created test modules is returned, triggering a warning if it's empty.

If revise is true, Revise, which must be loaded beforehand in your Julia session, is used to track the test files (in particular testsets). Note that this might be brittle, and it's recommended instead to load your test module via using ModTests. The revise keyword defaults to true when Revise is loaded and VERSION >= v"1.5", and to false otherwise.

Julia 1.5

This function requires at least Julia 1.5 when revise is true.

source

### Filtering tests

ReTest.notFunction
not(pattern)

Create an object suitable for filtering testsets (in the retest function), which "negates" the meaning of pattern: a testset matches not(pattern) if and only if it doesn't match pattern.

For example not("a") matches any testset whose subject doesn't contain "a", and not(1:3) matches all the testsets but the first three of a module.

If pattern is an integer or a ReTest object (i.e. not a AbstractString, Regex, Tuple or AbstractArray), not(pattern) can be expressed as -pattern.

String patterns can also be negated by prepending '-', see retest for details.

source
ReTest.passConstant
pass
fail

Filtering patterns which match any testset which already ran, succesfully for pass or with at least one error for fail. The pattern [pass, fail] therefore matches any testset which already ran.

source
ReTest.reachableFunction
reachable(pattern)

Create a filtering pattern which matches any testset matching pattern or whose parent testset, if any, matches reachable(pattern). In other words, if a testset matches pattern, all its recursive nested testsets will also match.

When pattern::String, reachable(pattern) has the same effect as pattern, because the subject of a testset is contained in the subjects of all its nested testsets. So reachable is typically useful when pattern is an integer.

Examples

julia> module T
using ReTest
@testset "a" verbose=true begin
@test true
@testset "b" begin
@test true
end
end
@testset "c" begin
@test true
end
end;

julia> retest(T, reachable(1), dry=true)
1| a
2|   b

julia> retest(T, not(reachable(1)), dry=true)
3| c

Note that the algorithm for reachable is currently not optimized, i.e. it will match pattern against all parents of a testset until success, even when this match was already performed earlier (i.e. the result of matching against pattern is not cached).

Also, in the current implementation, the subject of a parent testset is inferred from the subject of a testset, by chopping off the last component, determined by the last occurrence of '/'. This has two consequences. It will produce incorrect results if the description of a testset contains '/', and also, with interpolated when the subject is "unknown" due to un-interpolated descriptions. Consider the following example:

julia> module Fail
using ReTest
@testset "a" begin
x = 1
@testset "b$x" begin @testset "c" begin end end end end; julia> retest(Fail, reachable(1), verbose=9, dry=true) 1| a 2| "b$(x)"
3|     c

julia> retest(Fail, reachable(interpolated), verbose=9, dry=true)
1| a

Here, both testsets with id 2 and 3 have an unknown subject (at filtering time), which prevents the algorithm to detect that one of their parents (testset 1) actually has an "interpolated" description.

On the other hand, even with these unknown subjects, something like reachable("a") would work as expected:

julia> retest(Fail, reachable("a"), verbose=9, dry=true)
1| a
2|   "b$(x)" 3| c julia> retest(Fail, reachable("a"), verbose=9, dry=true, static=true) 1| a Julia 1.3 This function requires at least Julia 1.3. source ReTest.interpolatedConstant interpolated Singleton pattern which matches any testset whose subject can be interpolated "statically", i.e. at filtering time before testset are actually run. Non-inferrable subjects include those constructed from descriptions containing interpolated values which can't be known until run time. This pattern has an effect closely related to that of the static keyword of retest, discussed below, which is probably more generally useful. Examples Given these testset: @testset "outer" verbose=true begin @test true inner = "inner" @testset "$inner" begin
@test true
end
end
@testset "other" begin
@test true
end

We get:

julia> retest("other", dry=true)
Main
1| outer
2|   "$(inner)" 3| other julia> retest("other", dry=false) Pass outer | 1 other | 1 Main | 2 julia> retest("other", dry=true, interpolated) Main 3| other Without interpolated, retest can't decide at filtering time whether the "inner" testset will run, so must mark the "outer" testset as having to run. At run time, "inner" is not run because it doesn't match the pattern, but "outer" still had to run to determine this. With the interpolated pattern, "inner" is filtered out and retest selects only testsets which are statically known to have to run. So again, interpolated doesn't have the same effect at filtering time (like when dry=true) and at run time. For example, one can see the list of non-interpolated subjects as follows with dry=true, but not run them (because everything is interpolated at run time): julia> retest(not(interpolated), dry=true) 1| outer 2| "$(inner)"

julia> retest(not(interpolated), dry=false)
Pass
Main:
outer |      1

static keyword

Unlike interpolated, the static keyword of retest, when true, filters out only testsets which can't be proven to have to run at filtering time, let's call them "undecidable". It can have sometimes the same effect as when using interpolated, e.g. retest("other", dry=true, static=true) and retest("other", dry=true, interpolated) give the same result.

But in some cases we might want to filter out noisy testsets whose subjects can't be interpolated, but still include those which are relevant. For example, assume we want to run testsets 1 and 2, while excluding other testsets with uninterpolated subjects:

julia> retest(1:2, dry=true, interpolated)
Main
1| outer

julia> retest(1:2, dry=true, static=true)
Main
1| outer
2|   "$(inner)" The solution with interpolated is not what we want, as we specifically want testset 2 to run. Given the filtering specifications (1:2 here), the filtering algorithm can determine that 2 should run even though its subject is unknown at this point. Given a filtering specification, there are three kind of testsets: • "undecidable" (see above) • "match": they are known statically to have to run • "nomatch": they are known statically to not have to run The default value of the static keyword is nothing, which means to run testsets which are not known with certainty to not match, i.e. "match" and "undecidable" testsets. As seen above, when static == true, only "match" testsets are run. When static == false, the behavior is the opposite: only "undecidable" testsets are run. Of course, other combinations involving "nomatch" testsets can be had by reversing the filtering pattern via not. For example, to get the equivalent to the not(interpolated) example above, but with an effect which persists at run time (dry = false), you can use static = false together with the match-all regex pattern r".*", which will mark the "inner" testset as "undecidable" (the algorithm inspects slightly patterns just to recognize the simple match-all patterns "" and r"", but won't detect that r".*" would match "$(inner)"):

julia> retest(r".*", static=false, dry=true)
Main
1| outer
2|   "$(inner)" julia> retest(r".*", static=false, dry=false) Pass Main: outer | 2 inner | 1 One example of a rare case where a given testset is not in a single of the above three categories is as follows: @testset "a" begin x = 2 @testset "b$(i==1 ? 1 : x)" for i=1:2
@testset "c" begin
# subject is "match" at first iteration and
# "undecidable" at second iteration
@test true
end
end
end

One thing to understand is that the "identity" of a testset is determined by a given occurrence of the @testset macro. In the example above, for either the patterns "b" or "c", the two inner testsets are both "match" and "undecidable". In this case, the filtering algorithm selects a testset to run if at least one iteration would lead to this decision. Here, if static=true the first iteration would run, and if static=false the second iteration would run. This results in the same selection whatever the value of static is.

source
ReTest.depthFunction
depth(d::Integer)

Create a pattern which matches testsets at "depth" d. Toplevel testsets have depth 1, their direct children (nested testsets) depth 2, and so on.

Examples

julia> module Depth
using ReTest
@testset "1" begin
@testset "2" begin
@testset "3" begin end
end
@testset "4" begin end
end
end;

julia> Depth.runtests(dry=true, verbose=3, depth(2))
1| 1
2|   2
4|   4

julia> Depth.runtests(dry=true, verbose=3, depth(3))
1| 1
2|   2
3|     3

julia> Depth.runtests(dry=true, verbose=3, reachable(depth(2)))
1| 1
2|   2
3|     3
4|   4

julia> Depth.runtests(dry=true, verbose=3, depth.(2:3))
1| 1
2|   2
3|     3
4|   4
source

## Caveats

ReTest.@testset comes with a couple of caveats/limitations, some of which should be fixable:

• Toplevel testsets (which are not nested within other testsets), when run, are evaled at the toplevel of their parent module, which means that they can't depend on local variables for example.

• "testsets-for" (@testset "description" for ...), when run, imply evaling their loop variables at the toplevel of their parent module; this implies that iteration expressions shouldn't depend on local variables (otherwise, the testset subject usually can't be known statically and the testset can't be filtered out with a Regex).

• Testsets can not be "custom testsets" (cf. Test documentation).

• Nested testsets can't be "qualified" (i.e. written as ReTest.@testset).

• Regex filtering logic might improve in future versions, which means that with the same regex, less tests might be run (or more!). See retest's docstring to know which testsets are guaranteed to run.

• Descriptions of testsets must be unique within a module, otherwise they are overwritten and a warning is issued, unless Revise is loaded; the reason is the current implemented heuristic to allow Revise do its magic.

• There is not yet a good solution to factor out testsets into functions called within other testsets; a work-around is to use @testset_macro, or to use only the @test and @test_* macros within these functions.

• Interrupting running testsets with Control-C sometimes doesn't work well, because of the use of multiple tasks in retest. There can also be usability annoyances when some tests are failing. This will hopefully be fixed soon.

#### Including files from within testsets

TLDR: don't use include(file) within testsets when file defines other testsets.

There is limited support for include(path) expressions within testsets: all what ReTest does is to adjust the path according to the location of the containing file parentfile. This is necessary, because include is not run immediately when that file is evaluated; when the given testset is triggered (via a retest call), include doesn't have the same "context" as parentfile, which would lead to path being interpreted as non-existing (unless path is an absolute path). So when parsing testsets, ReTest prepends the directory name of parentfile to path.

The important point is that include is executed at retest-time; if the included file defines other @testset expressions, this will define new testsets in the enclosing module, but these won't be run immediately; upon a new retest() invocation, these new testsets will be run, but the old one too (the one containing include), which will redefine included testsets. This is brittle, and it's recommended to not include, within testsets, files defining other testsets.

## Switching from Test to ReTest

When used in a package MyPackage, the recommended way to organize test code is as follows:

1. replace using Test by using ReTest in the "runtests.jl" file (and in all other test files having using Test)
2. wrap the whole content of "runtests.jl" within a module named MyPackageTests
3. rename "runtests.jl" to "MyPackageTests.jl"
4. create a "runtests.jl" file with the following content: include("MyPackageTests.jl"); MyPackageTests.runtests()

This means that running "runtests.jl" will have the same net effect as before. The "MyPackageTests.jl" file can now be included in your REPL session (include("MyPackageTests.jl")), and you can run all or some of its tests (e.g. MyPackageTests.runtests("addition")). This test file can also be included via the ReTest.load function or via the load keyword of retest.

Wrapping the tests in MyPackageTests allows to not pollute Main and keeps the tests of different packages separated. Also, you can modify "MyPackageTests.jl" and re-include it to have the corresponding tests updated (the MyPackageTests module is replaced in Main); otherwise, without a MyPackageTests module, including the file a second time currently triggers a warning for each overwritten toplevel testset.

#### Keeping the ability to use Test

One might want to have the possibility to use either Test or ReTest depending on the context. Reasons to still use Test include:

• when running retest for the first time in a Julia session, more code has to be compiled than when running tests with Test, so in the case of running the whole test suite, few seconds can be spared (although using ReTest in parallel mode would generally compensate for this);
• ReTest is not yet a fully mature and battle tested package, so you might want to not rely exclusively on it, e.g. for C.I.

An alternate way to organize the test files is as follows, assuming using Test is only present in "runtests.jl":

1. remove using Test from "runtests.jl"
2. rename "runtests.jl" to "tests.jl"
3. create a "MyPackageTests.jl" file with the following content:
module MyPackageTests
using ReTest
include("tests.jl")
end
4. create a "runtests.jl" file with the following content:
using Test
include("tests.jl")

That way, include("test/runtests.jl") or Pkg.test() will run tests using Test, while include("test/MyPackageTests.jl"); MyPackageTests.runtests() will use ReTest.

## Filtering

Most of the time, filtering with a simple string is likely to be enough. For example, in

@testset "a" begin
@test true
@testset "b" begin
end
@testset "c" begin
end
end

running retest(M, "a") will run everything, retest(M, "b") will run @test true and @testset "b" but not @testset "c". Note that if you want to run @testset "b", there is no way to not run @test true in @testset "a"; so if it was an expensive test to run, instead of @test true, it could be useful to wrap it in its own testset, so that it can be filtered out.

## Running tests in parallel with Distributed

Currently, the tests are automatically run in parallel whenever there are multiple workers, which have to be set manually. Running the tests looks like:

using Distributed
@everywhere include("test/MyPackageTests.jl")
MyPackageTests.runtests()

If the test code doesn't use ReTest (cf. Working with test files which use Test), this can be done as follows:

using Distributed
using ReTest
@everywhere begin
using ReTest, MyPackage
ReTest.hijack(MyPackage)
end
MyPackageTests.runtests()
Note

As was already mentioned, testset-for iterators are evaluated at load time in the enclosing module, but this currently happens only in the main process. This can lead to unexpected errors when the package was written without a Distributed use-case in mind.

For example, say the package defines a constant singleton object X which is normally equal to itself (because X === X). But if X is assigned to a testset-for loop variable x, it will be the one from the main process, so within the testset-for, a test like x == X might fail because X refers to the singleton object defined in another process; a solution in this case could be to define explicitly == for objects of the type of X.

It should be relatively easy to support threaded execution of testsets (it was actually implemented at one point). But it often happens that compiling package code and testset code (which currently is not threaded) takes quite more time than actually running the code, in which case using Distributed has more tangible benefits.

## Working with Revise

When Revise is loaded and a testset is updated, ReTest will observe that a new testset is added with the same description as a previously existing one, which is then overwritten. This works only if the description is not modified, otherwise both the old and new versions of the testset will co-exist.

For testsets in a "script" loaded with includet, e.g. those in a "test/MyPackageTests.jl" file, you can request Revise to "load" the updated testsets by putting __revise_mode__ = :eval in the enclosing module.

When files are included recursively, plain includet won't work (it is currently documented to be "deliberately non-recursive"). There are three work-arounds, of which the first is recommended:

1. load MyPackageTests as a module, i.e. via using MyPackageTests instead of include("test/MyPackageTests.jl") (this might involve updating your LOAD_PATH to include "test/" and making sure the required packages are found)
2. load MyPackageTests via ReTest.load(MyPackage, revise=true), but this works only in "simple enough" situations
3. use the following recursive_includet function instead of includet:
function recursive_includet(filename)
includet(filename)
for (mod, file) in newly_included
Revise.track(mod, file)
end
end

## Working with test files which use Test

It's sometimes possible to use ReTest features on a test code base which uses Test, without modifications:

• if you have a package Package, you can try ReTest.hijack(Package), which will define a PackageTests module when successful, on which you can call retest. To have Revise track changes to test files, use ReTest.hijack(Package, revise=true).
• if you have a test file "testfile.jl", try ReTest.hijack("testfile.jl") (this will define a fresh module like above).
• Base and standard library modules can also be passed to ReTest.hijack (corresponding tests are loaded via the lower level ReTest.hijack_base function).
ReTest.hijackFunction
ReTest.hijack(source, [modname];
parentmodule::Module=Main, lazy=false, [revise::Bool],
[include::Symbol], include_functions=[:include])

Given test files defined in source using the Test package, try to load them by replacing Test with ReTest, wrapping them in a module modname defined within parentmodule. If successful, the newly created module modname is returned and modname.runtests() should be callable.

If source::AbstractString, then it's interpreted as the top level test file (possibly including other files), and modname defaults to a name based on basename(source).

If source::Module, then it's interpreted as the name of a package, and the "test/runtests.jl" file from this package is loaded. In this case, modname defaults to Symbol(source, :Tests).

The current procedure is as follows:

1. replace toplevel using Test occurrences by using ReTest (using can have multiple arguments);
2. apply recursively these two rules:
• for all included files, provided the include statement is at the toplevel, or nested within these toplevel constructs: begin, let, for, while, if, try;
• on the content of all included modules.

When source is Base or a standard library module, a slightly different mechanism is used to find test files (which can contain e.g. non-toplevel includes), i.e. ReTest.hijack_base is used underneath.

lazy keyword

The lazy keyword specifies whether some toplevel expressions should be skipped:

• false means nothing is skipped;
• true means toplevel @test* macros are removed, as well as those defined within these toplevel (but possible nested) blocks: begin, let, for, while, if, try;
• :brutal means toplevel @test* macros are removed, as well as toplevel begin, let, for or if blocks.

include keyword

The include keyword can help to handle the case where @testsets contain include expressions, like in the following example:

@testset "parent" begin
@test true
include("file_with_other_testsets.jl")
end

This works well with Test because testsets are run immediately, as well as testsets contained in the included files, which are also recognized as children of the testset which include them. With ReTest, the include expressions would be evaluated only when the parent testsets are run, so that included testsets are not run themselves, but only "declared".

If the include keyword is set to :static, include(...) expressions are evaluated when @testset expressions containing them are parsed, before filtering and before testsets are run. Testsets which are declared (within the same module) as a side effect of include(...) are then inserted in place of the call to include(...).

If the include keyword is set to :outline, hijack inspects topelevel @testset expressions and puts toplevel include(...) expressions outside of the containing testset, and should therefore be evaluated immediately. This is not ideal, but at least allows ReTest to know about all the testsets right after the call to hijack, and to not declare new testsets when parent testsets are run.

The :outline option might be deprecated in the future, and include=:static should generally be preferred. One case where :outline might work better is when the included file defines a submodule: ReTest doesn't have the concept of a nested testset belonging to a different module than the parent testset, so the best that can be done here is to "outline" such nested testsets; with include=:outline, hijack will "process" the content of such submodules (replace using Test by using ReTest, etc.), whereas with include=:static, the subdmodules will get defined after hijack has returned (on the first call to retest thereafter), so won't be "processed".

include_functions keyword

When the include=:static keyword argument is passed, it's possible to tell hijack to apply the same treatment to other functions than include, by passing a list a symbols to include_functions. For example, if you defined a custom function custom_include(x) which itself calls out to include, you can pass include_functions=[:custom_include] to hijack.

revise keyword

The revise keyword specifies whether Revise should be used to track the test files (in particular the testsets). If true, Revise must be loaded beforehand in your Julia session. Note that this might be brittle and not work in all cases. revise defaults to true when Revise is loaded, and to false otherwise.

Julia 1.5

This function requires at least Julia 1.5.

source
ReTest.hijack_baseFunction
hijack_base(tests, [modname];
parentmodule::Module=Main, lazy=false, [revise::Bool])

Similar to ReTest.hijack, but specifically for Base and stdlib tests. tests speficies which test files should be loaded, in the exact same format as Base.runtests (i.e. it uses the same choosetests function to select the tests).

Tests corresponding to a "test/[somedir/]sometest.jl" file are loaded in the BaseTests.[somedir.]sometest module (if sometest is defined in Base, then sometest_ is used instead).

Tests corresponding to a standard library Lib are loaded in the StdLibTests.Lib_ module. When there are "testgroups", submodules are created accordingly.

If modname is specified (experimental), this will be the name of the module in which testsets are defined; passing modname is allowed only when all the loaded tests would otherwise be defined in the same second top-most module, the one under BaseTests or StdLibTests (e.g. somedir if any, or sometest otherwise, or Lib_). This unique module is then named modname, and not enclosed within BaseTests or StdLibTests.

The lazy and revise keywords have the same meaning as in ReTest.hijack. Depending on the value of lazy, some test files are skipped when they are known to fail.

Julia 1.5

This function requires at least Julia 1.5.

source