GitHub

Persistent Tasks

Motivation

Julia 1.10 and higher wait for all running Tasks to finish before writing out the precompiled (cached) version of the package. One consequence is that a package that launches Tasks in its __init__ function may precompile successfully, but block precompilation of any packages that depend on it.

Example

Let's create a dummy package, PkgA, that launches a persistent Task:

module PkgA
const t = Ref{Any}()   # to prevent the Timer from being garbage-collected
__init__() = t[] = Timer(0.1; interval=1)   # create a persistent `Timer` `Task`
end

PkgA will precompile successfully, because PkgA.__init__() does not run when PkgA is precompiled. However,

module PkgB
using PkgA
end

fails to precompile: using PkgA runs PkgA.__init__(), which leaves the Timer Task running, and that causes precompilation of PkgB to hang.

Example with expr

You can test that an expression using your package finishes without leaving any persistent tasks by passing a quoted expression:

Aqua.test_persistent_tasks(MyPackage, quote
    # Code to run after loading MyPackage
    server = MyPackage.start_server()
    MyPackage.stop_server!(server)
end)

Here is an example test with a dummy expr which will obviously fail, because it's explicitly spawning a Task that never dies.

julia> using Aqua
julia> Aqua.test_persistent_tasks(Aqua, expr = quote
           Threads.@spawn while true sleep(0.5) end
       end)Test Failed at /home/runner/work/Aqua.jl/Aqua.jl/src/persistent_tasks.jl:38
  Expression: !(has_persistent_tasks(package; kwargs...))

ERROR: There was an error during testing

How the test works

This test works by launching a Julia process that tries to precompile a dummy package similar to PkgB above, modified to signal back to Aqua when PkgA has finished loading. The test fails if the gap between loading PkgA and finishing precompilation exceeds time tmax.

How to fix failing packages

Often, the easiest fix is to modify the __init__ function to check whether the Julia process is precompiling some other package; if so, don't launch the persistent Tasks.

function __init__()
    # Other setup code here
    if ccall(:jl_generating_output, Cint, ()) == 0   # if we're not precompiling...
        # launch persistent tasks here
    end
end

In more complex cases, you may need to set up independently-callable functions to launch the tasks and set conditions that allow them to cleanly exit.

Test functions

Aqua.test_persistent_tasks — Function
Aqua.test_persistent_tasks(package)

Test whether loading package creates persistent Tasks which may block precompilation of dependent packages. See also Aqua.find_persistent_tasks_deps.

If you provide an optional expr, this tests whether loading package and running expr creates persistent Tasks. For example, you might start and shutdown a web server, and this will test that there aren't any persistent Tasks.

On Julia version 1.9 and before, this test always succeeds.

Arguments

  • package: a top-level Module or Base.PkgId.

Keyword Arguments

  • broken::Bool = false: If true, it uses @test_broken instead of @test.
  • tmax::Real = 5: the maximum time (in seconds) to wait after loading the package before forcibly shutting down the precompilation process (triggering a test failure).
  • expr::Expr = quote end: An expression to run in the precompile package.
Note

Aqua.test_persistent_tasks(package) creates a package with package as a dependency and runs the precompilation process. This requires that package is instantiable with the information in the Project.toml file alone. In particular, this will not work if some of package's dependencies are deved packages or are given as a local path or a git repository in the Manifest.toml.

source