Skip to content

AsyncFlow — Installation Guide¶

AsyncFlow is a Python library built on top of asyncio, designed for orchestrating complex asynchronous workflows easily and reliably. This guide will help you install AsyncFlow in a clean Python environment.

Prerequisites¶

  • Python ≥ 3.11
  • pip ≥ 22.0
  • Optional: conda ≥ 4.10 for Conda environments

Make sure Python is installed on your system:

python --version
pip --version

It is strongly recommended to install AsyncFlow in an isolated environment (Conda or venv) to avoid conflicts with system packages.

Install with Conda¶

Create and activate a Conda environment:¶

conda create -n asyncflow python=3.11 -y
conda activate asyncflow

Install AsyncFlow:¶

pip install radical-asyncflow

Verify installation:¶

python -c "import radical.asyncflow; print('AsyncFlow installed ✅')"

Install with venv (built-in)¶

If you don't use Conda, you can use Python's built-in venv module.

Create and activate a virtual environment:¶

python -m venv ~/.venvs/asyncflow
source ~/.venvs/asyncflow/bin/activate

(For Windows: ~/.venvs/asyncflow/Scripts/activate)

Install AsyncFlow:¶

pip install radical-asyncflow

Verify installation:¶

python -c "import radical.asyncflow; print('AsyncFlow installed ✅')"

Install RHAPSODY for HPC Execution (optional)¶

To run workflows on HPC clusters, supercomputers, or with Dask, install RHAPSODY:

pip install rhapsody-py

RHAPSODY provides HPC execution backends (RadicalExecutionBackend, DaskExecutionBackend, DragonExecutionBackendV3, etc.) that plug directly into AsyncFlow — see Execution Backends & HPC Integration for usage, configuration, and examples.

Development Installation (optional)¶

If you want to contribute to AsyncFlow or use the latest code from GitHub:

git clone https://github.com/radical-cybertools/radical.asyncflow.git
cd radical.asyncflow
pip install -e .[dev,lint,doc]

The -e flag installs it in editable mode — any local changes you make to the code are reflected immediately.

Running the tests¶

tox -e py312                    # unit tests, including 150 random workflows, the same each run
tox -e py312-all                # integration tests
tox -e py312 -- --eager-tasks   # unit tests under asyncio.eager_task_factory (3.12+)
tox -e stress                   # scale tests and 2,000 random workflows, as nightly

Every test also checks what it leaves behind: ERROR log records, such as asyncio's "Future exception was never retrieved", and futures or records an engine keeps after shutdown(). A test that leaves any of these fails. Set ASYNCFLOW_INVARIANTS=warn (or pass --engine-invariants=warn) to report them as warnings instead, or off to skip the checks; a test marked no_engine_invariants is not checked.

benchmarks/engine_overhead.py compare --base PYTHON --head PYTHON compares the per-task cost and the memory of two installed versions, each in its own virtual environment. It exits with an error when the head makes the garbage collector run more than 5% more often than the base, which fails the benchmark job on a pull request. With --scale 40000 it runs 40,000 tasks at once instead, as an application does, and also reports their wall time and the peak RSS. The nightly benchmark compares main with the commit of the night before this way, and a failing nightly run opens an issue.

Tip

  • Always activate your virtual environment before using AsyncFlow.

  • To deactivate an environment:

    • conda deactivate (Conda)
    • deactivate (venv)
    • You can list installed packages with pip list.
    • It is a good practice to upgrade pip and setuptools:
pip install --upgrade pip setuptools

🚀 Next Steps¶