Conda and Bioconda Tutorial: Install Bioinformatics Tools in Environments

Chapter 4 of 5: The solver, environment files and reproducibility

How the solver decides

When you ask for fastqc multiqc, conda does not just download two files. It must choose one version of every package in the environment (often 100+) so that every dependency rule is satisfied at the same time. This is a hard logic puzzle, and the part of conda that solves it is the solver. Since conda 23.10 the default solver is libmamba, built on the fast libsolv library; before that, conda used a slower “classic” solver.

When no combination fits, conda stops before changing anything and prints a conflict report such as LibMambaUnsatisfiableError. That is a feature: a broken environment is worse than no environment. The usual fix is to split the tools into two environments, or relax a pin.

# Simplified dependency solving
VARIABLES: one boolean per candidate build (from channels + subdir)
CLAUSES:
  request:      at least one build of fastqc AND of multiqc
  one-version:  at most one build of each package name
  dependencies: if build B is chosen -> one of B's allowed deps is chosen
  system:       respect virtual packages (__glibc, __osx, __cuda ...)
  pins:         respect pinned and already-installed specs
SOLVE: find an assignment where every clause is true
OPTIMISE (in order): channel priority -> newest versions ->
                     highest build numbers -> fewest packages changed
if no assignment exists:
    report the chain of conflicting requirements; change nothing

Step 10: Write an environment file

Why: typing install commands from memory is not reproducible. An environment.yml file is a short text recipe for an environment. You keep it in your project folder (and in Git), and anyone can rebuild the environment from it with one command.

Create a file called environment.yml with any text editor (for example nano environment.yml) and write:

name: align
channels:
  - conda-forge
  - bioconda
  - nodefaults
dependencies:
  - bwa=0.7.18
  - samtools=1.21
  - fastp

• name: the environment name used by default.
• channels: in priority order, highest first. nodefaults is a special entry that stops conda adding the defaults channel from someone else’s settings.
• dependencies: one package per line, with optional version specs as in the pinning table. Pin the tools whose numbers appear in your results; leave libraries free.

The versions above are examples. Check what is current with conda search bwa before you pin. Now build the environment from the file:

conda env create -f environment.yml
conda activate align

After editing the file (for example adding a tool), bring the environment up to date:

conda env update -f environment.yml --prune

• -f: the file to read.
• --prune: also remove packages that are no longer listed, so the environment matches the file exactly.

If you need a Python package that exists only on PyPI, install it through the same file so that conda knows about it. Put pip in the list and add a pip: block at the end. Use this as a last resort; prefer conda packages:

dependencies:
  - python=3.12
  - pip
  - pip:
      - some-pypi-only-package

(some-pypi-only-package is a placeholder for the real name.)

Step 11: Export an environment you already built

If you installed things interactively, you can turn the environment into a file afterwards. There are three useful kinds of export, from most portable to most exact:

conda env export -n qc --from-history > environment.yml   # only what you asked for
conda env export -n qc --no-builds > environment-full.yml # every package + version
conda list -n qc --explicit > explicit-linux-64.txt       # exact download URLs (a lock file)

Recreate from an explicit file with conda create -n qc-copy --file explicit-linux-64.txt. Recent conda versions also have a newer conda export command with a --format option (for example conda export --format=explicit); the commands above keep working.

Best practice: commit both environment.yml (from history, human-readable, portable) and an explicit lock file for the platform where you ran the analysis.

Step 12: Use conda inside workflow managers and containers

Real pipelines rarely activate environments by hand. Workflow managers create them for you from small files, one per step:

• Snakemake: add conda: "envs/qc.yaml" to a rule and run with snakemake --use-conda (newer versions also accept --software-deployment-method conda). Snakemake builds and caches one environment per YAML file.
• Nextflow / nf-core: a process can declare conda 'bioconda::fastqc=0.12.1'; nf-core pipelines are run with -profile conda, -profile docker or -profile singularity.
• BioContainers: for every Bioconda package, a Docker image is built automatically and published at quay.io/biocontainers, and Singularity/Apptainer copies are served from depot.galaxyproject.org/singularity. Image tags look like <version>--<build>; look up the exact tag for your tool on quay.io before you pull it.

The Reproducible Pipelines with Nextflow, Snakemake, Docker and HPC course covers these tools in depth.