Virtual Environments

What are Virtual Environments?

A virtual environment is an isolated workspace that contains its own installation (and desired version) of software (like Python or R) and a set of packages. This ensures that:

• Projects do not interfere with each other.

• Specific versions of packages can be maintained for reproducibility.

• Experiments and scripts run in a controlled, predictable setup.

Whatever you install inside the environment won’t affect the global system installation or other projects. It is like a “sandbox” for your project, that can be reproduced on other systems.

How to set them up

Miniforge3 is a lightweight alternative to Anaconda that simplifies installation and avoids some licensing issues. Libmamba solver is much faster and more reliable than the classic Conda solver, especially for large environments with many dependencies. Using the old solver can take hours or even fail due to conflicts.

miniforge3 is installed via Miniforge3-Linux-x86_64.sh Important: Solver has to be set to libmamba. Check via conda info and change like this if needed:

conda info | grep solver
conda config --set solver libmamba

Installation

Set the desired software/packages and if needed versions in environment.yml. All undefined Versions will be fitted to work together. Create the Virtual Environment with the specified requirements.

Tip: Installing R inside the environment avoids mismatches between Python and R libraries and ensures your scripts are reproducible across machines.

mamba env create -f environment.yml

Check after installation

To see all installed software and packages with their exact version use conda list or filter for a specific item

conda list

conda list | grep numpy

Deployment

First, the desired Virtual Environment has to be activated and the desired project directory set. Then we can deploy scripts by specifying the type and then the file.

conda activate gchm
cd /home/emilio/canopy_height/
Rscript deploy.R

Usefull bash code

Real time GPU supervision:

watch -n 1 nvidia-smi

Example environment

The environment.yml file contains all info about what we want to install and tells conda our requirements. The solver then sorts out the correct versions for the virtual environment to work with each other. It works from top to bottom. gdal is included here; the version will fit to the required python and R. We included R here as well to have a fully encapsulated environment ensuring compatibility, instead of deploying scripts from outside the branch. -e. is added to install it as executable python “package”.

name: gchm

channels:
  - conda-forge
  - pytorch
  - nvidia

dependencies:
  - python>=3.12.0
  - r-base>=4.4.0
  - gdal
  - sentinelhub
  - pytables
  - libgdal-jp2openjpeg
  - ipython
  - matplotlib
  - numpy
  - pathlib
  - tqdm
  - botocore
  - urllib3
  - wandb
  - tensorboard
  - scikit-image
  - scikit-learn
  - pytorch::pytorch
  - pytorch::pytorch-cuda
  - pytorch::torchvision
  - pytorch::torchaudio
  - conda-forge::cudatoolkit==11.8.0
  - typing
  - jupyter
  - pip
  - pip:
    - -e .