Virtual Environments

A virtual environment is an isolated work space that contains its own installation (and desired version) of software (like Python or R) and a set of packages. This ensures that:

Projects do not interfere with each other.
Specific versions of packages can be maintained for reproducibility.
Experiments and scripts run in a controlled, predictable setup.

Whatever you install inside the environment won’t affect the global system installation or other projects. It is like a “sandbox” for your project, that can be reproduced on other systems.

Setup

Miniforge3 is a lightweight alternative to Anaconda that simplifies installation and avoids some licensing issues. Libmamba solver is much faster and more reliable than the classic Conda solver, especially for large environments with many dependencies. Using the old solver can take hours or even fail due to conflicts.

miniforge3 is installed via Miniforge3-Linux-x86_64.sh Important: Solver has to be set to libmamba. Check via conda info and change like this if needed:

conda info | grep solver
conda config --set solver libmamba

Installation

Set the desired software/packages and if needed versions in environment.yml. All undefined Versions will be fitted to work together. Create the Virtual Environment with the specified requirements.

Tip: Installing R inside the environment avoids mismatches between Python and R libraries and ensures your scripts are reproducible across machines. See example yml

mamba env create -f environment.yml

Check after installation

To see all installed software and packages with their exact version use conda list or filter for a specific item

conda list

conda list | grep numpy

Deployment

After activating the Virtual environment and changing into the correct repository we are ready to deploy. There are two methods, direct deployment and via a bash script deploying in parallel.

conda activate gchm
cd /home/emilio/canopy_height/

Direct deployment of the full R script.

Rscript deploy.R

Deployment via a bash script, passing arguments to R and deploying parallel tile Groups.

chmod +x deploy_all.sh
./deploy_all.sh

Usefull bash code

Real time GPU supervision:

nvtop
watch -n 1 nvidia-smi

See running processes on the mashine.

htop

Example environment

The environment.yml file contains all info about what we want to install and tells conda our requirements. The solver then sorts out the correct versions for the virtual environment to work with each other. It works from top to bottom. gdal is included here; the version will fit to the required python and R. We included R here as well to have a fully encapsulated environment ensuring compatibility, instead of deploying scripts from outside the branch. -e. is added to install it as executable python “package”.

name: gchm

channels:
  - conda-forge
  - pytorch
  - nvidia

dependencies:
  - python>=3.12.0
  - r-base>=4.4.0
  - gdal
  - sentinelhub
  - pytables
  - libgdal-jp2openjpeg
  - ipython
  - matplotlib
  - numpy
  - pathlib
  - tqdm
  - botocore
  - urllib3
  - wandb
  - tensorboard
  - scikit-image
  - scikit-learn
  - pytorch::pytorch
  - pytorch::pytorch-cuda
  - pytorch::torchvision
  - pytorch::torchaudio
  - conda-forge::cudatoolkit==11.8.0
  - typing
  - jupyter
  - pip
  - pip:
    - -e .