Frequently Asked Questions
How do I use
Conda
Conda distributions, such as miniconda or anaconda, must be located on either shared storage or temporary local storage on the node. See: Storage on Slurm
Pip
Packages installed with pip are automatically located in temporary local storage on the node. To install packages to shared storage, you must specify the install location with the --target option, and to use these packages you must add that location to your $PYTHONPATH environment variable.
Example:
pip install --target /data/scratch/$USER
export PYTHONPATH=/data/scratch/$USER:$PYTHONPATH
X-Forwarding
X-Forwarding to a compute node requires SSH. See: SSH
VSCode
Connecting VSCode to a compute node requires SSH. See: SSH
Docker Containers
Use Apptainer (formerly Singularity) instead. You’ll need the --nv flag to run your container with GPU access.
The Docker Engine is not compatible with Slurm, as it would bypass Slurm’s ability to limit a process’s resources.
Apptainer is installed on our compute nodes; it is configured to run as an unprivileged user and to pass through shared directories such as /data and /tmp.
See the Apptainer documentation and also the [Apptainer Docker compatibility page] for detailed information about using Singularity with Docker containers.
Why can’t I access files in my AFS home directory?
Your AFS tokens can’t pass through to the system where your job is being run, so any files being used when you submit won’t be available to the job (unless the AFS directory is world-readable). See: https://tig.csail.mit.edu/shared-computing/slurm/storage/
What are the best practices for transferring data to my compute node?
Use sbcast in an sbatch script to transfer your files to the /tmp directory of a compute node before running a task.
If your files are very large and cannot be staged in this way, another solution is to use our NFS filesystems, which are available from all nodes and mounted at /data.
Anyone can create a directory on /data/scratch, but we don’t guarantee data stored here won’t disappear. If you need something more stable, refer to our NFS documentation to request a filesystem.
Do not use the compute nodes for permanent storage. These filesystems are not backed up and may be purged without notice.
Can you please install ${SOFTWARE} for me?
Possibly. Please send your request to help@csail.mit.edu.