Integrating RStudio Server Pro with Slurm

Info

Launcher is a new feature of RStudio Server Pro 1.2 that is only available under named user licensing. RStudio Server Pro 1.2 without Launcher is available under existing server-based licensing. For questions about using Launcher with RStudio Server Pro, please contact sales@rstudio.com.

Overview

These steps describe how to integrate RStudio Server Pro with Launcher and Slurm.

In this configuration, the RStudio Server Pro and Launcher services will be installed to one node in the Slurm cluster, and the RStudio Server Pro Session Components will be installed on all other Slurm nodes.

Prerequisites

  • Slurm cluster
  • Home directories mounted on shared file storage on all Slurm nodes with matching user and group IDs

Pre-Flight Configuration Checks

Verifying active Slurm compute nodes

On a machine with Slurm configured, ensure that you have one or more worker nodes that are ready to accept jobs as part of the Slurm cluster by running the following command:

Terminal

$ sinfo

Verifying functionality with a test job

On a machine with Slurm configured, ensure that you are able to deploy a sample jobs to your Slurm cluster by running a test command:

Terminal

$ srun date

You can also create a sample script called submit.sh:

File: submit.sh

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=res.txt
#
#SBATCH --ntasks=1
#SBATCH --time=10:00

srun hostname
srun sleep 60

Then submit the job by running the following command:

Terminal

$ sbatch submit.sh

And verify that it runs successfully on the Slurm cluster.

1. Install RStudio Server Pro on one Slurm node

You can install RStudio Server Pro on any Slurm node that has access to the Slurm tooling such as sbatch, srun, sinfo, etc. This can be a Slurm login/submission node, controller node, or compute node.

Note that you will only need to install RStudio Server Pro on one node. In a later step, you will install the RStudio Server Pro Session Components on other Slurm nodes.

2. Configure RStudio Server Pro with Launcher

On the node where RStudio Server Pro is installed, add the following lines to the RStudio Server Pro configuration file:

File: /etc/rstudio/rserver.conf

# Launcher Config
launcher-address=127.0.0.1
launcher-port=5559
launcher-sessions-enabled=1
launcher-default-cluster=Slurm
launcher-sessions-callback-address=http://<SERVER-ADDRESS>:8787

In the launcher-sessions-callback-address setting, you should replace <SERVER-ADDRESS> with the hostname or IP address of RStudio Server Pro. You should also change the protocol and port if you are using HTTPS or a different port.

The launcher-sessions-callback-address needs to be reachable from the Slurm compute nodes to the instance of RStudio Server Pro.

3. Configure Launcher settings and plugins

On the node where RStudio Server Pro is installed, add the following lines to the Launcher configuration file:

File: /etc/rstudio/launcher.conf

[server]
address=127.0.0.1
port=5559
server-user=rstudio-server
admin-group=rstudio-server
authorization-enabled=1
thread-pool-size=4
enable-debug-logging=1

[cluster]
name=Slurm
type=Slurm

4. Configure profile for Launcher Slurm plugin

On the node where RStudio Server Pro is installed, add the following lines to the Launcher profiles configuration file:

File: /etc/rstudio/launcher.slurm.profiles.conf

[*]
default-cpus=1
default-mem-mb=512
max-cpus=2
max-mem-mb=1024

5. Configure Launcher with Slurm

On the node where RStudio Server Pro is installed, add the following lines to the Launcher Slurm configuration file:

File: /etc/rstudio/launcher.slurm.conf

slurm-service-user=slurm
enable-debug-logging=1

The slurm-service-user should match the Slurm service user in your cluster.

6. Install RStudio Server Pro Session Components on other Slurm nodes

On each Slurm compute node in the cluster (where you did not install RStudio Server Pro), you will need to install the RStudio Server Pro session components to be able to start R sessions via Slurm.

sudo yum install libcurl-devel libuser-devel openssl-devel rrdtool

curl -O https://s3.amazonaws.com/rstudio-ide-build/session/centos6/rsp-session-centos6-1.2.5001-3.tar.gz
sudo mkdir -p /usr/lib/rstudio-server
sudo tar -zxvf ./rsp-session-centos6-1.2.5001-3.tar.gz -C /usr/lib/rstudio-server/
sudo mv /usr/lib/rstudio-server/rsp-session*/* /usr/lib/rstudio-server/
sudo rm -rf /usr/lib/rstudio-server/rsp-session*
rm -f ./rsp-session-centos6-1.2.5001-3.tar.gz
sudo apt-get install curl libcurl4-gnutls-dev libssl1.0.0 libssl-dev libuser libuser1-dev rrdtool

curl -O https://s3.amazonaws.com/rstudio-ide-build/session/trusty/rsp-session-trusty-1.2.5001-3.tar.gz
sudo mkdir -p /usr/lib/rstudio-server
sudo tar -zxvf ./rsp-session-trusty-1.2.5001-3.tar.gz -C /usr/lib/rstudio-server/
sudo mv /usr/lib/rstudio-server/rsp-session*/* /usr/lib/rstudio-server/
sudo rm -rf /usr/lib/rstudio-server/rsp-session*
rm -f ./rsp-session-trusty-1.2.5001-3.tar.gz
sudo apt-get install rrdtool libuser libuser1-dev

curl -O https://s3.amazonaws.com/rstudio-ide-build/session/debian9/rsp-session-debian9-1.2.5001-3.tar.gz
sudo mkdir -p /usr/lib/rstudio-server
sudo tar -zxvf ./rsp-session-debian9-1.2.5001-3.tar.gz -C /usr/lib/rstudio-server/
sudo mv /usr/lib/rstudio-server/rsp-session*/* /usr/lib/rstudio-server/
sudo rm -rf /usr/lib/rstudio-server/rsp-session*
rm -f ./rsp-session-debian9-1.2.5001-3.tar.gz
sudo zypper install libuser-devel

curl -O https://s3.amazonaws.com/rstudio-ide-build/session/opensuse/rsp-session-opensuse-1.2.5001-3.tar.gz
sudo mkdir -p /usr/lib/rstudio-server
sudo tar -zxvf ./rsp-session-opensuse-1.2.5001-3.tar.gz -C /usr/lib/rstudio-server/
sudo mv /usr/lib/rstudio-server/rsp-session*/* /usr/lib/rstudio-server/
sudo rm -rf /usr/lib/rstudio-server/rsp-session*
rm -f ./rsp-session-opensuse-1.2.5001-3.tar.gz

7. Restart RStudio Server Pro and Launcher Services

Terminal

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

8. Test RStudio Server Pro with Launcher and Slurm

In your browser, navigate to the RStudio Server Pro interface and log in.

Select New Session, then click the Start Session button.

You can then use the RStudio Session, which will be running as a Slurm job.

Additional Documentation

For more information on RStudio Server Pro and Launcher, refer to the following reference documentation:

Troubleshooting RStudio Server Pro and Slurm

Refer to the support article on Troubleshooting Launcher and Slurm in RStudio Server Pro for additional information on troubleshooting RStudio Server Pro with Launcher and Slurm.