11 Job Launcher

11.1 Overview

The RStudio Job Launcher provides the ability for various RStudio applications, such as RStudio Server Pro and RStudio Connect, to start processes within various batch processing systems (e.g. IBM Spectrum LSF) and container orchestration platforms (e.g. Kubernetes). RStudio Server Pro integrates with the Job Launcher to allow you to run your R Sessions within your compute cluster software of choice, and allows you to containerize your sessions for maximum process isolation and operations efficiency. Furthermore, users can submit standalone jobs to your compute cluster(s) to run computationally expensive R or Python scripts.

11.2 Configuration

11.2.1 Job Launcher Configuration

Before the Job Launcher can be run, it must be properly configured via the config file /etc/rstudio/launcher.conf; see the Job Launcher documentation for supported configuration options.

The Launcher configuration parameter admin-group should be configured to the group value of the RStudio Server Pro server user, specified in the server-user configuration paremter in rserver.conf (which defaults to rstudio-server). This makes the server user a Job Launcher admin, which is necessary to properly launch sessions on behalf of other users.

11.2.2 RStudio Server Pro Integration

RStudio Server Pro must be configured in order to integrate with the Job Launcher. There are several files which house the configuration, and they are described within subsequent sections.

11.2.2.1 Server Configuration

The RStudio Server process rserver must be configured to communicate with the Job Launcher in order to enable session launcing. The following table lists the various configuration options that are available to be specified in the rserver.conf configuration file:

/etc/rstudio/rserver.conf

Config Option Description Required (Y/N) Default Value
launcher-sessions-enabled Enables launching of rsession processes via the Job Launcher. This must be enabled to use the Job Launcher. N 0
launcher-address TCP host/IP of the launcher host, or unix domain socket path (must match `/etc/rstudio/launcher.conf configuration value) Y
launcher-port Port that the launcher is listening on. Only required if not using unix domain sockets. Y
launcher-default-cluster Name of the cluster to use when launching sessions. Can be overridden by the launching user. Y
launcher-sessions-callback-address Address (http or https) of RStudio Server Pro that will be used by launcher sessions to communicate back for project sharing features. This is only required if using plugins that do not use containers, as project sharing is not yet supported for containerized sessions. Y
launcher-use-ssl Whether or not to connect to the launcher over HTTPS. Only supported for connections that do not use unix domain sockets. N 0
launcher-sessions-clusters Whitelist of clusters to allow for submitting interactive session jobs to. The default allows all job launcher clusters to run interactive sessions. N
launcher-sessions-container-image The default container image to use when creating sessions. Only required if using a plugin that requires containerization. If none is specified, the Job launcher-specified default will be used, if the plugin supports it. N
launcher-sessions-container-run-as-root Whether or not to run as root within the session container. N 1
launcher-sessions-create-container-user Whether or not to create the session user within the container. Only applicable if using container sessions and not running containers as root. The created user will have the same UID and GID as the user that launched the session. It is recommended that this option be used, unless your containers connect to an LDAP service to manage users and groups. N 0
launcher-sessions-connection-timeout-seconds Number of seconds to allow for making the initial connection to a launcher session. Connection failures are retried automatically - this is simply to prevent unreachable hosts from hanging the retry process as the default connection timeout on most systems is very high. Only change this if you are having trouble connecting to sessions. A value of 0 indicates that there should be no timeout (system default). N 3

For example, your rserver.conf file might look like the following:

/etc/rstudio/rserver.conf

launcher-address=localhost
launcher-port=5559
launcher-sessions-enabled=1
launcher-default-cluster=Kubernetes
launcher-sessions-callback-address=http://localhost:8787
launcher-use-ssl=1
launcher-sessions-container-image=rstudio:R-3.5
launcher-sessions-container-run-as-root=0
launcher-sessions-create-container-user=1

11.2.2.2 Authentication

RStudio Server Pro authenticates with the Job Launcher via the secure-cookie-key file, a secret key that is read on startup of both the launcher and RSP which is only readable by the root account. The file is present at /etc/rstudio/secure-cookie-key. If the Job Launcher is running on a different machine than RSP, you will need to make sure that the exact same secure-cookie-key file is present on both machines.

To do this, create a secure cookie key file on one of the nodes like so:

# generate secure-cookie-key as a simple UUID
sudo uuid > /etc/rstudio/secure-cookie-key

# ensure that the cookie is only readable by root
sudo chmod 700 /etc/rstudio/secure-cookie-key

Once this file has been created, copy it to the other node to the same location so that both services use the same key. Alternatively, you could accomplish this via a symlink to a location on a file share.

The path to the secure-cookie-key file can be changed, but it is not recommended in most cases. If you need to change it, it can be done by adding the following line to the /etc/rstudio/rserver.conf and /etc/rstudio/launcher.conf configuration files:

/etc/rstudio/rserver.conf and /etc/rstudio/launcher.conf

secure-cookie-key-file=/path/to/secure-cookie-key

11.2.2.3 Containerized sessions

In order to run your R sessions in containers, you will need a Docker image that contains the necessary rsession binaries installed. RStudio provides an official image for this purpose, which you can get from Docker Hub.

For example, to get the RHEL6 image, you would run:

docker pull rstudio/r-session:centos6-latest

After pulling the desired image, you will need to create your own Dockerfile that extends from the r-session base image and adds whatever versions of R you want to be available to your users, as well as adding any R packages that they will need. For example, your Dockerfile should look similar to the following:

FROM rstudio/r-session:centos6-latest

# install desired versions of R
RUN yum install -y R

# install R packages
...
11.2.2.3.1 Launcher Mounts

When creating containerized sessions via the Job Launcher, you will need to specify mount points as appropriate to mount the users’ home drives and any other desired paths. In order for sessions to run properly within containers, it is required to mount the home directories into the containers.

To specify mount points, modify the /etc/rstudio/launcher-mounts file to consist of multiple mount entries separated by a blank line. The following table lists the fields that are available for each mount entry in the file.

Field Description Required (Y/N) Default Value
Path The source directory of the mount, i.e. where the mount data comes from. Y
Host The NFS host name for the NFS mount. Only used if the mount is NFS. N
MountPath The path within the container that the directory will be mounted to. Y
ReadOnly Whether or not the mount is read only. Can be true or false. N false

Additionally, paths may contain the special variable {USER} to indicate that the user’s name be substituted, enabling you to mount user-specific paths.

An example /etc/rstudio/launcher-mounts file is shown below.

/etc/rstudio/launcher-mounts

# User home mount - This is REQUIRED for the session to run
Host: nfs01
Path: /home/{USER}
MountPath: /home/{USER}
ReadOnly: false

# Shared code mount
Host: nfs01
Path: /dev64
MountPath: /code
ReadOnly: false

It is important that each entry consists of the fields as specified above. Each field must go on its own line. There should be no empty lines between field definitions. Each entry must be separated by one full blank line (two new-line \n characters).

If you choose to run your containers as root, the user home drive must be mapped to /root. For example:

/etc/rstudio/launcher-mounts

Host: nfs01
Path: /home/{USER}
MountPath: /root
ReadOnly: false

It is recommended that you also mount the Shared Storage path (see Shared Storage for configuration) into the session container to support various RSP features. Failure to do so could cause subtle, unintended issues. When mounting the shared storage path, ensure that the folder is mounted to the same path within the container to ensure that the rsession executable will correctly find it. For example:

/etc/rstudio/launcher-mounts

Host: nfs01
Path: /rstudio/shared-storage
MountPath: /rstudio/shared-storage
ReadOnly: false
11.2.2.3.2 Launcher Environment

You may optionally specify environment variables to set when creating containerized sessions.

To specify environment variables, modify the /etc/rstudio/launcher-env file to consist of KEY=VALUE pairs, one per line. Additionally, you can use the special {USER} variable to specify the value of the launching user’s username, similar to the mounts file above.

An example /etc/rstudio/launcher-env file is shown below.

/etc/rstudio/launcher-env

VAR1=VAL1
USER_HOME=/home/{USER}
11.2.2.3.3 Launcher Ports

You may optionally specify ports that should be exposed when creating containerized sessions. This will allow the ports to be exposed within the host running the container, allowing the ports to be reachable from external services. For example, for Shiny applications to be usable, you must expose the desired Shiny port, otherwise the browser window will not be able to connect to the Shiny application running within the container.

To specify ports, modify the /etc/rstudio/launcher-ports file to consist of port numbers, one per line.

An example /etc/rstudio/launcher-ports file is shown below.

/etc/rstudio/launcher-ports

5873
5874
64234
64235

11.3 Running the Launcher

Once it is configured, you can run the Job Launcher by invoking the command sudo rstudio-launcher start, and stop it with sudo rstudio-launcher stop. The Job Launcher must be run with root privileges, but similar to rstudio-server, privileges are immediately lowered. Root privileges are used only to impersonate users as necessary.

11.4 Creating plugins

Plugins allow communication with specific batch cluster / container orchestration systems like Platform LSF and Kubernetes. However, you may be using a system that RStudio does not natively support. Fortunately, the Job Launcher provides a fairly simple means of creating custom plugins that can allow you to spawn jobs on any cluster software you desire.

Documentation for creating plugins can be found here.