High Availability and Load Balancing

Multiple instances of RStudio Package Manager can share the same data in highly available (HA) and load-balanced configurations. In this document, we refer to these configurations as "HA" for brevity.

HA Checklist

Follow the checklist below to configure multiple RStudio Package Manager instances for HA:

  1. Ensure that all node clocks are synchronized - see the HA Time Synchronization Requirements section.
  2. Ensure that all server configurations (i.e. contents of the /etc/rstudio-pm directory) are identical.
  3. Install and configure the same version of RStudio Package Manager on each node - see the Getting Started section.
  4. Migrate to a PostgreSQL database (if running SQLite) - see the Changing the Database Provider section. All nodes in the cluster must use the same PostgreSQL database.
  5. When using NFS for shared storage, configure each server's Server.DataDir via Variable Data to point to the same shared location. Be sure to read HA Shared Directory section for additional information on the recommended settings for the shared directory. For more granular control of data directories, see the Variable Data Classes section for information on customizing the locations of each storage class.
  6. When using S3 for shared storage, each server's Server.EncryptionKeyPath must point to a file that contains the same encryption key.

HA Time Synchronization Requirements

The clocks on all nodes in an HA configuration must be synchronized. We recommend configuring NTP for clock synchronization.

HA Limitations

Node Management

RStudio Package Manager nodes in an HA configuration are not self-aware of HA. The load-balancing responsibility is fully assumed by your load balancer, and the load balancer is responsible for directing requests to specific nodes and checking whether nodes are available to accept requests.

The CLI provides limited node management capabilities. See Managing Cluster Nodes for more details.

Database Requirements

RStudio Package Manager only supports HA when using a PostgreSQL database. If you are using SQLite, please switch to PostgreSQL. See the Changing the Database Provider section for more information.

Shared Data Directory Requirements

RStudio Package Manager manages repository content within the server's data and variable data directories. These directories must be at shared locations, and each node must be configured to point to the same shared locations. See the Variable Data section for more information on the server's data directories.

RStudio Package Manager supports using either NFS or AWS S3 storage for shared data directories. You can also use a combination of both NFS and AWS S3 for different Variable Data Classes.

NFS

We recommend and support NFS version 3 or 4 for file sharing.

RStudio Package Manager relies on being able to efficiently detect new files inside of the NFS-shared DataDir. By default, NFS clients are configured to cache responses for up to 60 seconds, which means that it can take up to a minute before an RStudio Package Manager service is able to respond to certain requests. For most deployments, this is an unacceptably long delay.

Therefore, we strongly recommend that you modify your NFS client settings for the mount on which you'll be hosting your DataDir. Typically, the best way to accomplish this is to set lookupcache=pos for your NFS mount, which will allow existing files to be cached but will contact the NFS server directly to check for the existence of new files. If this setting is not acceptable for your mount, you could alternatively consider shortening acdirmax or actimeo so that your client becomes aware of new files within, say, 5 seconds, instead of the default of 60.

S3

When using S3 for shared storage, each server's Server.EncryptionKeyPath must point to a file that contains the same encryption key. See also the Server Configuration section in the appendix. The easiest way to ensure a consistent encryption key on all nodes is to start RStudio Package Manager on one of the nodes and then copy the key file created at /var/lib/rstudio-pm/rstudio-pm.key to the same location on the other nodes. Set each key's file mode to 0600.

Please refer to the Data Destinations section for information on configuring RStudio Package Manager to store variable data on S3. For help configuring your server with the credentials and settings you need to interact with S3, see the S3 Configuration section.

Managing Cluster Nodes

The admin CLI provides limited node management capabilities. You can list nodes, take nodes offline, and bring offline nodes back online.

Listing Nodes

To enumerate nodes in your cluster, run the following command.

Terminal

rspm cluster nodes

Each line of the response includes:

  • The node hostname. The hostname corresponds to the Server.HostName property. If Server.HostName is not set, then the server's hostname will be used.
  • The RSPM version used by the node.
  • The mode (offline/online) for the node.

Changing Offline/Online Mode

To take nodes offline or bring them back online, use the rspm cluster offline and rspm cluster online commands. You must specify the nodes for the operation, and you can optionally specify a timeout in seconds.

Note

The admin CLI also supports the commands rspm offline and rspm online, which can be used to take a single RSPM instance offline or to bring it back online. These commands only affect the instance at which you issue the command. See Online and Offline Modes for more details.

Terminal

# Take Node1 and Node2 offline.
rspm cluster offline --nodes=Node1,Node2

# Bring Node1 and Node2 back online.
rspm cluster online --nodes=Node1,Node2

# Bring Node3 online with a 5 minute timeout.
rspm cluster online --nodes=Node3 --timeout=300

When the rspm offline and rspm online commands complete, the nodes will be listed.

Upgrading a Cluster

Upgrade Steps

To reduce downtime during cluster upgrades, we allow specific nodes to be taken offline for upgrading. This provides the ability to always have multiple nodes running to maintain high availability.

Note

Take all nodes offline before bringing any upgraded nodes back online to avoid version mismatches. If you forget to take any non-upgraded nodes offline when bringing an upgraded node back online, the non-upgraded nodes will be using a binary that expects an earlier schema version and will be subject to unexpected and potentially serious errors. These nodes will detect an out-of-date database schema within 30 seconds and shut down automatically.

To upgrade a cluster with minimal downtime, follow these steps:

  1. Take one or more nodes offline using the rspm cluster offline command. See Managing Cluster Nodes for more details.
  2. Upgrade the offline nodes.
  3. Take the remaining nodes offline.
  4. Bring the upgraded nodes back online.
  5. Upgrade the remaining nodes and bring them back online.

Example Upgrade

Below is an example procedure for upgrading a 4-node cluster. We assume the nodes are named Node1, Node2, etc.

Terminal

# Take nodes 1 and 2 offline
rspm cluster offline --nodes=Node1,Node2

# Upgrade nodes 1 and 2
ssh node1
sudo gdebi rstudio-pm_1.1.6.1-5_amd64.deb
exit
ssh node2
sudo gdebi rstudio-pm_1.1.6.1-5_amd64.deb
exit

# Take nodes 3 and 4 offline
rspm cluster offline --nodes=Node3,Node4

# Bring nodes 1 and 2 back online. Use a long timeout after
# upgrades to allow for database migrations.
rspm cluster online --nodes=Node1,Node2 --timeout=300

# Upgrade nodes 3 and 4
ssh node4
sudo gdebi rstudio-pm_1.1.6.1-5_amd64.deb
exit
ssh node3
sudo gdebi rstudio-pm_1.1.6.1-5_amd64.deb
exit

# Bring nodes 3 and 4 back online
rspm cluster online --nodes=Node3,Node4

Downgrading

If you wish to move from an HA environment to a single-node environment, please follow these steps:

  1. Stop all RStudio Package Manager services on all nodes.
  2. Reconfigure your network to route traffic directly to one of the nodes, unless you wish to continue using a load balancer.
  3. If you wish to move all shared file data to the node, configure the server's Server.DataDir to point to a location on the node, and copy all the data from the NFS share to this location. See the Variable Data section for more information.
  4. If you wish to move the databases to this node, install PostgreSQL on the node and copy the data. Moving the PostgreSQL databases from one server to another is beyond the scope of this guide. Please note that we do not support migrating from PostgreSQL to SQLite.
  5. Start the RStudio Package Manager process.