Troubleshooting Launcher and Kubernetes:
Verify NFS Server Configuration#
Symptoms#
- New remote sessions in Kubernetes appear to start, but are stuck in PENDING status
- New remote sessions in Kubernetes appear to start, then immediately show SUSPENDED status
Error messages#
When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:
Workbench Home Page - Session Info Dialog Box
Cluster Kubernetes
Status Unable to mount volumes for pod "session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf_rstudio(648baae4-83c1-46a2-a893-e3afa007b799)": timeout expired waiting for volumes to attach or mount for pod "rstudio"/"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf". list of unmounted volumes=[mount0]. list of unattached volumes=[mount0 default-token-cxqwp]
Workbench Home Page - Session Info Dialog Box
Cluster Kubernetes
Status MountVolume.SetUp failed for volume "mount0" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/9faacd08-7c04-49b8-9bc2-f0662c790350/volumes/kubernetes.io~nfs/mount0 --scope -- mount -t nfs 172.31.30.472:/home/rstudio /var/lib/kubelet/pods/9faacd08-7c04-49b8-9bc2-f0662c790350/volumes/kubernetes.io~nfs/mount0 Output: Running scope as unit run-7219.scope. mount.nfs: Failed to resolve server 172.31.30.472: Name or service not known mount.nfs: Operation already in progress
Workbench Home Page - Session Log
04 May 2020 06:34:25 [rserver-launcher] Scanning container for R versions... 04 May 2020 06:34:27 [rserver-launcher] Found 1 R versions 04 May 2020 06:34:27 [rserver-launcher] R version: 3.6.3 - /opt/R/3.6.3/lib/R 04 May 2020 06:34:27 [rserver-launcher] Setting R environment for R version: 3.6.3 04 May 2020 06:34:27 [rserver-launcher] R_HOME=/opt/R/3.6.3/lib/R 04 May 2020 06:34:27 [rserver-launcher] R_SHARE_DIR=/opt/R/3.6.3/lib/R/share 04 May 2020 06:34:27 [rserver-launcher] R_INCLUDE_DIR=/opt/R/3.6.3/lib/R/include 04 May 2020 06:34:27 [rserver-launcher] R_DOC_DIR=/opt/R/3.6.3/lib/R/doc 04 May 2020 06:34:27 [rserver-launcher] LD_LIBRARY_PATH=/opt/R/3.6.3/lib/R/lib: 04 May 2020 06:34:27 [rserver-launcher] RSTUDIO_R_VERSION_LABEL= 04 May 2020 06:34:27 [rserver-launcher] RSTUDIO_R_MODULE= 04 May 2020 06:34:27 [rserver-launcher] RSTUDIO_R_PRELAUNCH_SCRIPT= 04 May 2020 06:34:27 [rserver-launcher] Creating container user and group... 04 May 2020 06:34:27 [rserver-launcher] Successfully created user and group 04 May 2020 06:34:27 [rserver-launcher] User dirs: /home/rstudio /home/rstudio/.rstudio Invalid session scope 1 - is the user's home drive properly mounted and writeable by both RStudio Server and the session?
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 06:28:08 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":240,"sequences":[{"seqId":7,"requestId":262}],"id":"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rwlj","status":"Pending","statusMessage":"MountVolume.SetUp failed for volume \"mount0\" : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/648baae4-83c1-46a2-a893-e3afa007b799/volumes/kubernetes.io~nfs/mount0 --scope -- mount -t nfs 172.31.30.46:/home/rstudio /var/lib/kubelet/pods/648baae4-83c1-46a2-a893-e3afa007b799/volumes/kubernetes.io~nfs/mount0\nOutput: Running scope as unit run-7507.scope.\nmount.nfs: No route to host\n\n","name":"Session 03872cfc78a313505c20c (rstudio) - RStudio Session"}
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 06:27:59 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"ADDED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf.160bbf5849d894ee","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf.160bbf5849d894ee","uid":"80fbbb7e-2bfb-4057-af48-2692286c8d5c","resourceVersion":"20181","creationTimestamp":"2020-05-04T06:27:59Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf","uid":"648baae4-83c1-46a2-a893-e3afa007b799","apiVersion":"v1","resourceVersion":"19971"},"reason":"FailedMount","message":"Unable to mount volumes for pod \"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf_rstudio(648baae4-83c1-46a2-a893-e3afa007b799)\": timeout expired waiting for volumes to attach or mount for pod \"rstudio\"/\"session-03872cfc78a313505c20c-rstudio---rstudio-session-9rrpvcf\". list of unmounted volumes=[mount0]. list of unattached volumes=[mount0 default-token-cxqwp]","source":{"component":"kubelet","host":"ip-172-31-18-187.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T06:27:59Z","lastTimestamp":"2020-05-04T06:27:59Z","count":1,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 06:34:44 [rstudio-kubernetes-launcher] Received watchPodLogs chunk: Invalid session scope 1 - is the user's home drive properly mounted and writeable by both RStudio Server and the session?|||
04 May 2020 06:34:44 [rstudio-kubernetes-launcher] Queueing response: {"messageType":5,"requestId":22,"responseId":55,"complete":false,"outputType":"mixed","output":"Invalid session scope 1 - is the user's home drive properly mounted and writeable by both RStudio Server and the session?\n","seqId":16}
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 12:45:01 [rstudio-kubernetes-launcher: ERROR jsonrpc error 9 (Parameter type mismatch) [description=expected '<String>' got '<null>']; OCCURRED AT: rstudio::core::Error rstudio::core::rapid_json::readObject(const Object&, const string&, T*) [with T = std::basic_string<char> rstudio::core::rapid_json::Object = rstudio::core::rapid_json::Value::Object std::string = std::basic_string<char>] /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/json/RapidJsonRpc.hpp:437; LOGGED FROM: void rstudio::job_launcher::impls::kubernetes::KubernetesJobStatusWatcher::onEvent(const Object&, uint_least64_t) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesJobStatusWatcher.cpp:342
Possible cause#
For Workbench, Launcher, and Kubernetes to function properly, you should have user's home directories exported via an NFS server and have this NFS server configured in the Launcher mounts configuration.
Error messages in Launcher sessions related to invalid session scope or failed volume mounts are typically due to a misconfigured NFS server or misconfigured Launcher mounts.
The following troubleshooting steps will help you confirm that your NFS server is configured correctly for use with Launcher and Kubernetes.
Troubleshooting steps#
Refer to the requirements for an NFS server in the documented steps for integrating Workbench with Kubernetes
In particular, ensure that:
- You are exporting the
/homedirectory from the NFS server - You have mounted the NFS shared at
/homeon the machine with RStudio Server Pro - User IDs and group IDs match across the NFS server and the machine with Workbench and Launcher
You can verify that the /home directory is mounted by running the following command on the machine with Workbench:
Terminal
$ mount -l | grep nfs
which should return output similar to the following if the /home directory is properly mounted from NFS:
172.31.0.123:/home on /home type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.31.6.220,local_lock=none,addr=172.31.0.123)
Verify that the contents of your Launcher mounts configuration file is similar to the following:
File: /etc/rstudio/launcher-mounts
# Required home directory mount for RSP, Launcher, and Kubernetes
Host: 172.31.30.47
Path: /home/{USER}
MountPath: /home/{USER}
ReadOnly: false
Cluster: Kubernetes
The Host setting should be configured with the IP address of your NFS server. The Path and MountPath contain the special variable {USER} to indicate that the user’s name will be substituted when the container starts, so there is no need to change that variable in this configuration file.
Restart services and test#
After verifying that your NFS server is configured correctly in terms of exports/mounts and updating your Launcher mounts configuration file (if needed), restart the Workbench and Launcher services:
Terminal
$ sudo rstudio-server restart
$ sudo rstudio-launcher restart
Verify that the services are running and try to start a new session from the Workbench home page.
If you are still experiencing errors when starting a new session, then proceed to
Step 16 - Verify NFS Server Connectivity.