Skip to content

Troubleshooting Launcher and Kubernetes:
Verify Kubernetes Worker Node Resources#

Symptoms#

  • New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages#

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box
Cluster Kubernetes
Status  0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 06:09:22 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":81,"responseId":213,"jobs":[{"id":"session-03872cfc78a315205b546-rstudio---rstudio-session-przws","name":"Session 03872cfc78a315205b546 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio:5000/r-session-complete:jammy-2023.06.2"},"status":"Canceled","lastUpdateTime":"2020-05-04T06:05:12Z","submissionTime":"2020-05-04T05:28:29Z","tags":["03872cfc78a315205b546","03872cfc78a31","5205b546","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a315205b546"]},{"id":"session-03872cfc78a315a08d07b-rstudio---rstudio-session-kkq6h","name":"Session 03872cfc78a315a08d07b (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-2023.06.2"},"status":"Running","lastUpdateTime":"2020-05-04T06:07:03Z","submissionTime":"2020-05-04T06:00:16.347857Z","tags":["03872cfc78a315a08d07b","03872cfc78a31","5a08d07b","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a315a08d07b"]},{"id":"session-03872cfc78a31c9008b91-rstudio---rstudio-session-shsl7","name":"Session 03872cfc78a31c9008b91 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-2023.06.2"},"status":"Pending","statusMessage":"0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.","lastUpdateTime":"2020-05-04T06:08:22Z","submissionTime":"2020-05-04T06:05:54.394732Z","tags":["03872cfc78a31c9008b91","03872cfc78a31","c9008b91","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a31c9008b91"]},{"id":"session-03872cfc78a31e6246906-rstudio---rstudio-session-9kc4d","name":"Session 03872cfc78a31e6246906 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-2023.06.2"},"status":"Canceled","lastUpdateTime":"2020-05-04T06:01:43Z","submissionTime":"2020-05-04T05:50:52.411649Z","tags":["03872cfc78a31e6246906","03872cfc78a31","e6246906","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a31e6246906"]}]}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Received getJobStatus request for rstudio: jobID: * cancel: false
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Adding request id 82 to job status stream for job *
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":214,"sequences":[{"seqId":1,"requestId":82}],"id":"session-03872cfc78a315205b546-rstudio---rstudio-session-przws","status":"Canceled","name":"Session 03872cfc78a315205b546 (rstudio) - RStudio Session"}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":215,"sequences":[{"seqId":2,"requestId":82}],"id":"session-03872cfc78a315a08d07b-rstudio---rstudio-session-kkq6h","status":"Running","name":"Session 03872cfc78a315a08d07b (rstudio) - RStudio Session"}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":216,"sequences":[{"seqId":3,"requestId":82}],"id":"session-03872cfc78a31c9008b91-rstudio---rstudio-session-shsl7","status":"Pending","statusMessage":"0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.","name":"Session 03872cfc78a31c9008b91 (rstudio) - RStudio Session"}

Possible cause#

For Workbench, Launcher, and Kubernetes to function properly, you should have at least one worker node in the Kubernetes cluster that is available to run pods/containers, and the node should have an adequate amount of CPU and RAM to run Launcher sessions in containers on the Kubernetes cluster.

When starting a new remote session in Kubernetes, if you experience errors in Workbench and Launcher related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of adequate CPU and RAM resources on the Kubernetes worker nodes.

The following troubleshooting steps will help you determine if the worker nodes on your Kubernetes cluster have an adequate amount of CPU and RAM to run sessions.

Troubleshooting steps#

Run the following command to verify the amount of CPU and RAM on the worker nodes in your Kubernetes cluster:

Terminal
$ kubectl describe nodes

which should return output similar to the following for each node in the Kubernetes cluster (only the relevant CPU and memory information is included here):

Capacity:
 cpu:                         4
 memory:                      15950184Ki
Allocatable:
 cpu:                         3920m
 memory:                      13204840Ki

If you are using the default CPU and RAM values for new sessions from the documentation on integrating Workbench with Kubernetes, then each session will require 1 CPU and 512 MB RAM. The default CPU and RAM values are specified using the default-cpus and default-mem-mb configuration in the Kubernetes profile configuration file.

If you do not have an adequate amount of CPU and RAM on the nodes in your Kubernetes cluster, then you should refer to the documentation for your Kubernetes cluster and add additional worker nodes to the cluster that have an adequate amount of CPU and RAM to be able to run sessions.

Test and verify#

After adding additional worker nodes that have an adequate amount of CPU and RAM to run sessions, try to start a new session from the Workbench home page.

If you are still experiencing errors when starting a new session, then proceed to
Step 15 - Verify NFS Server Configuration.