Troubleshooting Launcher and Kubernetes:
Verify Kubernetes Worker Nodes#

Symptoms#

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages#

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Status  no nodes available to schedule pods

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:52:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":0,"requestId":0,"responseId":0}
04 May 2020 05:52:24 [rstudio-kubernetes-launcher] Received getJobState request for rstudio: jobID: * startTime: none endTime: none statuses: none
04 May 2020 05:52:24 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":23,"responseId":29,"jobs":[{"id":"session-03872cfc78a315205b546-rstudio---rstudio-session-przws","name":"Session 03872cfc78a315205b546 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio:5000/r-session-complete:jammy-2023.06.2"},"status":"Canceled","lastUpdateTime":"2020-05-04T05:51:22Z","submissionTime":"2020-05-04T05:28:29Z","tags":["03872cfc78a315205b546","03872cfc78a31","5205b546","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a315205b546"]},{"id":"session-03872cfc78a31e6246906-rstudio---rstudio-session-9kc4d","name":"Session 03872cfc78a31e6246906 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-2023.06.2"},"status":"Pending","statusMessage":"no nodes available to schedule pods","lastUpdateTime":"2020-05-04T05:51:52Z","submissionTime":"2020-05-04T05:50:52.411649Z","tags":["03872cfc78a31e6246906","03872cfc78a31","e6246906","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a31e6246906"]}]}

Possible cause#

For Workbench, Launcher, and Kubernetes to function properly, you should have at least one worker node in the Kubernetes cluster that is available to run pods/containers, and the node should have an adequate amount of CPU and RAM to run Launcher sessions in containers on the Kubernetes cluster.

When starting a new remote session in Kubernetes, if you experience errors in Workbench and Launcher related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of adequate CPU and RAM resources on the Kubernetes worker nodes.

The following troubleshooting steps will help you confirm the existence of worker nodes on your Kubernetes cluster.

Troubleshooting steps#

Run the following command to verify that you have worker nodes present on your Kubernetes cluster:

Terminal

$ kubectl get nodes

which should return output similar to the following if there are worker nodes in your Kubernetes cluster:

NAME                STATUS   ROLES    AGE    VERSION
compute1.internal   Ready    <none>   4h2m   v1.15.10
compute2.internal   Ready    <none>   4h2m   v1.15.10

If there are no worker nodes present on your Kubernetes cluster, then you will see output similar to the following:

No resources found.

If your Kubernetes cluster does not contain any worker nodes, then you should refer to the documentation for your Kubernetes cluster and add one or more worker nodes to the cluster.

For example, when you create a Kubernetes cluster in Amazon EKS, this will only provision the Kubernetes control plane with no worker nodes. You need to perform additional steps to provision worker nodes on your Kubernetes cluster, which is typically done by adding one or more node groups to your EKS cluster.

Test and verify#

After adding worker nodes to the Kubernetes cluster, try to start a new session from the Workbench home page.

If you are still experiencing errors when starting a new session, then proceed to
Step 14 - Verify Kubernetes Worker Nodes.