Troubleshooting Launcher and Kubernetes:
Verify Docker Image Configuration and Access#
Symptoms#
- New remote sessions in Kubernetes appear to start, but are stuck in PENDING status
Error messages#
When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:
Cluster Kubernetes
Back-off pulling image "rstudio/r-session-complete:jammy-2023.06.2"
04 May 2020 05:24:43 [rstudio-kubernetes-launcher] Received pod update with ID 39
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":0,"requestId":0,"responseId":0}
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Received getJobState request for rstudio: jobID: * startTime: none endTime: none statuses: none
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":21,"responseId":44,"jobs":[{"id":"session-03872cfc78a31679f847f-rstudio---rstudio-session-6ctxm","name":"Session 03872cfc78a31679f847f (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-2023.06.2"},"status":"Pending","statusMessage":"Back-off pulling image \"rstudio/r-session-complete:2023.12.1-402.pro1\"","lastUpdateTime":"2020-05-04T05:23:07Z","submissionTime":"2020-05-04T05:21:22.815492Z","tags":["03872cfc78a31679f847f","03872cfc78a31","679f847f","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a31679f847f"]}]}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received pod update with ID 53
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:jammy-2023.06.2\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:jammy-2023.06.2\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 54
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 55
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":63,"sequences":[{"seqId":37,"requestId":7}],"id":"session-03872cfc78a318a922518-rstudio---rstudio-session-sgr7g","status":"Pending","statusMessage":"Error: ImagePullBackOff","name":"Session 03872cfc78a318a922518 (rstudio) - RStudio Session"}
Possible cause#
For Workbench, Launcher, and Kubernetes to function properly, the Kubernetes worker nodes should be able to pull the Docker images that are used for Launcher sessions from Docker Hub or the Docker registry that is configured in your Kubernetes profile configuration.
The following troubleshooting steps will help you determine if the Docker images are configured correctly in your Kubernetes profile configuration and that the Kubernetes worker nodes have the required network access to pull Docker images from Docker Hub or your Docker registry.
Troubleshooting steps#
You should test the functionality of Launcher and Kubernetes using the rstudio/r-session-complete Docker image that is referenced in the documentation for integrating Workbench with Kubernetes. Even if you are planning to use custom Docker images with Launcher, you should initially test the Launcher functionality with the rstudio/r-session-complete Docker images.
Verify that you are using the correct Docker images in your configuration. You should have lines similar to the following in your Kubernetes profile configuration file:
container-images=rstudio/r-session-complete:jammy-2023.06.2
default-container-image=rstudio/r-session-complete:jammy-2023.06.2
Info
Ensure that the version of the Docker images in your Kubernetes profile configuration file matches the version of Workbench that you have installed.
Verify that you are able to pull the default Docker images from Docker Hub by running the following command from a Kubernetes worker node or a server that is in the same network group as the Kubernetes worker nodes:
$ docker pull rstudio/r-session-complete:jammy-2023.06.2
Info
Using our default images from Docker Hub requires that your Kubernetes worker nodes have access to pull images from Docker Hub for testing.
If that is not possible in your restricted network environment, then you will need to follow the steps outlined in the Using Docker images in an offline environment" section of the support article for Using Docker images with Workbench, Launcher, and Kubernetes.
Restart services and test#
After verifying that you are using the correct Docker images in your configuration and checking connectivity to Docker Hub or your Docker registry, restart the Workbench and Launcher services:
$ sudo rstudio-server restart
$ sudo rstudio-launcher restart
Verify that the services are running and try to start a new session from the Workbench home page.
If you have verified that the Kubernetes worker is able to pull images for Docker Hub or made the images available in your own Docker registry and you are still experiencing errors when starting a new session, then proceed to
Step 13 - Verify Kubernetes Worker Nodes.