Skip to content

Troubleshooting Launcher and Kubernetes:
Verify Docker Image Configuration and Access#

Symptoms#

  • New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

RStudio Workbench Home Page - New Session Pending Status

Error messages#

When inspecting the log files for RStudio Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

RStudio Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Back-off pulling image "rstudio/r-session-complete:centos7-2021.09.0-351.pro6"

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:24:43 [rstudio-kubernetes-launcher] Received pod update with ID 39
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":0,"requestId":0,"responseId":0}
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Received getJobState request for rstudio: jobID: * startTime: none endTime: none statuses: none
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":21,"responseId":44,"jobs":[{"id":"session-03872cfc78a31679f847f-rstudio---rstudio-session-6ctxm","name":"Session 03872cfc78a31679f847f (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:centos7-2021.09.0-351.pro6"},"status":"Pending","statusMessage":"Back-off pulling image \"rstudio/r-session-complete:2021.09.0-351.pro6\"","lastUpdateTime":"2020-05-04T05:23:07Z","submissionTime":"2020-05-04T05:21:22.815492Z","tags":["03872cfc78a31679f847f","03872cfc78a31","679f847f","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:03872cfc78a31679f847f"]}]}

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received pod update with ID 53
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:centos7-2021.09.0-351.pro6\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:centos7-2021.09.0-351.pro6\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 54
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-03872cfc78a318a922518-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 55
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":63,"sequences":[{"seqId":37,"requestId":7}],"id":"session-03872cfc78a318a922518-rstudio---rstudio-session-sgr7g","status":"Pending","statusMessage":"Error: ImagePullBackOff","name":"Session 03872cfc78a318a922518 (rstudio) - RStudio Session"}

Possible cause#

For RStudio Workbench, Launcher, and Kubernetes to function properly, the Kubernetes worker nodes should be able to pull the Docker images that are used for Launcher sessions from Docker Hub or the Docker registry that is configured in your Kubernetes profile configuration.

The following troubleshooting steps will help you determine if the Docker images are configured correctly in your Kubernetes profile configuration and that the Kubernetes worker nodes have the required network access to pull Docker images from Docker Hub or your Docker registry.

Troubleshooting steps#

You should test the functionality of Launcher and Kubernetes using the rstudio/r-session-complete Docker image that is referenced in the documentation for integrating RStudio Workbench with Kubernetes. Even if you are planning to use custom Docker images with Launcher, you should initially test the Launcher functionality with the rstudio/r-session-complete Docker images.

Verify that you are using the correct Docker images in your configuration. You should have lines similar to the following in your Kubernetes profile configuration file:

File: /etc/rstudio/launcher.kubernetes.profiles.conf

container-images=rstudio/r-session-complete:centos7-2021.09.0-351.pro6
default-container-image=rstudio/r-session-complete:centos7-2021.09.0-351.pro6

Info

Ensure that the version of the Docker images in your Kubernetes profile configuration file matches the version of RStudio Workbench that you have installed.

Verify that you are able to pull the default Docker images from Docker Hub by running the following command from a Kubernetes worker node or a server that is in the same network group as the Kubernetes worker nodes:

Terminal

$ docker pull rstudio/r-session-complete:centos7-2021.09.0-351.pro6

Info

Using our default images from Docker Hub requires that your Kubernetes worker nodes have access to pull images from Docker Hub for testing.

If that is not possible in your restricted network environment, then you will need to follow the steps outlined in the Using Docker images in an offline environment" section of the support article for Using Docker images with RStudio Workbench, Launcher, and Kubernetes.

Restart services and test#

After verifying that you are using the correct Docker images in your configuration and checking connectivity to Docker Hub or your Docker registry, restart the RStudio Workbench and Launcher services:

Terminal

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the RStudio Workbench home page.

If you have verified that the Kubernetes worker is able to pull images for Docker Hub or made the images available in your own Docker registry and you are still experiencing errors when starting a new session, then proceed to
Step 12 - Verify Kubernetes Worker Nodes.

Back to top