Skip to content

Troubleshooting Launcher and Kubernetes:
Verify Kubernetes Token Configuration#

Symptoms#

  • Unable to start new remote sessions in Kubernetes

Error messages#

When inspecting the log files for RStudio Server Pro, Launcher, and Kubernetes, you might see errors similar to the following:

RStudio Server Pro Home Page

Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197

File: /var/lib/rstudio-server/monitor/log/rstudio-server.log

04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678

File: /var/lib/rstudio-launcher/rstudio-launcher.log

04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause#

For RStudio Server Pro, Launcher, and Kubernetes to function properly, Launcher uses a service account that has access to the rstudio namespace in Kubernetes. All of this communication occurs from Launcher to the Kubernetes API.

You should have created the service account and service account token as part of the steps in the integrating RStudio Server Pro with Kubernetes documentation.

The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes service account token in the Kubernetes plugin configuration file.

Troubleshooting steps#

You should have created a service account token in Kubernetes as part of the steps in the integrating RStudio Server Pro with Kubernetes documentation, which should appear similar to the following line in your Kubernetes plugin configuration:

File: /etc/rstudio/launcher.kubernetes.conf

auth-token=<KUBERNETES-CLUSTER-TOKEN>

An example of a correctly configured service account token in the Kubernetes plugin configuration will appear similar to:

File: /etc/rstudio/launcher.kubernetes.conf

auth-token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJyc3R1ZGlvIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImpvYi1sYXVuY2hlci10b2tlbi1nNXA2aiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJqb2ItbGF1bmNoZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmZTk4N2MwNi1kZDRlLTRmZTktOTA0Yy1iZDY2MjAwNzIxNjIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6cnN0dWRpbzpqb2ItbGF1bmNoZXIifQ.BRTX51Xj14AwTe_HuKD8RofWxgLwhobhHqRUlPTSCcHSoe3Olv4iXfrdjWOPFTEv2NAJaLSpjozSUELW8n5kxZzJnUUgWqgrCMwIUoZEC-JHncDACD-76W0Qohy9wAF20g6gmjEqEDzFpeOTB8O1HpBYArTd60iU4naN72Foj6_2NyXCETNZ24rIyIDTCX1MQTm64N0o4NnVL-zYl8ifHWlJMCBI3NMpYL014er3JLYyVKeIxPjRPryML48DhD4NZyDNzzPUr4Os_urAeht4wqScsmaTZjUQARYJ0FBgCp1C6vc1fZdAqJAIIJoHblOxNIv5wPLgsn2b3VuiBf92Zg

Verify that your service account token appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.

Restart services and test#

After you've verified that you have configured the correct Kubernetes auth-token in the Kubernetes plugin configuration file, restart the RStudio Server Pro and Launcher services:

Terminal

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the RStudio Server Pro home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to Step 10 - Verify Kubernetes API Access, which will verify that the service account token has access to the rstudio namespace on the Kubernetes cluster.