Troubleshooting Launcher and Kubernetes:
Verify Kubernetes Token Configuration#
Symptoms#
- Unable to start new remote sessions in Kubernetes
Error messages#
When inspecting the log files for RStudio Workbench, Launcher, and Kubernetes, you might see errors similar to the following:
Job information unavailable. Sessions are read-only.
Contact your administrator for help.
Error occurred while executing method (100)
Connection refused
void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197
04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678
04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134
Possible cause#
For RStudio Workbench, Launcher, and Kubernetes to function properly, Launcher
uses a service account that has access to the rstudio namespace in Kubernetes.
All of this communication occurs from Launcher to the Kubernetes API.
You should have created the service account and service account token as part of the steps in the integrating RStudio Workbench with Kubernetes documentation.
The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes service account token in the Kubernetes plugin configuration file.
Troubleshooting steps#
You should have created a service account token in Kubernetes as part of the steps in the integrating RStudio Workbench with Kubernetes documentation, which should appear similar to the following line in your Kubernetes plugin configuration:
auth-token=<KUBERNETES-CLUSTER-TOKEN>
An example of a correctly configured service account token in the Kubernetes plugin configuration will appear similar to:
auth-token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJyc3R1ZGlvIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImpvYi1sYXVuY2hlci10b2tlbi1nNXA2aiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJqb2ItbGF1bmNoZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJmZTk4N2MwNi1kZDRlLTRmZTktOTA0Yy1iZDY2MjAwNzIxNjIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6cnN0dWRpbzpqb2ItbGF1bmNoZXIifQ.BRTX51Xj14AwTe_HuKD8RofWxgLwhobhHqRUlPTSCcHSoe3Olv4iXfrdjWOPFTEv2NAJaLSpjozSUELW8n5kxZzJnUUgWqgrCMwIUoZEC-JHncDACD-76W0Qohy9wAF20g6gmjEqEDzFpeOTB8O1HpBYArTd60iU4naN72Foj6_2NyXCETNZ24rIyIDTCX1MQTm64N0o4NnVL-zYl8ifHWlJMCBI3NMpYL014er3JLYyVKeIxPjRPryML48DhD4NZyDNzzPUr4Os_urAeht4wqScsmaTZjUQARYJ0FBgCp1C6vc1fZdAqJAIIJoHblOxNIv5wPLgsn2b3VuiBf92Zg
Verify that your service account token appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.
Restart services and test#
After you've verified that you have configured the correct Kubernetes
auth-token in the Kubernetes plugin configuration file, restart the RStudio
Server Pro and Launcher services:
$ sudo rstudio-server restart
$ sudo rstudio-launcher restart
Verify that the services are running and try to start a new session from the RStudio Workbench home page.
If the services are still not starting or you are still experiencing errors when
starting a new session, then proceed to Step 11 - Verify Kubernetes API Access, which
will verify that the service account token has access to the rstudio namespace
on the Kubernetes cluster.