Troubleshooting Launcher and Kubernetes:
Verify Kubernetes API Access#
Symptoms#
- Unable to start new remote sessions in Kubernetes
Error messages#
When inspecting the log files for RStudio Server Pro, Launcher, and Kubernetes, you might see errors similar to the following:
RStudio Server Pro Home Page
Job information unavailable. Sessions are read-only.
Contact your administrator for help.
Error occurred while executing method (100)
Connection refused
void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197
File: /var/lib/rstudio-server/monitor/log/rstudio-server.log
04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678
File: /var/lib/rstudio-launcher/rstudio-launcher.log
04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134
Possible cause#
For RStudio Server Pro, Launcher, and Kubernetes to function properly, Launcher
uses a service account that has access to the rstudio
namespace in Kubernetes.
All of this communication occurs from Launcher to the Kubernetes API.
You should have created a service account and service account token as part of the steps in the integrating RStudio Server Pro with Kubernetes documentation.
The following troubleshooting steps ensure that you can connect to the
Kubernetes API, that your service account token is valid, and that the service
account has access to the rstudio
namespace.
Troubleshooting steps#
Verify that you can connect to the Kubernetes API and that the service account
token that you generated has access to the rstudio
namespace:
Terminal
$ export TOKEN=<KUBERNETES-CLUSTER-TOKEN>
$ curl -k -H "Authorization: Bearer $TOKEN" https://<KUBERNETES-API-ENDPOINT>/api/v1/namespaces/rstudio/
where <KUBERNETES-CLUSTER-TOKEN>
is the Kubernetes service account token that
you generated from the documentation for Integrating RStudio Server Pro with
Kubernetes,
and <KUBERNETES-API-ENDPOINT>
is the URL for the Kubernetes API.
If the Kubernetes cluster token is valid and the associated service account is
authorized to access the rstudio
namespace, then you should see a response
from the Kubernetes API such as:
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "rstudio",
"selfLink": "/api/v1/namespaces/rstudio",
"uid": "d802450e-2c5b-4520-a7d5-6ef2ffa5c969",
"resourceVersion": "769",
"creationTimestamp": "2020-04-09T17:20:45Z"
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"phase": "Active"
}
}
If the Kubernetes cluster token is not valid or the associated service account
is not authorized to access the rstudio
namespace, then you will see a
response from the Kubernetes API such as:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
If you are receiving an "Unauthorized" message as shown above, then you can attempt to recreate the service account and generate a service account token as part of the steps in the integrating RStudio Server Pro with Kubernetes documentation.
After you've generated a new service account token, you can repeat these
troubleshooting steps to ensure that your service account token is valid and has
access to the rstudio
namespace.
Once you've verified that the Kubernetes service account token is valid, ensure that you've configured it correctly in the Kubernetes plugin configuration:
File: /etc/rstudio/launcher.kubernetes.conf
auth-token=<KUBERNETES-CLUSTER-TOKEN>
Restart services and test#
After you've verified that you have configured the correct Kubernetes auth token
and that it is authorized to access the rstudio
namespace on the Kubernetes
cluster, restart the RStudio Server Pro and Launcher services:
Terminal
$ sudo rstudio-server restart
$ sudo rstudio-launcher restart
Verify that the services are running and try to start a new session from the RStudio Server Pro home page.
If the services are still not starting or you are still experiencing errors when
starting a new session, then proceed to
Step 11 - Verify Docker Image Configuration and Access.