Skip to content

Troubleshooting Launcher and Kubernetes:
Verify Certificate Configuration#

Symptoms#

  • Unable to start new remote sessions in Kubernetes

Error messages#

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page
Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197
File: /var/lib/rstudio-server/monitor/log/rstudio-server.log
04 May 2020 03:36:03 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678
File: /var/lib/rstudio-launcher/rstudio-launcher.log
04 May 2020 03:34:35 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 03:34:35 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 03:34:35 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 03:34:35 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 03:34:35 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR system error 84 (Invalid or incomplete multibyte or wide character) [reason=string length 1367 is not a multiple of 4]; OCCURRED AT: rstudio::core::Error rstudio::core::base64::{anonymous}::Decoder::operator()(const Byte*, std::size_t, std::string*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/Base64.cpp:231; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134
04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR Could not base64 decode certificate-authority; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::impls::kubernetes::KubernetesApi::decodeCertificateAuthority() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:75
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 218529960 (wrong tag); OCCURRED AT: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67; LOGGED FROM: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67
04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 336134278 (certificate verify failed); OCCURRED AT: void rstudio::core::http::TcpIpAsyncClientSsl::handleHandshake(const rstudio_boost::system::error_code&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:142; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause#

For Workbench, Launcher, and Kubernetes to function properly, the Launcher service needs to be able to communicate to the Kubernetes API over HTTPS.

This requires that the certificate authority (CA) for the Kubernetes API is configured correctly in the Kubernetes plugin configuration file.

The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes certificate authority in the Kubernetes plugin configuration file.

Troubleshooting steps#

Verify that you have configured the correct certificate-authority in the Kubernetes plugin configuration file:

File: /etc/rstudio/launcher.kubernetes.conf
certificate-authority=<BASE-64-ENCODED-CA-CERTIFICATE>

where <BASE-64-ENCODED-CA-CERTIFICATE> is the Base64 encoded CA certificate for the Kubernetes API.

You can usually obtain the Kubernetes CA certificate from your Kubernetes cluster console or dashboard, or from your kubectl configuration in ~/.kube/config, and it will often be provided in a Base64-encoded format already.

You can tell if your CA certificate is Base64 encoded if it appears similar to the following:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXdNVEU0TURRME4xb1hEVE13TURReU9URTRNRFEwTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3NMClgzemZYNDVaYlhpRHllSGxhZWRZSm1UQ2lzVFYvc284aVlYaThpaHRaMnRnNkNmUnQzR1NtWDBMeDk5OC9rbFQKa0I5UVJOdHQ5VUJWZmQ2VW1DVXhFNU56UUV2YnI3MmNNaFh5a0wvcFdZdEdGTTJyWDUyTzBVZ3J2L3AwMURiZApTeGF6OWV3OXcvRjRRbHlxekZPNTJsaXZPeXpIbDRJL3hJRUd3L0IyN0I1ZVR3Z1pURVYyUDU4eTlmVE91bjBDCjlmZ0VBYUUvR1I1S3NrQWVaa2ZseWZaYk52WjQ2MG90R0tXaWlXaEllT0NNSUdOM0FnLzNOa1llOW1sZmVUb2MKRzY1THhVSTgxaTNjeXlvOVVGa1UvZ3N6TVpFYkM3VUhSSkdORG9MaS9aVnFucHFhMEJKM3JXeHl5WW93V3kyZwpPR2JGeFBHM0Y0ekloclFCeDNVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSjN0eWtyRHZKUkFDVGc1dE50MGFia3dOVk4KL0JLVjUxb0Z5dnVkTGp6NjBOUGVrb0FwRHUxZFRsMzhiMmI4WFVVVTZ6Q2FmYTdtc2RZNDJZbmU5blpyMnI4KwpHZTJ4c1F2enRHSmljOGZjVGdma0xZczVkblF2S3hiWWdEVlFwVmQ0Y2JoRkRiZGJGaHI2M2VVU3JCZWptSjMvCnpCUDltb0VaekVDd0dudVY5WXVvV2UrZW5ZdDYvaVY5OTlDTWtrZW9kdTFoRlA5R1pwcGdtQWVWN0h4R2ZvaHcKWGhneWtDbGI2T0d2dWsvT05wdTc3S1JYSGhTYm00aFhVck1MNnJlenZpaDZyNWQxM1dBMzZCUE1XQUFjZUV5RwpOVGFGczN0dytrWHpXOENkSWdZa0pCTFFoMG1SYzlCMitqdUF0L1YwZlkrTElzUmdYeDZRZnFzUjZWYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

Whereas a CA certificate that is Base64 decoded appears similar to the following:

-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIwMDUwMTE4MDQ0N1oXDTMwMDQyOTE4MDQ0N1owFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKsL
X3zfX45ZbXiDyeHlaedYJmTCisTV/so8iYXi8ihtZ2tg6CfRt3GSmX0Lx998/klT
kB9QRNtt9UBVfd6UmCUxE5NzQEvbr72cMhXykL/pWYtGFM2rX52O0Ugrv/p01Dbd
Sxaz9ew9w/F4QlyqzFO52livOyzHl4I/xIEGw/B27B5eTwgZTEV2P58y9fTOun0C
9fgEAaE/GR5KskAeZkflyfZbNvZ460otGKWiiWhIeOCMIGN3Ag/3NkYe9mlfeToc
G65LxUI81i3cyyo9UFkU/gszMZEbC7UHRJGNDoLi/ZVqnpqa0BJ3rWxyyYowWy2g
OGbFxPG3F4zIhrQBx3UCAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAKJ3tykrDvJRACTg5tNt0abkwNVN
/BKV51oFyvudLjz60NPekoApDu1dTl38b2b8XUUU6zCafa7msdY42Yne9nZr2r8+
Ge2xsQvztGJic8fcTgfkLYs5dnQvKxbYgDVQpVd4cbhFDbdbFhr63eUSrBejmJ3/
zBP9moEZzECwGnuV9YuoWe+enYt6/iV999CMkkeodu1hFP9GZppgmAeV7HxGfohw
XhgykClb6OGvuk/ONpu77KRXHhSbm4hXUrML6rezvih6r5d13WA36BPMWAAceEyG
NTaFs3tw+kXzW8CdIgYkJBLQh0mRc9B2+juAt/V0fY+LIsRgXx6QfqsR6Vc=
-----END CERTIFICATE-----

Ensure that you are using the Base64-encoded version of the CA certificate in the Kubernetes plugin configuration file.

If you have correctly configured the Kubernetes certificate authority in Launcher, then you should have a line similar to the following in your Kubernetes plugin configuration file:

File: /etc/rstudio/launcher.kubernetes.conf
certificate-authority=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXdNVEU0TURRME4xb1hEVE13TURReU9URTRNRFEwTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3NMClgzemZYNDVaYlhpRHllSGxhZWRZSm1UQ2lzVFYvc284aVlYaThpaHRaMnRnNkNmUnQzR1NtWDBMeDk5OC9rbFQKa0I5UVJOdHQ5VUJWZmQ2VW1DVXhFNU56UUV2YnI3MmNNaFh5a0wvcFdZdEdGTTJyWDUyTzBVZ3J2L3AwMURiZApTeGF6OWV3OXcvRjRRbHlxekZPNTJsaXZPeXpIbDRJL3hJRUd3L0IyN0I1ZVR3Z1pURVYyUDU4eTlmVE91bjBDCjlmZ0VBYUUvR1I1S3NrQWVaa2ZseWZaYk52WjQ2MG90R0tXaWlXaEllT0NNSUdOM0FnLzNOa1llOW1sZmVUb2MKRzY1THhVSTgxaTNjeXlvOVVGa1UvZ3N6TVpFYkM3VUhSSkdORG9MaS9aVnFucHFhMEJKM3JXeHl5WW93V3kyZwpPR2JGeFBHM0Y0ekloclFCeDNVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSjN0eWtyRHZKUkFDVGc1dE50MGFia3dOVk4KL0JLVjUxb0Z5dnVkTGp6NjBOUGVrb0FwRHUxZFRsMzhiMmI4WFVVVTZ6Q2FmYTdtc2RZNDJZbmU5blpyMnI4KwpHZTJ4c1F2enRHSmljOGZjVGdma0xZczVkblF2S3hiWWdEVlFwVmQ0Y2JoRkRiZGJGaHI2M2VVU3JCZWptSjMvCnpCUDltb0VaekVDd0dudVY5WXVvV2UrZW5ZdDYvaVY5OTlDTWtrZW9kdTFoRlA5R1pwcGdtQWVWN0h4R2ZvaHcKWGhneWtDbGI2T0d2dWsvT05wdTc3S1JYSGhTYm00aFhVck1MNnJlenZpaDZyNWQxM1dBMzZCUE1XQUFjZUV5RwpOVGFGczN0dytrWHpXOENkSWdZa0pCTFFoMG1SYzlCMitqdUF0L1YwZlkrTElzUmdYeDZRZnFzUjZWYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

Verify that your certificate authority appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.

Restart services and test#

After you've verified that you have configured the correct certificate-authority in the Kubernetes plugin configuration file, restart the Workbench and Launcher services:

Terminal
$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 9 - Verify Kubernetes Certificate Configuration.