11 Job Spawner
11.1 Overview
The RStudio Job Spawner provides an HTTP API for applications to start processes within various batch processing systems (e.g. IBM Spectrum LSF) and container orchestration platforms (e.g. Kubernetes). RStudio Server Pro integrates with the Job Spawner to allow you to run your R Sessions within your compute cluster software of choice, and allows you to containerize your sessions for maximum process isolation and operations efficiency. Furthermore, users can submit standalone jobs to your compute cluster(s) to run computationally expensive R or Python scripts.
API documentation for the Job Spawner is outside of the scope of this document, but can be found here.
11.2 Configuration
11.2.1 Job Spawner Configuration
Before the Job Spawner can be run, it must be properly configured via the config file /etc/rstudio/spawner.conf. The following table lists the supported configuration options.
11.2.1.1 Configuration Options
Server Options
There should be one [server] section in the configuration file (see sample config below).
| Config Option | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| address | IPv4 or IPv6 address, or path to Unix domain socket | Y | |
| port | Port number (0-65535) | Y (when using IP Address) | |
| enable-ssl | Toggle usage of SSL encryption for connections | N | 0 |
| certificate-file | Certificate chain file of public certificates to present to incoming connections | Y | (Only required when SSL is enabled) |
| certificate-key-file | Certificate private key file used for encryption | Y | (Only required when SSL is enabled) |
| server-user | User to run the executable as. The Spawner should be started as root, and will lower its privilege to this user for normal execution. | N | rstudio-server |
| authorization-enabled | Enables/disables authorization - this is required for all but test systems. Can be 1 (enabled) or 0 (disabled) | N | 1 |
| admin-users | Comma-separated list of users that are able to see/control all jobs in the system of other users. | N | Empty |
| thread-pool-size | Size of the thread pools used by the spawner | N | Number of CPUs * 2 |
| request-timeout-seconds | Number of seconds a plugin has to process a request before it is considered timed out | N | 120 |
| enable-debug-logging | Enables/disables verbose debug logging. Can be 1 (enabled) or 0 (disabled) | N | 0 |
| scratch-path | Scratch directory where the spawner and its plugins write temporary state | N | /var/lib/rstudio-spawner |
Cluster Options
There should be one [cluster] section in the configuration file per cluster to connect to / plugin to load (see sample config below).
| Config Option | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| name | Friendly name of the cluster | Y | |
| type | Type of the cluster (for human consumption, display purposes) | Y | |
| exe | Path to the plugin executable for this cluster | Y |
11.2.1.2 Sample Configuration
[server]
address=127.0.0.1
port=5559
server-user=rstudio-server
admin-users=bdylan
authorization-enabled=1
thread-pool-size=4
enable-debug-logging=1
[cluster]
name=Local
type=Local
exe=/usr/lib/rstudio-server/bin/rstudio-local-spawner11.2.2 Job Spawner Plugin Configuration
Each specific cluster plugin can be additionally configured via its own configuration file, and some plugins (such as the Kubernetes plugin) require additional configuration. Documentation for all plugins created by RStudio can be found in the following sections.
11.2.2.1 Local Plugin
The Local Job Spawner Plugin provides the capability to launch executables on the local machine (same machine that the Spawner is running on). It also provides the capability of running arbitrary PAM profiles. All of the sandboxing capability is provided via rsandbox.
11.2.2.1.1 Configuration Options
The local plugin does not require configuration, and it is recommended you do not change any of the defaults.
/etc/rstudio/spawner.local.conf
| Config Option | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| server-user | User to run the executable as. The plugin should be started as root, and will lower its privilege to this user for normal execution. | N | rstudio-server |
| thread-pool-size | Size of the thread pool used by the plugin | N | Number of CPUs * 2 |
| enable-debug-logging | Enables/disables verbose debug logging. Can be 1 (enabled) or 0 (disabled) | N | 0 |
| scratch-path | Scratch directory where the plugin writes temporary state | N | /var/lib/rstudio-spawner |
| job-expiry-hours | Number of hours before completed jobs are removed from the system | N | 24 |
| save-unspecified-output | Enables/disables saving of stdout/stderr that was not specified in submitted jobs. This will allow users to view their output even if they do not explicitly save it, at the cost of disk space. | N | 1 |
| rsandbox-path | Location of rsandbox executable. |
N | /usr/lib/rstudio-server/bin/rsandbox |
11.2.2.1.2 Job Configuration
Each plugin can expose its own configuration overrides for submitting jobs that pertain to just that cluster type.
The Local Spawner Plugin allows you to specify a PAM profile to run when a job is submitted and the user’s password that should be used for authentication.
For example, the following would be added to the submitted job payload:
"config": [
{
"name": "pamProfile",
"value": "MyPamProfile"
},
{
"name": "encryptedPassword",
"value": <encrypted password>
},
{
"name": "initializationVector",
"value": <At least 8 bytes of randomly generated text in base-64 encoded format>
}
]The pamProfile field is simply the name of the PAM profile you wish to authenticate against.
The encryptedPassword field is the user’s password encrypted in the following manner:
iv = random() // at least 8 bytes
encryptedPassword = aes_128_cbc(password, secure-cookie-key, iv)The initializationVector field is simply the initialization vector randomly generated in the encryption step, base-64 encoded.
11.2.2.2 Kubernetes Plugin
The Kubernetes Job Spawner Plugin provides the capability to launch executables on a Kubernetes cluster.
11.2.2.2.1 Configuration Options
It is recommended not to change the default values which come from the Job Spawner itself, and only configure required fields as outlined below.
/etc/rstudio/spawner.kubernetes.conf
| Config Option | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| server-user | User to run the executable as. The plugin should be started as root, and will lower its privilege to this user for normal execution. | N | rstudio-server |
| thread-pool-size | Size of the thread pool used by the plugin | N | Number of CPUs * 2 |
| enable-debug-logging | Enables/disables verbose debug logging. Can be 1 (enabled) or 0 (disabled) | N | 0 |
| scratch-path | Scratch directory where the plugins writes temporary state | N | /var/lib/rstudio-spawner |
| job-expiry-hours | Number of hours before completed jobs are removed from the system | N | 24 |
| api-url | The Kubernetes API base URL. This can be an HTTP or HTTPS URL. The URL should be up to, but not including the /api endpoint. | Y | Example: https://192.168.99.100:8443 |
| auth-token | Y | The auth token for the job-spawner service account. This is used to authenticate with the Kubernetes API. |
11.2.2.2.2 Job Configuration
Each plugin can expose its own configuration overrides for submitting jobs that pertain to just that cluster type.
At this time, the Kubernetes plugin does not expose its own specific configuration options.
11.2.2.2.3 Kubernetes Cluster Requirements
In order for the Kubernetes plugin to run correctly, the following assumptions about the Kubernetes cluster must be true:
- The Kubernetes API must be enabled and reachable from the machine running the Job Spawner. It must not be exposed via
kubectl proxy, as that does not proxy websockets, which are required by the Kubernetes plugin. - There must be a namespace named rstudio
- There must be a service account named job-spawner in the rstudio namespace
- The service account must have full cluster-admin privileges, provided by a rolebinding
- The cluster must have the metrics-server addon running and working properly to provide job resource utilization streaming.
11.2.3 RStudio Server Pro Integration
RStudio Server Pro must be configured in order to integrate with the Job Spawner. There are several files which house the configuration, and they are described within subsequent sections.
11.2.3.1 Server Configuration
The RStudio Server process rserver must be configured to communicate with the Job Spawner in order to enable session launcing. The following table lists the various configuration options that are available:
| Config Option | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| spawner-sessions-enabled | Enables launching of rsession processes via the Job Spawner | N | 0 |
| spawner-address | TCP host/IP of the spawner host, or unix domain socket path (must match `/etc/rstudio/spawner.conf configuration value) | Y | |
| spawner-port | Port that the spawner is listening on. Only required if not using unix domain sockets. | Y | |
| spawner-default-cluster | Name of the cluster to use when launching sessions. Can be overridden by the launching user. | Y | |
| spawner-sessions-callback-address | Address (http or https) of RStudio Server Pro that will be used by spawner sessions to communicate back for project sharing features. This is only required if using plugins that do not use containers, as project sharing is not yet supported for containerized sessions. | Y | |
| spawner-use-ssl | Whether or not to connect to the spawner over HTTPS. Only supported for connections that do not use unix domain sockets. | N | 0 |
| spawner-sessions-container-image | The default container image to use when creating sessions. Only required if using a plugin that requires containerization. | Y | |
| spawner-sessions-container-run-as-root | Whether or not to run as root within the session container. | N | 1 |
| spawner-sessions-create-container-user | Whether or not to create the session user within the container. Only applicable if using container sessions and not running containers as root. The created user will have the same UID and GID as the user that launched the session. It is recommended that this option be used, unless your containers connect to an LDAP service to manage users and groups. | N | 0 |
11.2.3.2 Spawner Mounts
When creating containerized sessions via the Job Spawner, you will need to specify mount points as appropriate to mount the user’s home drive and any other desired paths.
To specify mount points, modify the /etc/rstudio/spawner-mounts file to consist of multiple mount entries separated by a blank line. The following table lists the fields that are available for each mount entry in the file.
| Field | Description | Required (Y/N) | Default Value |
|---|---|---|---|
| Path | The source directory of the mount, i.e. where the mount data comes from. | Y | |
| Host | The NFS host name for the NFS mount. Only used if the mount is NFS. | N | |
| MountPath | The path within the container that the directory will be mounted to. | Y | |
| ReadOnly | Whether or not the mount is read only. Can be true or false. | N | false |
Additionally, paths may contain the special variable {USER} to indicate that the user’s name be substituted, enabling you to mount user-specific paths.
An example /etc/rstudio/spawner-mounts file is shown below.
/etc/rstudio/r-versions
# User home mount
Host: nfs01
Path: /home/{USER}
MountPath: /home/{USER}
ReadOnly: false
# Shared code mount
Host: nfs01
Path: /dev64
MountPath: /code
ReadOnly: falseIt is important that each entry consists of the fields as specified above. Each field must go on its own line. There should be no empty lines between field definitions. Each entry must be separated by one full blank line (two new-line \n characters).
11.2.3.3 Spawner Environment
You may optionally specify environment variables to set when creating containerized sessions.
To specify environment variables, modify the /etc/rstudio/spawner-env file to consist of KEY=VALUE pairs, one per line. Additionally, you can use the special {USER} variable to specify the value of the launching user’s username, similar to the mounts file above.
An example /etc/rstudio/spawner-env file is shown below.
/etc/rstudio/spawner-env
VAR1=VAL1
USER_HOME=/home/{USER}11.2.3.4 Spawner Ports
You may optionally specify ports that should be exposed when creating containerized sessions. This will allow the ports to be exposed within the host running the container, allowing the ports to be reachable from external services. For example, for Shiny applications to be usable, you must expose the desired Shiny port, otherwise the browser window will not be able to connect to the Shiny application running within the container.
To specify ports, modify the /etc/rstudio/spawner-ports file to consist of port numbers, one per line.
An example /etc/rstudio/spawner-ports file is shown below.
/etc/rstudio/spawner-ports
5873
5874
64234
6423511.3 Running the Spawner
Once it is configured, you can run the Job Spawner by invoking the command sudo rstudio-spawner start, and stop it with sudo rstudio-spawner stop. The Job Spawner must be run with root privileges, but similar to rstudio-server, privileges are immediately lowered. Root privileges are used only to impersonate users as necessary.
11.4 Creating plugins
Plugins allow communication with specific batch cluster / container orchestration systems like Platform LSF and Kubernetes. However, you may be using a system that RStudio does not natively support. Fortunately, the Job Spawner provides a fairly simple means of creating custom plugins that can allow you to spawn jobs on any cluster software you desire. This section explains how custom cluster plugins can be created.
11.4.1 Plugin Architecture
As you may have seen from configuring the Job Spawner, plugins are simply exectuables that are started by the main Job Spawner executable, rstudio-spawner. Once your plugin is configured as an available plugin and the rstudio-spawner process starts, it will execute your plugin process as a child of itself, and communicate with it via stdin/stdout.
As requests come in to the Job Spawer via the HTTP API, the Job Spawner creates simple JSON-based requests and sends them to your plugin through stdin. Your plugin should respond to each request as appropriate, and send responses back to the main rstudio-spawner process via stdout.
11.4.2 Plugin Configuration
Each plugin is passed a set of configuration values as arguments to the plugin when it is started. These configuration options contain default values determined by the rstudio-spawner binary, and include configuration options set in the /etc/rstudio/spawner.conf config file.
The following configuration options are passed to the plugin when started:
| Argument name | Description |
|---|---|
| server-user | The name of the user that the process should be run as. Plugins start with root privilege and SHOULD lower their privilege to run as the user specified by this argument. Plugins start with root privilege to allow for user impersonation when communicating with cluster software that requires it. |
| enable-debug-logging | A 0 or 1 indicating whether or not debug logging should be enabled. If so, a higher level of verbosity should be used when logging information. |
| scratch-path | The path assigned by the Job Spawner that the plugin is allowed to write to. This is where the plugin may store temporary data, such as cached information about jobs. |
| heartbeat-interval-seconds | The number of seconds in between heartbeats. If this is 0, then heartbeats should be disabled. If the plugin misses 3 heartbeats in a row, the Job Spawner considers it to be in an invalid state and will kill and restart the plugin. For more information, see the subsequent section on heartbeat messages. |
Each plugin should also allow for configuration overrides of these values and any additional necessary configuration via file, located at /etc/rstudio/spawner.<plugin name>.conf.
11.4.3 Plugin Messages
The following sections outline the various requests and responses that can be sent and received to and from the rstudio-spawner process.
11.4.3.1 General message structure
Each message has a 4-byte big-endian message length indicating the length of the message to be read, followed by the message payload. While not strictly necessary for communication over stdin/stdout, the message length makes it easy to deliniate message boundaries. Big-endian is chosen in order to make any potential future transition to over-network communication with plugins fairly pain free.
Each message payload is one of three types: PluginRequest, PluginResponse, or ErrorResponse, which is simply a JSON object containing, at a mininum, the following fields:
PluginRequest (sent by the rstudio-spawner binary to the plugin)
| Field name | Description |
|---|---|
| messageType | The type ID of the message. The ID for a given message type can be found in the section for the given message. |
| requestId | The monotonically increasing request ID for this request. This should be saved by the plugin for use in responses. |
| username | The username of the user that initiated the request. Used for auding/security purposes. Some requests have the ability to specify * for this value, indicating all users’ information should be returned. This can occur if a super user (admin user) makes a request, or if authorization is disabled. |
PluginResponse (sent by the plugin to the rstudio-spawner binary)
| Field name | Description | Value |
|---|---|---|
| messageType | The type ID of the message. The ID for a given message type can be found in the section for the given message. | Int |
| requestId | The request ID for the request that initiated this response. This is used by the rstudio-spawner binary to properly determine which response belongs to which request. |
Int |
| responseId | The monitonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. | Int |
ErrorResponse (sent by the plugin to indicate an error)
| Field name | Description | Value |
|---|---|---|
| messageType | The type ID of the message. | -1 |
| requestId | The request ID for the request that initiated this response. This is used by the rstudio-spawner binary to properly determine which response belongs to which request. |
Int |
| responseId | The monitonically increasing response ID for this response. This must be properly generated for each response sent by the plugin. | Int |
| errorCode | The error code. More details can be found in the Errors section. | Int |
| errorMessage | The error message. More details can be found in the Errors section. | String |
ErrorResponses can be sent in response to all requests. For more information, see Errors.
11.4.3.2 Heartbeats
As mentioned in the configuration option heartbeat-interval-seconds, heartbeats should be sent to the plugin once every heartbeat-interval-seconds seconds, and the plugin should respond back with a heartbeat message.
HeartbeatRequest
| Field name | Field value |
|---|---|
| messageType | 0 |
| requestId | 0 |
Sample payload:
{
"messageType": 0,
"requestId": 0
}HeartbeatResponse
| Field name | Field value |
|---|---|
| messageType | 0 |
| requestId | 0 |
| responseId | 0 |
Sample payload:
{
"messageType": 0,
"requestId": 0,
"responseId": 0
}Note that for heartbeats, the request and response IDs are always 0.
11.4.3.3 Cluster Info
Cluster info requests allow for the retrieval of information about a particular cluster targeted by a plugin.
ClusterInfoRequest
| Field name | Field value |
|---|---|
| messageType | 1 |
Sample payload:
{
"messageType": 1,
"requestId": 0
}Note that the ClusterInfoRequest is the very first message sent to a plugin, so it will always have a requestId value of 0.
ClusterInfoResponse
| Field name | Description | Field value |
|---|---|---|
| messageType | 1 | |
| supportsContainers | Whether or not the cluster allows containerization. | Boolean |
| queues | Array of available queue names. Only used for batch processing systems. | String array |
| config | Cluster-specific configuration options available for a job. | Job Config Array. See below for details. |
| resourceLimits | Available resource limits for submitted jobs, for example max CPU usage or max memory usage. | ResourceLimit array. See below for details. |
For example:
ClusterInfoResponse
{
"messageType": 1,
"supportsContainers": false,
"queues": ["Prod", "Dev"],
"config": [
{"name": "SpecialVal", "valueType": "int"}
],
"resourceLimits": [
{"limitType": "memory"},
{"limitType": "cpuCount"}
]
}JobConfig
| Field name | Description | Value |
|---|---|---|
| name | Name of the job config param | String |
| valueType | A string representing the value type. | Can be one of “string”, “int”, “float”, or “enum”. |
ResourceLimit
| Field name | Description | |
|---|---|---|
| limitType | A string representing the limit type. | Can be one of “cpuCount”, “cpuTime”, “memory” or “memorySwap”. |
11.4.3.4 Submit Job
Submit job requests allow new jobs to be submitted to the cluster, which handles the scheduling of the job.
SubmitJobRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 2 | |
| username | The name of the user submitting the job. | String |
| job | The job object. | Object (see Swagger documentation for definition). |
A JobStateResponse is returned in response to a SubmitJobRequest. This will be described in the next section.
11.4.3.5 Job State
Job state requests allow users to get the current state of a specific job, or all of their jobs that match specific criteria.
JobStateRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 3 | |
| username | The name of the user requesting job information. If all users’ job information should be returned, this field will be “*“. You must ensure the security of job objects and only allow authorized users access to their own jobs, as appropriate. | String |
| jobId | The job ID to get information for. If all of the user’s jobs information is requested, this will be ““. If username is””, all jobs (for all users) in the entire system should be returned. | String |
| tags | If populated, return only jobs that match all specified tags. | String Array |
| startTime | A time in the format YYYY-MM-DDThh:mm:ss indicating the earliest UTC submission time. Only jobs submitted at or after this point in time will be included in the result set. |
String, optional. |
| endTime | A time in the format YYYY-MM-DDThh:mm:ss indicating the latest UTC submission time. Only jobs submitted at or before this point in time will be included in the result set. |
String, optional. |
| status | Desired job status. Only jobs that match this status will be included in the result set. | String, optional. |
JobStateResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 2 | |
| jobs | Array of jobs that meet the criteria. | Job Array |
Even if only one Job should be returned, jobs should be an array containing only one job.
11.4.3.6 Job Status Stream
A job status stream allows users to stream the status of a particular job or all jobs.
JobStatusRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 4 | |
| username | The name of the user requesting job information. If all users’ job statuses should be streamed, this field will be “*“. You must ensure the security of job objects and only allow authoirzed users access to their own jobs, as appropriate. | String |
| jobId | The job ID to stream status for. If all of a particular users’ job statuses should be streamed, this will be “*“. | String |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobStatusRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobStatusResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.
JobStatusResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 3 | |
| sequences | An array of stream sequences indicating which requests this update satisfies. | StreamSequence Array |
| jobId | The ID of the job which status has changed. | String |
| jobName | The name of the job. | String |
| status | The status of the job. | String |
StreamSequence
| Field name | Description | Value |
|---|---|---|
| requestId | The requestId of a stream request that matches this update. |
Int |
| seqId | The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. |
Int |
11.4.3.6.1 Stream example
As an example, say we receive two job status stream requests, one for all of user bob’s jobs, and one for a specific job of user bob.
Request 1
{
"requestId": 14,
"messageType": 4,
"username": "bob",
"jobId": "*"
}Request 2
{
"requestId": 45,
"messageType": 4,
"username": "bob",
"jobId": "AFA2532XSAZ"
}Now, let’s say that the status of job AFA2532XSAZ has changed. The following response would be sent out.
Response 1
{
"messageType": 3,
"jobId": "AFA2532XSAZ",
"jobName": "My job",
"status": "Running",
"sequences": [
{ "requestId": 14, "seqId": 1 },
{ "requestId": 45, "seqId": 1 }
]
}Now, let’s say later on, some other job B23532AD has its status change. The following response would be sent out.
Response 2
{
"messageType": 3,
"jobId": "B23532AD",
"jobName": "Another job",
"status": "Killed",
"sequences": [
{ "requestId": 14, "seqId": 2 }
]
}Note that only the matching sequences are sent.
Now let’s say that one of the streams is canceled.
Request 3
{
"requestId": 45,
"messageType": 4,
"username": "bob",
"jobId": "AFA2532XSAZ",
"cancel": true
}Note that the requestId matches that of the original request so that the plugin knows which request to cancel.
Now, let’s say once more the job AFA2532XSAZ changes its status. Now that one of the streams was canceled, we will not send sequence information for that particular request anymore. The response would be:
Response 3
{
"messageType": 3,
"jobId": "AFA2532XSAZ",
"jobName": "My job",
"status": "Finished",
"sequences": [
{ "requestId": 14, "seqId": 3 }
]
}11.4.3.7 Job Control
The job control request allows users to modify their jobs to suspend, resume, stop (SIGTERM), or kill (SIGKILL) them.
JobControlRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 5 | |
| username | Username of the initiator of this request. “*" if requested by a super user or auth is disabled. You must ensure that users can only control their own jobs, as appropriate. | String |
| jobId | The job ID of the job to control. | String |
| operation | The operation to perform. | JobOperation (described below). |
JobOperation (enum)
Suspend = 0,
Resume = 1,
Stop = 2,
Kill = 3JobControlResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 4 | |
| statusMessage | Human-friendly status message indicating the result of the operation. | String |
| operationComplete | Whether or not the operation has already fully completed successfully. Should be false if the request was merely submitted to the cluster and it must be processed asynchronously. | Boolean |
11.4.3.8 Job Output Stream
Job output streams allow users to stream the stdout/stderr of their jobs in real-time.
JobOutputRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 6 | |
| username | The username of the requestor. “*" if requested by a super user or auth is disabled. You must ensure that users can only view the output of their own jobs, as appropriate. | String |
| jobId | The ID of the job for which to get output. | String |
| outputType | The type of output to fetch. | JobOutputType (see below). |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
JobOutputType (enum)
Stdout = 0,
Stderr = 1,
Both = 2Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobOutputRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
JobOutputResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 5 | |
| seqId | Monotonically increasing sequence ID for this stream, starting at 1. | Int |
| output | Next chunk of output received from the job. | String |
| outputType | The type of the output (see JobOutputType) |
JobOutputType |
| complete | Flag indicating whether or not the output stream is complete. Set to true if the job has and will never have more output (such as if the job is finished and the output has been fully read). | Boolean |
Streaming should commence similarly to job status streaming (described above), except ouput streams are only ever associated with one request and as such do not have multiple sequences per response.
11.4.3.9 Job Resource Utilization Stream
Job resource utilization streams allow users to stream the most up-to-date resource utilization of their jobs in real-time. This is similar in concept to commands like top.
JobResourceUtilizationRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 7 | |
| username | The username of the requestor. “*" if requested by a super user or auth is disabled. You must ensure that users can only view the resource utilization of their own jobs, as appropriate. | String |
| jobId | The ID of the job for which to get resource utilization. | String |
| cancel | Flag used to indicate if the stream should be canceled. When this is set, the original requestId of the start of the stream will be used. |
Boolean |
Once the stream has been started, the plugin is responsible for sending updates as they occur. If the stream should be canceled, a second JobResourceUtilizationRequest will be sent with the cancel flag set to true and the requestId set to the same value as the requestId used to start the stream. When the stream is canceled, the plugin should stop sending updates for the particular stream.
Because multiple concurrent streams can overlap and be listening for streaming updates for the same job, the JobResourceUtilizationResponse is not tied to a single requestId for a stream, but can contain an update that is of interest to multiple streams and multiple requestIds. This is described in the StreamSequence definition below.
JobResourceUtilizationResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 6 | |
| cpuPercent | The CPU percent used since the last streaming update was sent. May be null. | Decimal Number |
| cpuSeconds | The number of total elapsed CPU seconds. May be null. | Decimal Number |
| virtualMemory | The total number of MB of virtual memory in use. May be null. | Decimal Number |
| residentMemory | The total number of MB of resident (actual/physical) memory in use. May be null. | Decimal Number |
| sequences | An array of stream sequences indicating which requests this update satisfies. | StreamSequence Array |
| complete | Flag indicating whether or not the resource utilization stream is complete. Set to true if the job has and will never have more utilization (such as if the job is finished or killed). | Boolean |
StreamSequence
| Field name | Description | Value |
|---|---|---|
| requestId | The requestId of a stream request that matches this update. |
Int |
| seqId | The monotonically increasing sequence number for the update for this particular requestId. Begins at 1. |
Int |
For an example stream flow, see the section Stream Example.
11.4.3.10 Job Network Request
The job network request allows users to fetch the network information about the host that is running a job.
JobNetworkRequest
| Field name | Description | Value |
|---|---|---|
| messageType | 8 | |
| username | The username of the requestor. “*" if requested by a super user or auth is disabled. You must ensure that users can only view the netwowrk info of their own jobs, as appropriate. | String |
| jobId | The ID of the job. | String |
JobNetworkResponse
| Field name | Description | Value |
|---|---|---|
| messageType | 7 | |
| host | The hostname or IP address of the host that is running the job. | String |
| ipAddresses | A list of IP addresses of the host. | String Array |
11.4.3.11 Errors
An error may be sent in response to a request if an error has occurred. For example, if a request comes in for a job that does not exist, a JobNotFound error can be returned. The various ErrorResponse error codes that can be sent in case of an error are detailed below.
UnknownError = 0,
RequestNotSupported = 1,
InvalidRequest = 2,
JobNotFound = 3,
JobNotRunning = 6,
JobOutputNotFound = 7,
InvalidJobState = 8,
JobControlFailure = 9For example, to indicate a JobNotFound error:
ErrorResponse
{
"messageType": -1,
"errorCode": 3,
"errorMessage": "Job does not exist"
}