1 of 13

operation-guide

For new operators, please please please read all the documentations here before proceeding!

Questions and notes from Pito

exactly how and where is "supervisor" used?
Is the copy of tb3-ros in pitosalas still used for anything?
Using lens.app to look at and work with Rover cluster

architecture.md

Overview

The cloud desktop architecture is simple. On a high level, it looks like this:

Cloud Desktop Cluster

Cloud desktop cluster is a cluster of cloud desktops. It is implemented as a K8s cluster for easy scheduling and orchestration of cloud desktop containers.

Cloud Desktop Container

The cloud desktop container provides a virtualized desktop environment that is isolated and portable. It consists of 3 components.

VNC server paired with a NoVNC server
VSCode server
Tailscale client

For details, see .

Networking

K8s network:

Used for communication with the load balancer to allow each container to be accessible from a URL
Implemented with Flannel

Tailscale network:

Used for communication between cloud desktops and robots globally
Managed with

AWS Route53:

Provides DNS records for redirecting traffic to the cluster

cluster.md

Cloud Desktop cluster is managed by Kubernetes. Specifically, it is a distro of Kubernetes named K3s.

Management

For cluster management, see Cluster Management.

Configurations

We use to run our k8s cluster. While leaving most configurations as default, we made the following customization,

container runtime: docker

Namespaces

clouddesktop-prod
- Main namespace
- Used for all active cloud desktops

Networking

The default networking backend is flannel with VXLAN as backend.

Ingress

The default ingress controller is traefik.

Storage

The default storage provider is local-path. In other words, we store all cloud desktop files locally on the node.

Node roles

As of May 2021

Name

Role

faq.md

User

How do I add a new user?

Read User Management.

How do I delete a user?

Read .

How do I upgrade/downgrade a desktop?

Read .

How do I see all the active desktops in the cluster?

Image

How do I add a package to the image?

Read .

To make sure that new desktops are receiving this update, modify the deployment template under .

Node

Is it safe to restart a node?

Yes, but make sure to drain the node first. To drain the node,

After the reboot, k3s will be started automatically. If it is not started,

Troubleshooting

Desktop connectivity issue

If a group or all desktops are not connecting,

If only 1 desktop is not connecting,

graphs

graphs.md

image.md

Cloud Desktop Container uses a custom docker image. The Dockerfile is located here.

Internals

Components

There are 3 main components in the container image,

VNC server paired with a NoVNC server
VSCode server
Tailscale client

Defaults

Catkin Workspace: /my_ros_data/catkin_ws

Ports:

novnc 80
vnc 5900
vscode 8080

Layers

The current container image is structured this way:

cosi119/tb3-ros

Installs ROS melodic and ROS packages
Installs custom packages used in class, like prrexamples

cosi119/ubuntu-desktop-lxde-vnc

Provides a Ubuntu image with novnc and lxde preconfigured.
Provides a CUDA enabled variant (image with -cuda tag suffix)

Process Management

Supervisord

Each of the components are managed by a process control system called supervisord. Supervisor is responsible for spawning and restarting these components. For detailed configs, see .

Modifing startup processes

Modify the supervisord.conf under tb3-ros/tb3-ros/files/supervisor/supervisord.conf.

Packages

Default packages

As of version 2.1.1,

turtlebot3_msgs
turtlebot3
turtlebot3_simulations

Adding a new package

To add a package to the default catkin workspace, modify the Dockerfile under tb3-ros/tb3-ros/Dockerfile:

Github repo

lifecycle.md

Desktop Creation

Desktop Destruction

nodes.md

Overview

Each nodes in the cluster is setup in a similar way. The software stack on a node looks like this:

Container runtime

The default runtime is nvidia for all Nvidia-enabled nodes. For details, see /etc/docker/daemon.json.

Software versions

As of May 2021

Name

K3s Version

Docker Version

Node hardwares

Name

Hardware Configuration

operating

cluster.md

We administrate all our operations through the use of kubectl.

Prerequisites

Read up on the following materials,

Requirements

kubectl

Setup

To access the K8s cluster, you will need to have a k3s.yaml credential file. It can be obtained by ssh into the master node of the cluster, under the directory /etc/rancher/k3s/k3s.yaml.

Once you have obtained the k3s.yaml file, make the following modification,

After the modification, this file is ready for use. Update your shell to always use this file,

To confirm it working,

Common Operations

See all nodes in cluster

See a specific node

See all deployed pods

Notice that -n clouddesktop-prod refers to the clouddesktop-prod k8s namespace

See a specific pod

Draining a node

users.md

Requirements

kubectl Installation Guide
terraform
AWS Credentials to AWS Route53
- Access Key
- Secret Key

Setup

First obtain the users repo from .

Setup .env file by filling in all required fields:

It's better to use an IAM user group to create a new user associated with the clouddesktop user group. It will generate a access and secret key for you to put in the above file. The ingress IP is the ip address of the main node. Once everything is properly setup, do:

Setup terraform:

Common Operations

To under what each of these commands do under the hood, see .

Add a new user

id is the ID of the new user.

Delete a user

Warning: This will remove any persisted data!!

Change resources allocation for user

For user example, modify the file example-clouddesktop/deployment.yaml.

To increase minimum resource

For detailed explanation of what units you can change it to, see .

To increase maximum resource limit

For detailed explanation of what units you can change it to, see .

Add GPU support

Note: Beware if we have enough free GPUs in the cluster

Note: Make sure the docker image is a CUDA enabled variant (ie. tb3-ros:v2.1.1-cuda)

Apply changes

Warning: This will restart the cloud desktop container!!

To apply the previously changed values,

Restarting a Desktop

To restart a desktop, you need to delete and redeploy the desktop.

This will NOT lead to loss of data.

sources.md

All the source codes of Cloud Desktop is distributed in the following GitHub repos,

Cloud Desktop Image https://github.com/pitosalas/tb3-ros
Cloud Desktop K8s files https://github.com/campusrover/clouddesktop-k8s
Standalone Cloud Desktop https://github.com/campusrover/clouddesktop-docker

users.md

Requirements

kubectl Installation Guide
terraform
AWS Credentials to AWS Route53
- Access Key
- Secret Key

Setup

First obtain the users repo from .

Setup .env file by filling in all required fields:

Setup terraform:

Common Operations

To under what each of these commands do under the hood, see .

Add a new user

id is the ID of the new user.

Delete a user

Warning: This will remove any persisted data!!

Change resources allocation for user

For user example, modify the file example-clouddesktop/deployment.yaml.

To increase minimum resource

For detailed explanation of what units you can change it to, see .

To increase maximum resource limit

For detailed explanation of what units you can change it to, see .

Add GPU support

Note: Beware if we have enough free GPUs in the cluster

Note: Make sure the docker image is a CUDA enabled variant (ie. tb3-ros:v2.1.1-cuda)

Apply changes

Warning: This will restart the cloud desktop container!!

To apply the previously changed values,

Restarting a Desktop

To restart a desktop, you need to delete and redeploy the desktop.

This will NOT lead to loss of data.

image.md

Cloud Desktop Container uses a custom docker image. The Dockerfile is located here.