For new operators, please please please read all the documentations here before proceeding!

Questions and notes from Pito

• exactly how and where is "supervisor" used?

• Is the copy of tb3-ros in pitosalas still used for anything?

• Using lens.app to look at and work with Rover cluster

Cloud Desktop Container uses a custom docker image. The Dockerfile is located here.

Internals

Components

There are 3 main components in the container image,

• VNC server paired with a NoVNC server

• VSCode server

• Tailscale client

Defaults

Catkin Workspace: /my_ros_data/catkin_ws

Ports:

• novnc 80

• vnc 5900

• vscode 8080

Layers

The current container image is structured this way:

cosi119/tb3-ros

• Installs ROS melodic and ROS packages

• Installs custom packages used in class, like prrexamples

cosi119/ubuntu-desktop-lxde-vnc

• Provides a Ubuntu image with novnc and lxde preconfigured.

• Provides a CUDA enabled variant (image with -cuda tag suffix)

Process Management

Supervisord

Each of the components are managed by a process control system called supervisord. Supervisor is responsible for spawning and restarting these components. For detailed configs, see supervisord.conf.

Modifing startup processes

Modify the supervisord.conf under tb3-ros/tb3-ros/files/supervisor/supervisord.conf.

Packages

Default packages

As of version 2.1.1,

• turtlebot3_msgs

• turtlebot3

• turtlebot3_simulations

• https://github.com/campusrover/prrexamples

• https://github.com/campusrover/gpg_bran4

Adding a new package

To add a package to the default catkin workspace, modify the Dockerfile under tb3-ros/tb3-ros/Dockerfile:

# Add the following lines
WORKDIR /my_ros_data/catkin_ws/src
RUN git clone --recursive --depth=1 https://github.com/ROBOTIS-GIT/turtlebot3_msgs.git

Github repo

pitosalas/tb3-ros

Overview

The cloud desktop architecture is simple. On a high level, it looks like this:

Cloud Desktop Cluster

Cloud desktop cluster is a cluster of cloud desktops. It is implemented as a K8s cluster for easy scheduling and orchestration of cloud desktop containers.

Cloud Desktop Container

The cloud desktop container provides a virtualized desktop environment that is isolated and portable. It consists of 3 components.

• VNC server paired with a NoVNC server

• VSCode server

• Tailscale client

For details, see Container Image.

Networking

K8s network:

• Used for communication with the load balancer to allow each container to be accessible from a URL

• Implemented with Flannel

Tailscale network:

• Used for communication between cloud desktops and robots globally

• Managed with Tailscale Dashboard

AWS Route53:

• Provides DNS records for redirecting traffic to the cluster

Desktop Creation

Desktop Destruction

Cloud Desktop cluster is managed by Kubernetes. Specifically, it is a distro of Kubernetes named .

Management

For cluster management, see .

Configurations

We use to run our k8s cluster. While leaving most configurations as default, we made the following customization,

• container runtime: docker

Namespaces

• clouddesktop-prod

  • Main namespace

  • Used for all active cloud desktops

• clouddesktop-dev

  • For testing cloud desktop images

Networking

The default networking backend is flannel with VXLAN as backend.

Ingress

The default ingress controller is traefik.

Storage

The default storage provider is local-path. In other words, we store all cloud desktop files locally on the node.

Node roles

As of May 2021

Overview

Each nodes in the cluster is setup in a similar way. The software stack on a node looks like this:

Container runtime

The default runtime is nvidia for all Nvidia-enabled nodes. For details, see /etc/docker/daemon.json.

Software versions

As of May 2021

Node hardwares

User

How do I add a new user?

Read .

How do I delete a user?

Read .

How do I upgrade/downgrade a desktop?

Read .

How do I see all the active desktops in the cluster?

Image

How do I add a package to the image?

Read .

Node

Is it safe to restart a node?

Yes, but make sure to drain the node first. To drain the node,

After the reboot, k3s will be started automatically. If it is not started,

Troubleshooting

Desktop connectivity issue

If a group or all desktops are not connecting,

If only 1 desktop is not connecting,

All the source codes of Cloud Desktop is distributed in the following GitHub repos,

• Cloud Desktop Image https://github.com/pitosalas/tb3-ros

• Cloud Desktop K8s files https://github.com/campusrover/clouddesktop-k8s

• Standalone Cloud Desktop https://github.com/campusrover/clouddesktop-docker

Requirements

• kubectl Installation Guide

• Setup Instructions

• terraform Installation Guide

• AWS Credentials to AWS Route53

  • Access Key

  • Secret Key

Setup

First obtain the users repo from here.

Setup .env file by filling in all required fields:

mv .env.sample .env

It's better to use an IAM user group to create a new user associated with the clouddesktop user group. It will generate a access and secret key for you to put in the above file. The ingress IP is the ip address of the main node. Once everything is properly setup, do:

export $(make env)

Setup terraform:

terraform init

Common Operations

To under what each of these commands do under the hood, see here.

Add a new user

id is the ID of the new user.

make add-user id=example

Delete a user

Warning: This will remove any persisted data!!

make delete-user id=example

Change resources allocation for user

For user example, modify the file example-clouddesktop/deployment.yaml.

To increase minimum resource

For detailed explanation of what units you can change it to, see here.

resources:
  requests:  # increase minimum to at least 4 cores
-   cpu: 2
+   cpu: 4
    memory: 2Gi

To increase maximum resource limit

For detailed explanation of what units you can change it to, see here.

resources:
  limits: # increase to maximum 16GB of ram
    cpu: 8
-   memory: 8Gi
+   memory: 16Gi

Add GPU support

Note: Beware if we have enough free GPUs in the cluster

Note: Make sure the docker image is a CUDA enabled variant (ie. tb3-ros:v2.1.1-cuda)

resources:
  limits: # increase to maximum 16GB of ram
    cpu: 8
    memory: 8Gi
+   nvidia.com/gpu: 1

Apply changes

Warning: This will restart the cloud desktop container!!

To apply the previously changed values,

kubectl apply -k example-clouddesktop

Restarting a Desktop

To restart a desktop, you need to delete and redeploy the desktop.

This will NOT lead to loss of data.

kubectl delete -k example-clouddesktop

kubectl apply -k example-clouddesktop

We administrate all our operations through the use of kubectl.

Prerequisites

Read up on the following materials,

• K8s Concepts

• kubectl cheatsheet

Requirements

• kubectl Installation Guide

Setup

To access the K8s cluster, you will need to have a k3s.yaml credential file. It can be obtained by ssh into the master node of the cluster, under the directory /etc/rancher/k3s/k3s.yaml.

Once you have obtained the k3s.yaml file, make the following modification,

# Update with the IP of the master node
- server: https://localhost:6443
+ server: https://123.123.123.123:6443

After the modification, this file is ready for use. Update your shell to always use this file,

export KUBECONFIG=/.../k3s.yaml

To confirm it working,

kubectl get pods --all-namespaces

Common Operations

See all nodes in cluster

kubectl get nodes

See a specific node

kubectl describe node robotics-rover1

See all deployed pods

• Notice that -n clouddesktop-prod refers to the clouddesktop-prod k8s namespace

kubectl -n clouddesktop-prod get pods

See a specific pod

kubectl -n clouddesktop-prod describe pod julianho-clouddesktop-deployment-abc123efg-123abc

Draining a node

kubectl drain robotics-rover2