by Kirsten Tapalla - Spring 2023 This is a quick guide to connecting to ChatGPT and getting it to generate a message that you can publish to a topic for use in other nodes.

Necessary Imports:

• import requests: used to connect to the internet and post the url to connect to ChatGPT

• import rospy: used to publish the response from ChatGPT to a topic

• from std_msgs.msg import String: used to publish the response as a ros String message

Node Initialization and Variables/Parameters:

• Since you want to publish the message into a topic, you will have to initialize a rospy node withing your file using rospy.init_node('ENTER-NODE-NAME-HERE').

• You will also want to initialze a rospy Publisher with the name of the topic you would like to publish the data to, for example: text_pub = rospy.Publisher('/chatgpt_text', String, queue_size=10).

• If you would like to be able to change the prompt you are passing it when running the node, you should add a line allowing you to do this by initializing an input string to get the parameter with the name of what you would like to use to specify the input message you would like to pass it.

  • For example, do this by writing input_string = rospy.get_param('~chatgpt_prompt')

  • When setting up your launch file later on, you will want to include a line to handle this argument. This can be done by including arg name="chatgpt_prompt" default="ENTER-YOUR-DEFAULT-PROMPT-HERE" into your launch file. You can set the default to whatever default prompt you would like to be passed if you are not giving it a specific one.

• You will also want to add this line into your code to to specify the URL going that will be used to connect to ChatGPT: url = 'https://api.openai.com/v1/completions'. Since the chat/text completions model is what we are using the get the output responses, the URL specifies 'completions' at the end.

ChatGPT Information:

Headers:

To be able to access ChatGPT, you will need to include the following information in your 'header' dictionary: Content-Type and Authorization. Below is an example of what yours might look like: headers = { 'Content-Type': 'application/json', 'Authorization': 'Bearer INSERT-YOUR-OPENAI-API-KEY-HERE', }

Data:

This will include the information you will want to pass into ChatGPT. The only required field will be specifying the model you want to use, but since you are passing in a prompt, you will also want to include that as well. You will be able to specify the maximum amount of tokens you want ChatGPT to generate, but the best way to get the full output messeage from ChatGPT is to enter the maximum amount for the model you are using. For example, as you can see below I am using the 'text-davinci-003' model, and the maximum tokens that this model can generate is 2048. Furthermore, you can adjust the sampling temperature, which will determine how creative the output of ChatGPT will be. The range goes between 0-2, and higher values will cause it to be more random, while lower values will cause it to be more focused and deterministic. An example of a request body you can make is shown below: data = { 'model': 'text-davinci-003', 'prompt': input_string, 'max_tokens': 2048, 'temperature': 0.5, }

Getting and Publishing the Response

To get the response for your input message from ChatGPT, include the following line in your code response = requests.post(url, headers=headers, json=data). Note that if you are using different names for your variables, you will want to pass in those names in place of 'headers' and 'data'. To publish your output, you will want to make sure that your request went through. If it did, you will be able to get the output from the json file that was returned in the response variable. An example of how to do this is shown below: if response.status_code == 200: generated_text = response.json()['choices'][0]['text'] text_pub.publish(generated_text) By doing all of the steps above, you will be able to connect to ChatGPT, pass it a prompt, and publish its response to that prompt to a topic in ros.

Steps for contributing to this section

1. Git clone this repository to your computer: campusrover/labnotebook

2. In the directory structure, find the folder called faq. In it you can create a .md file for your contribution.

3. Write your contribution using markdown. You can include images as well as links. There are many tutorials on Markdown but you most likely already know it.

4. If you want the title to be something other than the file name, then add:

---
title: Gooder Title
author: Pito Salas
description: nicer subtitle
order: where does this one appear in order
status: obsolete|new|tophit
date: month-year
---

as the first few lines of the file 5. Git add, commit and push your changes.

Note

Brand new files (vs. edits) will not appear in the directory until it is rebuilt by rpsalas@brandeis.edu so send them an email!

Ensure that the camera is enabled and there is enough GPU memory:

1. Add the following lines to the /boot/firmware/config.txt file:

start_x=1
gpu_mem=128

And then reboot. Note that the config.txt file will say that you should not modify it and instead make the changes in a file called userconfig.txt. However I found that the userconfig.txt file is not invoked. So I added the lines above directly to config.txt.

1. Check the amount of gpu_memory

vcgencmd get_mem gpu

It should show 256 or whatever number you put there.

Notes

You really need to know what you are doing when you are dealing at this level.

Radical Cleanup

• It turns out that it is safe to delete the build and devel subdirectories in ROS.

• Sometimes this helps:

cd ~/catkin_ws
rm -rf build/ devel/
catkin_make
source devel/setup.bash

Another way

• First: lean your build by running "catkin_make clean" in the root of your workspace.

• Second: remake your project with "catkin_make"

• Third: re-source the devel/setup.bash in your workspace.

Brandon J. Lacy

Overview

The utilization of GPS data on a robot is a common requirement within projects. However, a majority of hardware components that can be configured with the robot produce lackluster results. The iPhone uses sophisticated technology to produce more accurate GPS data, which makes it a prime candidatee for situation in which a robot is in need of accurate information. The application, GPS2IP, uses the technology of the iPhone and communicates it over the internet. Through this application and the iPhone technology an accurate vehicle to produce GPS data is obtained.

GPS2IP

The application is solely available on iOS. No research was conducted on applications on Android that produce similar functionality. There are two versions of the application on the App Store: GPS2IP ($8.99) and GPS2IP Lite (Free). The free version only allows transmission for 4 minutes before automatically turning off. The paid version has no restrictions.

Walkthrough

iPhone Configuration

Turn Off Auto-Lock

Settings > Display & Brightness > Auto-Lock > Never

GPS2IP Configuration

Enable GPS2IP

Open GPS2IP > Toggle On "Enable GPS2IP" Switch

NMEA Message Type

Open GPS2IP > Settings > NMEA Messages to Send > Only Toggle On "GLL" Switch

Python Code

#!/usr/bin/env python

'''
A module with a GPS node.

GPS2IP: http://www.capsicumdreams.com/gps2ip/
'''

import json
import re
import rospy
import socket

from std_msgs.msg import String

class GPS:
    '''A node which listens to GPS2IP Lite through a socket and publishes a GPS topic.'''
    def __init__(self):
        '''Initialize the publisher and instance variables.'''
        # Instance Variables
        self.HOST = rospy.get_param('~HOST', '172.20.38.175')
        self.PORT = rospy.get_param('~PORT', 11123)

        # Publisher
        self.publisher = rospy.Publisher('/gps', String, queue_size=1)

    def get_coords(self):
        '''A method to receive the GPS coordinates from GPS2IP Lite.'''
        # Instantiate a client object
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.connect((self.HOST, self.PORT))
            # The data is received in the RMC data format
            gps_data = s.recv(1024)

        # Transform data into dictionary
        gps_keys = ["message_id", "latitude", "ns_indicator", "longitude", "ew_indicator"]
        gps_values = re.split(',|\*', gps_data.decode())[:5]
        gps_dict = dict(zip(gps_keys, gps_values))

        # Cleanse the coordinate data
        for key in ['latitude', 'longitude']:
            # Identify the presence of a negative number indicator
            neg_num = False

            # The GPS2IP application transmits a negative coordinate with a zero prepended
            if gps_dict[key][0] == '0':
                neg_num = True
            
            # Transform the longitude and latitude into a format that can be utilized by the front-end web-client
            gps_dict[key] = float(gps_dict[key]) / 100

            # Apply the negative if the clause was triggered
            if neg_num:
                gps_dict[key] = -1 * gps_dict[key]

        # Publish the decoded GPS data
        self.publisher.publish(json.dumps(gps_dict))


if __name__ == '__main__':
    # Initialize a ROS node named GPS
    rospy.init_node("gps")

    # Initialize a GPS instance with the HOST and PORT
    gps_node = GPS()

    # Continuously publish coordinated until shut down
    while not rospy.is_shutdown():
        gps_node.get_coords()

Author - Eyal Cohen

Given two coordinates in a 2-dimensional plane and your robots' current direction, the smart rotation algorithm will calculate your target angle and return whether your robot should to turn left, turn right, or go straight ahead to reach its navigation goal.

How It Works:

Inputs: posex1 is a float that represents the x-axis coordinate of your robot, posey1 is a float that represents your robot's y-axis coordinate. posex2 and posey2 are floats that represent the x and y of your robot's goal. Lastly, theta represents your robot's current pose angle.

from math import atan2
import math


def smartRotate(posex1, posey1, posex2, posey2, theta):
    inc_x = posex2 -posex1
    inc_y = posey2 -posey1
    angle_goal = atan2(inc_y, inc_x)
    # if angle_to_goal < 0:
    #     angle_to_goal = (2* math.pi) + angle_to_goal
    print("angle goal = ",angle_goal)
    goal_range = theta + math.pi
    wrapped = goal_range - (2 * math.pi)
    if abs(angle_goal - theta) > 0.1:
        print(theta)
        print("goal_range = ",goal_range)
        if (goal_range) > (2 * math.pi) and (theta < angle_goal or angle_goal < wrapped):
            print("go left")
        elif (goal_range) < (2 * math.pi) and (theta < angle_goal and angle_goal < goal_range):
            print("go left")
        else:
            print("go right")

    else:
        print("go straight")

Firstly, the angle_goal from your robot's coordinate (not taking its current angle into account) is calculated by finding the arc tangent of the difference between the robot's coordinates and the goal coordinates.

In order to decide whether your robot should go left or right, we must determine where the angle_goal is relative to its current rotational direction. If the angle_goal is on the robot's left rotational hemisphere, the robot should rotate left, otherwise it should rotate right. Since we are working in Radians, π is equivilant to 180 degrees. To check whether the angle_goal is within the left hemisphere of the robot, we must add π to theta (the robot's current direction) to get the upperbound of the range of values we want to check the target may be included in. If the angle_goal is between theta and that upper bound, then the robot must turn in that direction to most efficiently reach its goal.

Consider This Example:

    smartRotation(0,0,2,2,0)

If your robot is at (0,0), its rotational direction is 0, and it's target is at (2,2), then its angle_goal would equal = 0.785. First we check whether its current angle's deviation from the angle_goal is significant by finding the difference and seeing if its larger than 0.1. If the difference between the angles is insignificant the robot should go straight towards its goal. In this case however, angle_goal - theta (0.785 - 0) is greater than 0.1, so we know that we must turn left or right to near our angle_goal. To find out whether this angle is to the left or the right of the robot's current angle, we must add π to its current angle to discover the point between its left and right hemispheres. In this case, if the angle_goal is between theta and its goal_range, 3.14 (0(theta) + π), then we would know that the robot must turn left to reach its goal.

However, if theta (your robot's current direction) + π is greater than 2π (maximum radians in a circle) then the left hemisphere of your robot is partially across the 0 radian point of the circle. To account for that case, we must calculate how far the goal range wraps around the circle passed the origin. If there is a remainder, we check whether the angle_goal is between theta and 2π or if the angle_goal is present within the remainder of the range that wraps around the origin. If either of these conditions are met then we know that your robot should turn left to most efficiently arrive at its goal, otherwise it should turn right.

1. plug in battery and turn on robot with the power switch, give it a moment and wait for the lidar to start spinning.

2. run tailscale status | grep <name> to find the robot’s IP address. Replace with the name of the robot you are trying to connect to.

1. go to .bashrc in my_ros_data folder and get it to look like this with your robot’s name and IP address instead:

1. Open a new terminal and you should see:

1. You can then ssh into the robot, ssh ubuntu@100.117.252.97 (enter your robot’s IP) and enter the password that is in the lab.

2. Once onboard the robot, enter the command bringup which starts roscore and the Turtlebot’s basic functionalities. For the Platform robots, run this: roslaunch platform full_bringup.launch

3. After that, open a new terminal (you’ll be in real mode again) and run your program!

4. To go back to simulation mode, go back to .bashrc and uncomment the settings for simulation mode and comment out the settings for a physical robot. Or type the command sim in the terminal. You will need to do this in every terminal that you open then. To switch to real mode, type command real.

When you're done, run sudo shutdown now onboard the robot (where is ran bringup) and then turn off the robot with the power switch.

Yolo8 is a cutting-edge, state-of-the-art (SOTA) model that builds upon the success of previous YOLO versions and introduces new features and improvements to further boost performance and flexibility. YOLOv8 is designed to be fast, accurate, and easy to use, making it an excellent choice for a wide range of object detection and tracking, instance segmentation, image classification and pose estimation tasks. It is a powerful model that can be used to detect multiple objects in an image.

It has been wrapped into a user-friendly python package Ultralytics (https://docs.ultralytics.com/). To detect objects of interest, the pre-trained model yolo8 can be used. Or one can customize the yolo8 model by training it with provided train image data. Here the website Roboflow (https://roboflow.com/) has a variety of object datasets, e.g. traffic sign dataset (https://universe.roboflow.com/usmanchaudhry622-gmail-com/traffic-and-road-signs/model/1). Once the dataset is downloaded and the Ultralytics package is installed, the yolo8 model can be trained easily:

yolo detect train model=yolov8n.pt data=traffic_sign.yaml

Where the traffic_sign.yaml is the path to the yaml file inside your downloaded dataset from roboflow. You have the options to use various yolo8 models. See Detection Docs for usage examples with these models.

The larger the model is, the higher the latency will present in the ros node that holds the model. Therefore, the smaller model should be used as long as the model works for the objects of interest.

This is a quick ann simply way to train a customized model that is powerful in objects detection of robot vision.

Author: Michael Jiang

How do I download files from my online VSCode to my local computer?

While the online VSCode does not offer a way to download entire folders and their contents with one click, there is a non-obvious functionality that allows you to download individual files directly from VSCode.

Right click the desired file and select 'Download' from the menu. The file will download to your default download location on your computer.

You will be able to now access the files from your local system and submit them to Gradescope. To submit packages like this, you can set up a Git repository and push the files there, or you can individually download each file in the package using this method, arrange them properly, and submit it on Gradescope.

by Helen Lin edited by Jeremy Huey

Introduction

This guide shows you how to create launch files for your code so that you can launch multiple nodes at once rather than running each node individually.

Step 1

Open a new terminal window and navigate to a package you want to create a launch file in. Create a folder called 'launch'.

mkdir launch

Step 2

Navigate to the launch directory and create a new launch file ending in .launch. Replace 'name' with the name of your launch file.

touch name.launch

Step 3

Add the following code and fill in the parameters within the double quotes.

Here is an example of what the node will look like filled in, using code from the Mini Scouter project:

The pkg name can be found in the package.xml. Eg.

Step 4

Make sure you have run the following command on all of the files used in the launch file so they can all be found by ROS launch. Replace 'name' with the name of the python file.

chmod +x name.py

Change the permissions of the launch file as well by going to the launch directory and running the following command. Replace 'name' with the name of the launch file.

chmmod +x name.launch

Step 5

Open a new terminal window and run the following command. Replace 'package_name' with the name of the package and 'name' with the name of the launch file.

roslaunch package_name name.launch

For example, to run the Mini Scouter launch file:

roslaunch mini_scouter teleop.launch

All of the nodes you specified in the launch file should now be running.

Optional, launch process in new window

To have a node launch and open in a new window, such as to run things like key_publisher.py, you can modify the line to include this: <node pkg="object_sorter" type="key_publisher.py" name="key" output="screen" launch-prefix="xterm -e"/> You must then run in terminal:

More info on launch files

To continue to get more information on launch files, go to here: labnotebook/faq/launch-files.md

Question

I want to run a computer vision algorithm on my robot, but I'm told that I need to calibrate my camera(s) first. What is camera calibration, and how can I do it?

What is Camera Calibration?

Camera calibration is the act of determining the intrinsic parameters of your camera. Roughly speaking, the intrinsic parameters of your camera are constants that, in a mathematical model of your camera, describe how your camera (via its interior mechanisms) converts a 3D point in the world coordinate frame to a 2D point on its image plane.

Intrinsic parameters are distinct from extrinsic parameters, which describe where your camera is in the world frame.

So, since calibration deals with the intrinsic parameters of your camera, it practically doesn't matter where you place your camera during calibration.

To hear more about the basics of camera calibration, watch the following 5-minute videos by Cyrill Stachniss in order:

, also by Cyrill Stachniss, is a deep dive into Zhang's method, which is what the camera_calibration package we discuss below uses under the hood.

How Can I Calibrate my Camera?

This note describes two ways you can calibrate your camera. The first is by using the camera_calibration ROS package. This is the easier approach, since it basically does almost all of the work for you. The second is by using OpenCV's library directly, and writing your own calibration code (or using one in circulation).

The camera_calibration Package

First, let's install the package:

Third, tape the corners of the paper to a firm, flat surface, like the surface of a piece of cardboard.

Fourth, measure a side of a single square, convert your measurement to millimeters, and divide the result by 1000. Let's call your result, RESULT.

Now, let the number of rows of your checkerboard be M and its number of columns N. Finally, let's say your camera node's name is CAM, such that, when you connect it with ROS, it publishes the /CAM/camera_info and /CAM/image_raw topics. Now, after ensuring that these two topics are being published, execute:

WARNING The two sections stated above are the only ones you actually want to follow in the official tutorial. Much of the rest of the material there is outdated or misleading.

OpenCV

Sometimes, you might want to use object detection or use certain algorithms that require a camera such as VSLAM. These algorithms usually require a very good calibration of the camera to work properly. The calibration fixes things like distortion by determining the camera’s true parameters such as focal length, format size, principal point, and lens distortion. If you see lines that are curved but are supposed to be straight, then you should probably calibrate your camera.

The most ideal way to do is to print the checkerboard on a large matte and sturdy piece of paper so that the checkerboard is completely flat and no reflections can be seen on it. However, it’s okay to just print it on a normal piece of paper as well and put it on a flat surface. Then, take at least ten photos with your camera from a variety of angles and positions so that the checkerboard is in all corners of the photos. Make sure the whole checkerboard is seen in each picture. Save those photos in an easy to find place and use the following to get your intrinsic calibration matrix.

Step by step:

• print out checkerboard pattern

• take at least 10 photos of the checkerboard at a variety of angles and positions (see image 1 for examples) and save in an easy to access place

• download/copy the opencv calibration code and run it after changing the folder path

• get the intrinsic matrix and distortion and enter it into whatever you need

Image 1: some examples of having the checkerboard cover all corners of the image

Potential intrinsic matrix:

[[688.00030953 0. 307.66412893]

[ 0. 689.47629485 274.053018 ]

[ 0. 0. 1. ]]

Pastable into python:

fx, fy, cx, cy = (688.00030953, 689.47629485, 307.66412893, 274.053018)

Distortion coefficients:

[9.39260444e-03, 4.90535732e-01, 1.48962564e-02, 4.68503188e-04, -1.77954077e+00]

Line follower may work well and easy to be done in gazebo because the color is preset and you don't need to consider real life effect. However, if you ever try this in the lab, you'll find that many factors will influence the result of your color.

Real Life Influencer:

1. Light: the color of the tage can reflect in different angles and rate at different time of a day depend on the weather condition at that day.

2. Shadow: The shadow on the tape can cause error of color recognization

3. type of the tape: The paper tage is very unstable for line following, the color of such tape will not be recognized correctly. The duct tape can solve most problem since the color that be recognized by the camera will not be influenced much by the light and weather that day. Also it is tougher and easy to clean compare to other tape.

4. color in other object: In the real life, there are not only the lines you put on the floor but also other objects. Sometimes robots will love the color on the floor since it is kind of a bright white color and is easy to be included in the range. The size of range of color is a trade off. If the range is too small, then the color recognization will not be that stable, but if it is too big, robot will recognize other color too. if you are running multiple robots, it might be a good idea to use electric tape to cover the red wire in the battery and robot to avoid recognizing robot as red line.

So if we use color pick we find online, we may need to rescale it to opencv's scale.

Introduction

Amazon Web Services RoboMaker is an online service which allows users to develop and test robotic applications online. Robomaker has a number of advanced features, but this notebook page will focus on developing nodes and simulating them in Gazebo. To use RoboMaker, you will also need to use AWS S3 and Cloud9.

Getting started

Create an AWS root account if you do not already have one. A free tier account will suffice for getting started, though make note that under a free tier membership you will be limited to 25 Simulation Units (hours) for the first twelve months.

Once you have an AWS account, you should create an IAM for your account. AWS recommends not using your root user account when using services like RoboMaker. . Remember the username and password of the account you create. Additionally, save the link to where you can log in with those credentials.

Going forward, you should be logged in with your IAM account. Log into AWS with your IAM, then proceed.

Amazon Documentation and Mini Tutorial

RoboMaker has a limited documentation set that can help you use the software. The “getting Started” section can help familiarize yourself with the software by working with a sample application. .

Creating an S3 bucket

From the AWS Management Console, type “S3” into the “find services” field and click on S3 in the autofill list below the entry box. From the S3 Management Console, click “Create Bucket”

• On the first page, enter a name for your bucket. Under region, make sure that it is US East (N Virginia), NOT US East (Ohio), as RoboMaker does not work in the Ohio region.

• Skip step 2, “configure options”

• In step 3, “Set Permissions”, uncheck all four boxes

• Click “Create Bucket”

This bucket is where your bundled robotic applications will be stored.

Creating a Development Environment

Beck at the AWS Management Console, type “robomaker” in to the same entry field as S3 in the last part to go to the RoboMaker Management Console. In the left hand menu, under Development, select “Development environments” then click “Create Environment”

• Give your environment a name

• Keep the instance type as its default, m4 large

• Under networking, select the default VPC and any subnet, then click “create” to finish creating your environment

At the end, you will have both a robot workspace and a simulation workspace. The robot workspace (robot_ws) contains source files which are to be run on the robot. The simulation workspace (simulation_ws) contains the launch files and the world files needed to run gazebo simulations. Going forward this guide assumes that you have directories set up exactly as described in the walkthrough linked above, especially that the folders robot_app and simulation_app exist inside the src folders of robot_ws and simulation_ws, respectively.

Adding new scripts to robot_ws

Python ROS node files should be stored in the scripts folder inside robot_app. When you add a new node, you must also add it to the CMakeLists.txt inside robot_app. In the section labelled “install” you will see scripts/rotate.py, below that line is where you should list all file names that you add (with scripts/ preceding the name).

Modifying the Simulation

When creating your directories, two files were put inside simulation_app/launch: example.launch and spawn_turtlebot.launch

• Inside spawn_turtlebot.launch, you will see a line that looks like this (it should be on line 3): <arg name="model" default="$(optenv TURTLEBOT3_MODEL waffle_pi)" doc="model type [burger, waffle, waffle_pi]"/> In this section: default="$(optenv TURTLEBOT3_MODEL waffle_pi)" you can replace waffle_pi with burger or waffle to change the model of turtlebot3 used in the simulation

• Inside example.launch you will see this line (it should be on line 8): <arg name="world_name" value="$(find simulation_app)/worlds/example.world"/> You can replace example.world with the name of another world file to change the world used for the simulation. Note that the world file must be present in the folder simulation_app/worlds. You can copy world files from the folder catkin_ws/src/turtlebot3_simulations/turtlebot3_gazebo/worlds (which is on your personal machine if you have installed ROS Kinetic)

Building and Bundling your applications

In order to use your applications in simulation, they must first be built and bundled to a format that RoboMaker likes to use. At the top of the IDE, click: RoboMaker Run → Add or Edit configurations, then click Colcon Build.

Give your configuration a name. I suggest it be robot or simulation>. For working directory, select the path to either robot_ws or simulation_ws (you will have to do this twice, once for each workspace).

Do the same, but this time for Colcon Bundle.

You now have shortcuts to build and bundle your robot and simulation applications.

Next, in the configuration window that you have been using, select workflow to create a shortcut which will build and bundle both applications with one click. Give your workflow a name, then put your actions in order. It is important that the builds go before the bundles.

Once you’ve made your workflow, go to: RoboMaker Run → workflow → your workflow to build and bundle your applications. This will create a bundle folder inside both robot_ws and simulation_ws. Inside bundle, there is a file called output.tar.gz. You can rename it if you like, but remember where it is.

Finally, we will go back to the configuration window to configure a simulation launcher.

• Give it a name

• Give it a duration (in seconds)

• Select “fail” for failure behavior

• Select an IAM role - it doesn’t necessarily matter which one, but AWSServiceRoleForRoboMaker is recommended

• Skip to the robot application section

• Give it a name

• Bundle path is the path to output.tar.gz (or whatever you renamed it) inside robot_ws/bundle

• S3 bucket should be the bucket you created at the beginning of this guide

• Launch package name is robot_app

• Launch file is the launch file you wish to use

NOTE: the name of the robot application and the launch file should related in some way The simulation application section is much of the same, except everything that was “robot” should be replaced with “simulation.”

OPTIONAL: Once your simulation launch configuration has been saved, you can add it as the final action of the workflow you made earlier.

Running a simulation

These are the steps that have been found to work when you want to run a simulation in RoboMaker. They are Kludgy, and perhaps a more elegant solution exists, but for now this is what has been found:

1. Make sure your applications have been built and bundled. Then from the IDE, go to RoboMaker Run → Simulation Launch → your simulation config. This will upload your application bundles to the S3 bucket you specified, then try to start the simulation. IT WILL PROBABLY FAIL. This is okay, the main goal of this step was to upload the bundles.

2. Go back to the RoboMaker Management Console, and in the left menu Select Simulations → Simulation Jobs, then click “Create simulation job”

3. Now we will configure the simulation job again:

   • Set the failure behavior to fail

   • For IAM role, select “create new role”, then give your role a name. Each simulation job will have its own IAM role, so make the name related to the simulation job.

   • Click next

   • For robot application, select the name you gave when you configured the simulation in the IDE. The launch package name and launch file will be the same too, but you must type those in manually.

   • Click next, the next page is for configuring the simulation application, the same rules apply here as the robot application

   • Click next, review the configuration then click create.

4. RoboMaker will begin preparing to launch your simulation, in a few minutes it will be ready and start automatically. Once it is running, you will be able to monitor it through gazebo, rqt, rviz and the terminal.

5. Be sure that once you are done with your simulation (if it is before the duration expires) to cancel the simulation job under the action menu in the upper right of the simulation job management screen.

This tutorial is largely based on what I have learnt here: . Please refer to this official tutorial if you need more details.

1. Open a Gazebo simulation

First, open Gazebo - either search for gazebo in the Unity Launcher GUI or simply type gazebo onto the terminal. Click on Edit --> Building Editor and you should see the following page. Note there are three areas:

• Platte: You can choose models that you wish to add into the map here.

• 2D View: The only place you make changes to the map.

• 3D View: View only.

2. Import a floor plan

You may create a scene from scratch, or use an existing image as a template to trace over. On the Platte, click on import and selet a 2D map plan image in the shown prompt and click on next.

To make sure the walls you trace over the image come up in the correct scale, you must set the image's resolution in pixels per meter (px/m). To do so, click/release on one end of the wall. As you move the mouse, an orange line will appear as shown below. Click/release at the end of the wall to complete the line. Once you successfully set the resolution, click on Ok and the 2D map plan image you selected should show up in the 2D-View area.

3. Add & Edit walls

• Select Wall from Platte.

• On the 2D View, click/release anywhere to start the wall. As you move the mouse, the wall's length is displayed.

• Click again to end the current wall and start an adjacent wall.

• Double-click to finish a wall without starting a new one.

• Double-clicking on an existing wall allows you to modify it.

4. Prepare a package

You need to create a package for your Gazebo world so that you can use roslaunch to launch your it later.

• Go to your catkin workspace

  $ cd ~/catkin_ws/src

• Create a package using the following command.

  $ catkin_create_pkg ${your_package_name}

• Go to your package and create three folders launch, worlds and models.

5.Save your map

Once you finish editing the map, give a name to your model on the top on the Platte and click on File -> Save As to save the model you just created into ../${your_package_name}/models.

Click on File -> Exit Building Editor to exit. Please note that once you exit the editor, you are no longer able to make changes to the model. Click on File -> Save World As into ../${your_package_name}/worlds.

I will refer to this world file as ${your_world_file_name}.world from now on.

6.Create a launch file for your gazebo map

Go to ../${your_package_name}/launch and make a new file ${your_launch_file} Copy and paste the following code into your launch file and substitute ${your_package_name} and {your_world_file_name} with their actual names.

7. Test

Go to the workspace where your new package was created e.g. cd ~/catkin_ws

run catkin_make and then roslaunch ${your_package_name} ${your_launch_file}

You should see the Gazebo map you just created along with a turtlebot loaded.

Using the Model Editor instead

The building editor is a faster, easier to use tool than the model editor, as it can create a map in mere minutes. With the model editor, you have more technical control over the world, with the trade off being a more tedious process. The model editor can help make more detailed worlds, as you can import .obj files that can be found on the internet or made in 3d modeling software such as Blender. For the purposes of use in this class, USE THE BUILDING EDITOR For your own recreational robotic experimentation purposes, of course, do whatever you like.

If you do wish to use the model editor, here are two tips that will help you to get started making basic, serviceable worlds.

1. Change the Color

The basic shapes that gazebo has are a greyish-black by default- which is difficult to see on gazebo's greyish-black background. To change the color, follow these steps: 1. Right click on the model 1. select "open link inspector" 1. go to the "visual" tab 1. scroll down to "material" and open that section 1. use the RGB values labeled "ambient" to alter the color - set them all to 1 to make it white.

2. Alter the shape

use the shortcut s to open the scaling tool - grab the three axis to stretch the shape. Hold ctrl to snap it to the grid. use the shortcut t to switch to the translation tool - this moves the model around. Hold ctrl to snap it to the grid. use the shortcut r to open the rotation tool. grab the rings to rotate the object.

3. Make it Static

If an object isn't static, it will fall over/ obey the laws of physics if the robot collides with it - to avoid this, click the object in the left hand menu and click the is_static field.

4. Use the Building Editor instead

Does the model editor seem like a hassle already? Then just use the building editor.

If you're having trouble getting a file from your code virtual machine onto your actual computer to submit it onto Gradescope, never fear, an easy solution is here:

1. Right click on the desired file from the file explorer (normally on the left panel) on your code viewer.

2. Select 'Download' and the file will download to your browser's default download location (typically your 'Downloads' folder).

Voila! You have successfully moved a file from your online code to your machine.

ImageNet file xml format to text format.

Full Repo here:

Installation

Usage

Example

Input xml file.

Output text file.

Motivation

The linux bluetooth struggle

There are some troubles that you may run into if you are trying to connect to bluetooth using linux or raspberry pi, so this is a guide to try and overcome those difficulties. Hopefully, it is helpful.

Install Pipewire

Run the following commands to install Pipewire and disable PulseAudio.

To check that Pipewire is properly installed, run

If this doesn't work, try restarting Pipewire or your computer:

If you get the error: Connection failure: Connection refused

Steps taken to get bluetooth headset to run

Check the status of your bluetooth:

To connect your bluetooth device, run the following commands:

Set profile

After this, run:

You'll get a list of devices and the bluetooth device will be in the form of bluez_card.84_6B_45_98_FD_8E

From what I understand, most bluetooth headsets have two different profiles: ad2p and headset-head-unit. To use the microphone, you will need to set the card profile of your bluetooth device to headset-head-unit

Then, test whether or not the device is recording and playing properly:

Change default

You can set the default input and output devices using the following commands.

First check what sources are available:

Then set the default source and sink devices:

By Harris Ripp

The detection of edges in image processing is very important when needing to find straight lines in pictures using a camera. One of the most popular ways to do so is using an algorithm called a Canny Edge Detector. This algorithm was developed by John F. Canny in 1986 and there are 5 main steps to using it. Below are examples of the algorithm in use:

The second image displays the result of using canny edge detection on the first image.

Steps to Edge Detection:

• Find intensity gradients of image

• Apply gradient magnitude thresholding or lower bound cut-off suppression to remove false results

• Track edge by surprisessing weak edges so only strong ones appear

Application of Canny Edge Detection

The below function demonstrates how to use this algorithm:

First, the image is converting into something usable by cv. It is then grayed and the intensity gradient for the kernel is found

The image is then blurred for canny preparation.

The lower and upper bounds are decided and the Canny algorithm is run on the image. In the case of this function, the new image is then published to a topic called "canny mask" for use by another node.

The above code was created for use in a project completed by myself and fellow student Adam Ring

Deploying a Pretrained Pytorch Model in Ubuntu 18.04 Virtual Environment

By Adam Ring

Using a pre-trained deep learning model from a framework such as Pytorch has myriad applications in robotics, from computer vision to speech recognition, and many places inbetween. Sometimes you have a model that you want to train on another system with more powerful hardware, and then deploy the model elsewhere on a less powerful system. For this task, it is extremely useful to be able to transfer the weights of your trained model into another system, such as a virtual machine running Ubuntu 18.04. These methods for model transfer will also run on any machine with pytorch installed.

Note

It is extremely discouraged to mix versions of Pytorch between training and deployment. If you train your model on Pytorch 1.8.9, and then try to load it using Pytorch 1.4.0, you may encounter some errors due to differences in the modules between versions. For this reason it is encouraged that you load your Pytorch model using the same version that is was trained on.

Saving and loading a trained model

Let's assume that you have your model fully trained and loaded with all of the necessary weights.

model = MyModel()

model.train()

For instructions on how to train a machine learning model, see in the lab notebook. There are multiple ways to save this model, and I will be covering just a few in this tutorial.

Saving the state_dict

This is reccommended as the best way to save the weights of your model as its state_dict, however it does require some dependencies to work. Once you have your model, you must specify a PATH to the directory in which you want to save your model. This is where you can name the file used to store your model.

PATH = "path/to/directory/my_model_state_dict.pt"

or

PATH = "path/to/directory/my_model_state_dict.pth"

You can either specify that the state_dict be saved using .pt or .pth format.

Then, to save the model to a path, simply call this line of code.

torch.save(model.state_dict(), PATH)

Loading the state_dict

Download the my_model_state_dict.pt/pth into the environment in which you plan on deploying it. Note the path that the state dict is placed in. In order to load the model weights from the state_dict file, you must first initialize an untrained istance of your model.

loaded_model = MyModel()

Keep in mind that this step requires you to have your model architecture defined in the environment in which you are deploying your model.

Next, you can simply load your model weights from the state dict using this line of code.

loaded_model.load_state_dict(torch.load("path/to/state/dict/my_model_state_dict.pt/pth"))

The trained weights of the model are now loaded into the untrained model, and you are ready to use the model as if it is pre-trained.

Saving and loading the model using TorchScript

TorchScript is a framework built into Pytorch which is used for model deployment in many different types of environments without having the model defined in the deployment environment. The effect of this is that you can save a model using tracing and load it from a file generated by tracing it.

What tracing does is follow the operations done on an input tensor that is run through your model. Note that if your model has conditionals such as if statements or external dependencies, then the tracing will not record these. Your model must only work on tensors as well.

Saving the trace of a model

In order to trace your trained model and save the trace to a file, you may run the following lines of code.

PATH = "path/to/traced/model/traced_model.pt/pth" dummy_input = torch.ones(typical_input_size, dtype=dype_of_typical_input) traced_model = torch.jit.trace(model, dummy_input)

torch.jit.save(traced_model, PATH)

The dummy_input can simply be a bare tensor that is the same size as a typical input for your model. You may also use one of the training or test inputs. The content of the dummy input does not matter, as long as it is the correct size.

Loading the trace of a model

In order to load the trace of a model, you must download the traced model .pt or .pth file into your deployment environment and note the path to it.

All you need to do to load a traced model for deployment in Pytorch is use the following line of code.

loaded_model = torch.jit.load("path/to/traced/model/traced_model.pt/pth")

Keep in mind that the traced version of your model will only work for torch tensors, and will not mimic the behavior of any conditional statements that you may have in your model.

Data Annotation

Problem

The InterbotixPincherX100 can rotate, move up and down, and extend. To get the arm to go to a specific point in space given (x, y, z) coordinates, the x and y compenents must be converted to polar coordinates.

Solution

Moving the arm to point (x, y), top-down view

Since the arm can move out/back and up/down to specific points without conversion, consider only a top-down view of the arm (x-y axis). To get the rotation and extension of the arm at a specific point, consider the triangle shown above. For rotation θ: $θ=atan(x/y)$. For extension r: $r=y/cos(θ)$. The up-down movement of the arm remains the same as the given z coordinate.

The arm can be moved to the desired point using set_ee_cartesian_trajectory. In this method, the extension and motion up/down is relative so the current extension and vertical position of the arm must be known as current_r and current_z.

Brendon Lu and Benjamin Blinder

Make sure you have the following packages imported: tkinter and rospy. The tkinter module is a basic and simple, yet effective way of implementing a usable GUI.

Because tkinter has a hard time recognizing functions created at strange times, you should next create any functions you want to use for your node. For this example, I recommend standard functions to publish very simple movement commands.

You still need to initialize the ros node, as well as any publishers and subscribers, which should be the next step in your code.

Now it is time to create a basic window for your GUI. First, write [window name] = tk.Tk() to create a window, then set the title and size of the window. Not declaring a window size will create a window that adapts automatically to the size of whatever widgets you create on the window.

The step is to populate your window with the actual GUI elements, which tkinter calls "Widgets". Here we will be making two basic buttons, but there are other common widget types such as the canvas, entry, label, and frame widgets.

And now that you have created widgets, you will notice that if you run your code, it is still blank. This is because the widgets need to be added to the window. You can use "grid", "place", or "pack" to put the widget on the screen, each of which have their own strengths and weaknesses. For this example, I will be using "pack".

And now finally, you are going to run the tkinter mainloop. Please note that you cannot run a tkinter loop and the rospy loop in the same node, as they will conflict with each other.

To run the node we have created here, you should have your robot already running either in the simulator or in real life, and then simply use rosrun to run your node. Here is the code for the example tkinter node I created, with some more notes on what different parts of the code does

Although this code does technically move the robot and with some serious work it could run a much more advanced node, I do not recommend doing this. I would recommend that you create two nodes: A GUI node and a robot node. In the GUI node, create a custom publisher such as command_pub=rospy.Publisher('command', Twist, queue_size=1) and use this to send messages for movement to the robot node. This way, the robot node can handle things like LiDAR or odometry without issues, since the tkinter update loop will not handle those kinds of messages very efficiently.

Overall, tkinter is an industry staple for creating simple GUIs in Python, being fast, easy to implement, versatile, and flexible, all with an intuitive syntax. For more information, check out the links below.

Author: Ken Kirio

This guide will show how to set up an external camera by connecting it to a computer running ROS. This guide assumes your computer is running ROS on Ubuntu natively, not via the VNC, in order to access the computer's USB port. (The lab has computers with Ubuntu preinstalled if you need one.)

1. Installation

   • Install the ros package usb_cam: sudo apt install ros-noetic-usb-cam

   • Install guvcview for the setup: sudo apt install guvcview

2. Edit the launch file, usb_cam-test.launch

   • Find the location of the file inside the usb_cam package: roscd usb_cam

   • Set video_device parameter to the port of your camera

     • Check which ports are in use: ls /dev and look for video0, video1, etc.

     • Check the output of each port: guvcview /dev/<port>

     • If you unplug the camera between uses or restart your computer, the camera may be on a different port. Check every time!

3. Run the node! roslaunch usb_cam usb_cam-test.launch

The Google Coral TPU is a USB machine learning accelerator which can plugged into a computer to increase machine learning performance on Tensorflow Lite models. The following documentation will detail how to install the relevant libraries and use the provided PyCoral library in Python to make use of the Coral TPU.

Requirements

• Linux Debian 10, or a derivative thereof (such as Ubuntu 18.04)

• A system architecture of either x86-64, Armv7 (32-bit), or Armv8 (64-bit), Raspberry Pi 3b+ or later

• One available USB port (for the best performance, use a USB 3.0 port)

• Python 3.6-3.9

• Note: For use with ROS, you will want to have ROS Noetic installed on Ubuntu 20.04.

For details on how to set up for Windows or Mac OS, see Get started with the USB Accelerator on the Coral website.

Installing Required Packages

Follow the following steps in order to get your environment configured for running models on the Coral TPU

1. Open up a terminal window and run the following commands:

   • echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list

   • curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

   • sudo apt-get update

2. Plug in your Coral TPU USB Accelerator into a USB 3.0 port on your computer. If you already have it plugged in, unplug it and then plug it back in.

3. Install one of the following Edge TPU Runtime libraries:

   To install the reduced clock-speed library, run the following command:

   • sudo apt-get install libedgetpu1-std

   Or run this command to install the maximum clock-speed library:

   • sudo apt-get install libedgetpu1-max

   Note: If you choose to install the maximum clock-speed library, an excessive heat warning message will display in your terminal. To close this window, simply use the down arrow to select OK and press enter and your installation will continue.

   In order to switch runtime libraries, just run the command corresponding to the runtime library that you wish to install. Your previous runtime installation will be deleted automatically.

4. Install the PyCoral Python library with the following command:

   • sudo apt-get install python3-pycoral

You are now ready to begin using the PyCoral TPU to run Tensorflow Lite models.

Running a pretrained TFLite model

The following section will detail how to download and execute a TFLite model that has been compiled for the Edge TPU

Downloading a model

Pretrained TFLite models that have been precompiled for the Coral TPU can be found on the models section of the Coral website.

• Once you have downloaded your model, place it into a folder within your workspace.

This FAQ section assumes understanding of creating a basic Gazebo world and how to manipulate a XML file. This tutorial relies on the assets native to the Gazebo ecosystem.

Setting up empty Gazebo world

By Nathan Cai

(If you have a prexisting Gazebo world you want to place an actor you can skip this part) Empty Gazebo worlds often lack a proper ground plane so it must be added in manually. You can directly paste this code into the world file.

<?xml version="1.0" ?>
<sdf version="1.6">
   <world name="default">
      <!-- A ground plane -->
      <include>
         <uri>model://ground_plane</uri>
      </include>
      <!-- A global light source -->
      <include>
         <uri>model://sun</uri>
      </include>

Placing an actor in the world

TL:DR Quick Setup

Here is the quick setup of everything, one can simply copy and paste this code and change the values to suit the need:

(If you do not have a Plugin for the model, please delete the Plugin section)

<actor name="actor1">
    <pose>0 0 0 0 0 0</pose>
    <skin>
        <filename>PERSON_MESH.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="NAME">
        <filename>ANIMATION_FILE_NAME.dae</filename>
        <scale>1.000000</scale>
        <interpolate_x>true</interpolate_x>
    </animation>
    <plugin name="PLUGIN NAME" filename="PLUGIN_FILE_NAME.so">
        ...
        ...
        ...
    </plugin>
    <script>
        <trajectory id="0" type="NAME">
            <waypoint>
                <time>0</time>
                <pose>0 2 0 0 0 -1.57</pose>
            </waypoint>
            <waypoint>
                <time>2</time>
                <pose>0 -2 0 0 0 -1.57</pose>
            </waypoint>
            ...
                ...
                ...
            ...
        </trajectory>
      </script>
</actor>

Defining an actor

Human or animated models in Gazebo are called actors, which contains all the information of an actor. The information can include: pose, skin, animation, or any plugins. Each actor needs a unique name. It uses the syntax:

<actor name="actor">
    ...
    ...
    ...
</actor>

Change the actor pose

The pose of the is determined using the pose parameter of an actor. The syntax is: (x_pos, y_pos, z_pos, x_rot, y_rot, z_rot)

<actor name="actor">
    <pose>0 0 0 0 0 0</pose>
    ...
    ...
</actor>

Add in Skins

The skin is the mesh of the actor or model that you want to place into the world. it is placed in the actor group and takes in the input of the filename. The syntax is:

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
</actor>

The mesh scale is also adjustable by changing the scale parameter.

Add in Animations

Though the actor can operate without animations, it is preferable for you to add one, especially if the model is to move, as it would make the enviorment more interesting and realistic.

To add an animation to the actor, all it needs is a name fore the animation, and the file that contains the animation. The syntax for this is: NOTE: THE FILE BECOMES BUGGY OR WILL NOT WORK IF THERE IS NO SKIN.

IMPORTANT: IN ORDER FOR THE ANIMATION TO WORK, THE SKELETON OF THE SKIN MUST BE COMPATABLE WITH THE ANIMATION!!!!

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="walking">
        <filename>walk.dae</filename>
        <scale>1.000000</scale>
    </animation>
</actor>

The animation can also be scaled.

Scripts

Scripts are tasks that you can assign an actor to do, in this case it is to make the actor walk around to specific points at specific times. The syntax for this is:

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="walking">
        <filename>walk.dae</filename>
        <scale>1.000000</scale>
    </animation>
    <script>
        <trajectory id="0" type="NAME">
            <waypoint>
                <time>0</time>
                <pose>0 2 0 0 0 -1.57</pose>
            </waypoint>
            <waypoint>
                <time>2</time>
                <pose>0 -2 0 0 0 -1.57</pose>
            </waypoint>
            ...
                ...
                ...
            ...
        </trajectory>
    </script>
</actor>

You can add as many waypoitns as you want so long as they are at different times. The actor will navigate directly to that point at the specified time of arrive in <time>0</time> and pose using <pose>0 0 0 0 0 0</pose>.

Plugin addons

The actor can also take on plugins such as obstacle avoidance, random navigation, and potentially teleop. The parameters for each plugin may be different, but the general syntax to give an actor a plugin is:

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="walking">
        <filename>walk.dae</filename>
        <scale>1.000000</scale>
    </animation>
    <plugin name="PLUGIN_NAME" filename="NAME_OF_PLUGIN_FILE">
        ...
    </plugin>
</actor>

With all of this you should be able to place a human or any model of actor within any Gazebo world. For reference, you can refer to the Gazebo actor tutorial for demonstration material.

References

• Make an animated model (actor)

I have spent many hours tracking down a tricky tf tool and in doing that came across some new techniques for troubleshooting. In this FAQ I introduce using rosbag with Webviz

Rosbag

This CLI from ROS monitors all topic publications and records their data in a timestamped file called a bag or rosbag. This bag can be used for many purposes. For example the recording can be "played back" which allows you to test the effects that a certain run of the robot would have had. You can select which topics to collect.

One other use is to analyze the rosbag moment to moment.

Webiz

Is an online tool (https://webviz.io). You need to understand topics and messages to use it and the UI is a little bit obscure. But it's easy enough. You supply your bag file (drag and drop) and then arrange a series of panes to visualize all of them in very fancy ways.

Evalyn Berleant, Kelly Duan

Arguments and parameters are important tags for roslaunch files that are similar, but not quite the same.

What are parameters?

Parameters are either set within a launch file or taken from the command line and passed to the launch file, and then used within scripts themselves.

Getting parameters

Parameters can be called inside their nodes by doing

# get a global parameter
rospy.get_param('/global_param_name')

# get a parameter from our parent namespace
rospy.get_param('param_name')

# get a parameter from our private namespace
rospy.get_param('~private_param_name')

Example from ROS parameter tutorial.

Adding parameters to launch files

Setting Parameters

Parameters can be set inside nodes like such (python):

rospy.set_param('some_numbers', [1., 2., 3., 4.])
rospy.set_param('truth', True)
rospy.set_param('~private_bar', 1+2)

For instance, if you wanted to generate a random number for some parameter, you could do as follows:

<param name="robot_position" command="$(find some_package)/scripts/generate_random_position.py"/>

which would generate a random position for the parameter.

Be careful that if you are setting parameters in more than one place that they are set in order correctly, or one file may overwrite the parameter’s value set by another file. (See links in resources for more detail).

What are arguments?

While parameters can pass values from a launch file into a node, arguments (that look like <arg name=”name”/> in the launch file) are passed from the terminal to the launch file, or from launch file to launch file. You can put arguments directly into the launch file like such and give it a value (or in this case a default value):

<launch>
  <arg name="x_pos" default="0.0" />
  <arg name="y_pos" default="0.0" />
  <arg name="z_pos" default="0.0" />
...

Or you can pass arguments into “included” files (launch files included in other launch files that will run):

<!-- start world and launch gazebo -->
  <include file="$(find gazebo_ros)/launch/empty_world.launch">
    <arg name="world_name" value="$(find swarmbots)/worlds/$(arg world).world"/>
    <arg name="paused" value="true"/>
    <arg name="use_sim_time" value="true"/>
    <arg name="gui" value="true"/>
    <arg name="headless" value="false"/>
    <arg name="debug" value="false"/>
  </include>

Substitution args

Substitution args, recognized by the $ and parentheses surrounding the value, are used to pass values between arguments. Setting the value of a parameter or argument as value=”$(arg argument_name)” will get the value of argument_name in the same launch file. Using $(eval some_expression) will set the value to what the python expression at some_expression evaluates to. Using $(find pkg) will get the location of a package recognized by the catkin workspace (very often used).

The if attribute can be used on the group tag, node tag, or include tag and work like an if statement that will execute what is inside the tag if true. By using eval and if together, it is possible to create loops to run files recursively. For example, running a launch file an arbitrary number of times can be done by specifying the number of times to be run in the launch file, including the launch file within itself, and decrementing the number of times to be run for each recursive include launch, stopping at some value checked by the if attribute. Here is an example of a recursive launch file called follower.launch to spawn in robots.

<launch>
  <arg name="followers" />
  <arg name="ns" />

  <!-- BEGIN robot[#] -->
  <group ns="$(arg ns)">

    <param name="tf_prefix" value="$(arg ns)" />
    <include file="$(find swarmbots)/launch/one_robot.launch">
      <arg name="robot_name" value="$(arg ns)" />
      <arg name="followers" value="$(arg followers)" />
    </include>
  </group>

  <!-- recursively start robot[#-1] -->
  <arg name="new_followers" value="$(eval arg('followers') - 1)" />
  <include file="$(find swarmbots)/launch/follower.launch" if="$(eval arg('new_followers') >= 0)">
    <arg name="followers" value="$(arg new_followers)" />
    <arg name="ns" value="robot$(arg new_followers)" />
  </include>
</launch>

followers here will impact the number of times the launch file is recursively called. $(eval arg('followers') - 1) will decrement the value of followers inside each recursive launch, and the if attribute

 if="$(eval arg('new_followers') >= 0)"

checks that once the new number is below 0, it will not call the launch file again.

Differences between arguments and parameters (important!)

Both arguments and parameters can make use of substitution args. However, arguments cannot be changed by nodes like parameters are with rospy.set_param(). Because of the limits of substitution, you cannot take the value of a parameter and bring it to an argument. If you want to use the same value between two params that require generating a specific value with rospy.set_param() then you should create another node that sets both parameters at once.

For example, this script

#!/usr/bin/env python
import random, rospy, roslib

rospy.init_node("generate_random_x")
pos_range = float(rospy.get_param('pos_range', 3))

x_pos = random.uniform(-pos_range / 2, pos_range / 2)
rospy.set_param('map_merge/init_pose_x',x_pos)

print(x_pos)

is called within a parameter using the command attribute.

<param name="x_pos" command="$(find swarmbots)/src/generate_random_x.py" />

The command attribute sets the value of the parameter to whatever is printed by stdout in the script. In this case, the script generates a random number for x_pos. In the same file, rospy.setparam is called to set another parameter to the same value of x_pos. In that way, both parameters can be set at once.

Too many parameters? Use rosparam!

If you have too many parameters and/or groups of parameters, not only is it inefficient to write them into a launch file, but is also prone to many more errors. That is when a rosparam file comes in handy--a rosparam file is a YAML file that stores parameters in an easier-to-read format. A good example of the utility of rosparam is the parameters for move_base, which uses the command

<rosparam file="$(find turtlebot3_navigation)/param/local_costmap_params.yaml" command="load" />

which loads the parameters from the yaml file here:

local_costmap:
  global_frame: odom
  robot_base_frame: base_footprint

  update_frequency: 10.0
  publish_frequency: 10.0
  transform_tolerance: 0.5  

  static_map: false  
  rolling_window: true
  width: 3
  height: 3
  resolution: 0.05

References

• Ros Parameters Tutorial

• Substitution args

• Order of launch files

Junhao Wang

Read CompressedImage type

When using Turtlebot in the lab, it only publishs the CompressedImage from '/raspicam_node/image/compressed'. Here's how to read CompressedImage and Raw if you need.

from cv_bridge import CvBridge
def __init__(self):
   self.bridge = CvBridge()
def image_callback(self, msg):
        # get raw image
        # image = self.bridge.imgmsg_to_cv2(msg)
        
        # get compressed image
        np_arr = np.frombuffer(msg.data, np.uint8)
        img_np = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
        image = cv2.cvtColor(img_np, cv2.COLOR_BGR2HSV)

Limit frame rate

When the computation resource and the bandwidtn of robot are restricted, you need limit the frame rate.

def __init__(self):
   self.counter = 1
def image_callback(self, msg):
        # set frame rate 1/3
        if self.counter % 3 != 0:
            self.counter += 1
            return
        else:
            self.counter = 1

Republish the image

Again, when you have multiple nodes need image from robot, you can republish it to save bandwidth.

#!/usr/bin/env python

import rospy
from sensor_msgs.msg import CompressedImage
from cv_bridge import CvBridge

def image_callback(msg):
    # Convert the compressed image message to a cv2 image
    bridge = CvBridge()
    cv2_image = bridge.compressed_imgmsg_to_cv2(msg, desired_encoding='passthrough')

    # Process the image (e.g., resize, blur, etc.)
    processed_image = process_image(cv2_image)

    # Convert the processed cv2 image back to a compressed image message
    compressed_image_msg = bridge.cv2_to_compressed_imgmsg(processed_image)

    # Publish the processed image on the new topic
    processed_image_pub.publish(compressed_image_msg)

def process_image(cv2_image):
    # Perform image processing (e.g., resize, blur, etc.) and return the processed image
    # Example: resized_image = cv2.resize(cv2_image, (new_width, new_height))
    return cv2_image

if __name__ == '__main__':
    rospy.init_node('image_processing_node')
    rospy.Subscriber('/robot/camera_topic', CompressedImage, image_callback)
    processed_image_pub = rospy.Publisher('/processed_image_topic', CompressedImage, queue_size=1)

    rospy.spin()

To use a sigmoid function instead of a PID controller, you need to replace the PID control algorithm with a sigmoid-based control algorithm. Here is a step-by-step guide to do this:

1. Understand the sigmoid function: A sigmoid function is an S-shaped curve, mathematically defined as: f(x) = 1 / (1 + exp(-k * x))

   where x is the input, k is the steepness factor, and exp() is the exponential function. The sigmoid function maps any input value to a range between 0 and 1.

2. Determine the error: Just like in a PID controller, you need to calculate the error between the desired setpoint and the current value (process variable). The error can be calculated as: error = setpoint - process_variable

3. Apply the sigmoid function: Use the sigmoid function to map the error to a value between 0 and 1. You can adjust the steepness factor (k) to control the responsiveness of the system: sigmoid_output = 1 / (1 + exp(-k * error))

4. Scale the output: Since the sigmoid function maps the error to a range between 0 and 1, you need to scale the output to match the actual range of your control signal (e.g., motor speed or actuator position). You can do this by multiplying the sigmoid_output by the maximum control signal value: control_signal = sigmoid_output * max_control_signal

5. Apply the control signal: Send the control_signal to your system (e.g., motor or actuator) to adjust its behavior based on the error.

Note that a sigmoid function-based controller may not provide the same level of performance as a well-tuned PID controller, especially in terms of overshoot and settling time. However, it can be useful in certain applications where a smooth, non-linear control response is desired.

Ben Soli

Before you use this tutorial, consult with the Campus Rover Packages which outline setting up ngrok with flask and getting to the Alexa Developer Console.

The Flask Alexa Skills Kit module allows you to define advanced voice functionality with your robotic application.

In your VNC terminal

pip3 install flask-ask

Then in your rospy node file

from flask_ask import Ask, statement, question, elicit_slot, confirm_slot

declare your Flask application as

app = Flask(__name__)

and connect it to the Flask ASK

ask = Ask(app, '[endpoint]')

So, if your ngrok subdomain is campusrover and your Alexa endpoint is /commands

your code would be

ask = Ask(app, '/commands')

On the Alexa Developer Consoles define your intents and determine which slots your intents will use. Think of intents as functions, and slots as parameters you need for that function. If a slot can have multiple items in a slot mark that in the developer console. You do not need to provide the Alexa responses on the Developer Console because you will be writing them with Flask-ASK. The advantage of doing this is you can write responses to the user that take into account the robot's state and publish information to other nodes as you receive input with Alexa.

Let's assume we have an intent called 'bring_items' that can contain multiple items to fetch. Assume that the robot needs at least one item to fulfill this intent and that the robot has a weight capacity for how much it can carry. Let's also assume we have some kind of lookup table for these items which tells us their weight. With Flask-ASK we can quickly build this.

You are required to have a response for launching the intent marked as

@ask.launch
def start_skill():
  return question('hello, what you need?)

This intent is not called every single time you use the skill, but its a good way to tell the user what the bot can do or tell the user what information the bot needs from them. There are a few different response types,

statement(str: response) which returns a voice response and closes the session. question(str: response) which you can use to ask the user a question and keep the session open. elicit_slot(str: slot_name, str: response) which you can use to prompt the user for a specific type of information needed to fulfill the intent. confirm_slot(str: slot_name, str: response) which you can use to confirm with the user that Alexa heard the slot correctly.\

It is important to note, that to use elicit_slot and confirm_slot you must have a Dialog Model enabled on the Alexa Developer Console. The easiest way I have to found to enable this is to create an intent on the console that requires confirmation. To avoid this intent being activated, set its activation phrase to gibberish like 'asdlkfjaslfh'. You can check a switch marking that this intent requires confirmation.

Now let's build out an intent in our flask application.

Start with a decorator for your skill response marking which intent you are programming a response for.

@ask.intent('bring_items')
def bring_items():

First, let's assure that the user has provided the robot with some items. When you talk to Alexa, it essentially just publishes a JSON dictionary to your endpoints which your flask app can read. To get a list of the slots for this intent use the line:

  slots = request.get_json()['request']['intent']['slots']

Let's call our items slots 'items'. To check that the user has provided at least one item, write.

if 'value' not in slots['items'].keys() 
  return elicit_slot('items', 'what do you want me to bring?)

This will check that there is an item, and if there is not, prompt the user for some items. The string python 'value' is a key in the dictionary if and only if the user has provided the information for this slot, so this is the best way to check.

Let's assume that our robot has a carrying capacity of 15 pounds and has been asked to carry some items that weigh more than 15 pounds. Once we've checked that the items are too heavy for the robot, you can elicit the slot again with a different response like

elicit_slot('items','those are too heavy, give me something else to bring')

and the user can update the order. Returning elicit_slots keeps the Alexa session open with that intent, so even though each call is a return statement, you are essentially making a single function call and updating a single JSON dictionary.

Once the robot is given a list of items that it can carry, you can use a rospy publisher to send a message to another node to execute whatever robotic behavior you've implemented.

You should also include

@ask.intent('AMAZON.FallbackIntent')

This is an intent you can use for any phrase that you have not assigned to an intent that your robot will use. If you do not implement a response for this, you will very likely get error messages from Alexa.

Skills come prebuilt with some intents such as these. If a user activates one of these intents, and you don't define a response, you will get a skills response error. You should thoroughly test possible utterances a user might use to avoid errors. Skills response errors do not crash the application, but it makes for bad user-experience.

Another default intent you will likely need is

@ask.intent('AMAZON.CancelIntent')

This handles negative responses from the user. An example of this is:

User: Alexa launch campus rover Alexa: hello, what do you need User: nothing Alexa: AMAZON.CancelIntent response\

I hope this tutorial has made clear the advantages of using the flask-ASK for your Alexa integration. It is a great way to rapidly develop different voice responses for your robot and quickly integrate those with robotic actions while avoiding the hassle of constantly rebuilding your Alexa skill in the Developer Console.

• Current best resource on web: The fastest way to clone an SD card on macOS

Identify your sd card

You’ll need to find out which disk your SD card represents. You can run diskutil list and should see an output like below:

$ diskutil list

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.0 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD — Data     396.0 GB   disk1s1
   2:                APFS Volume Preboot                 81.9 MB    disk1s2
   3:                APFS Volume Recovery                528.5 MB   disk1s3
   4:                APFS Volume VM                      4.3 GB     disk1s4
   5:                APFS Volume Macintosh HD            11.0 GB    disk1s5

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *31.9 GB    disk4
   1:             Windows_FAT_32 boot                    268.4 MB   disk4s1
   2:                      Linux                         31.6 GB    disk4s2

From that output we can see that our SD card must be /dev/disk4 as our card is 32GB in size and has a fat32 and linux partition (standard for most raspberry pi images). You should add an r in front of disk4 so it looks like this /dev/rdisk4. The r means when we’re copying, it will use the “raw” disk. For an operation like this, it is much more efficient.

Copy the SD card as a disk image (dmg)

Now you should run the following command, replacing 4 with whatever number you identified as your sd card:

sudo gdd if=/dev/rdisk4 of=sd_backup.dmg status=progress bs=16M

Tip: you can experiment with different numbers for the block size by replacing bs=16M with larger or smaller numbers to see if it makes a difference to the speed. I’ve found 16M the best for my hardware.

You should see some progress feedback telling you the transfer speed. If you’d like to experiment with different block sizes, just type ctrl + c to cancel the command, then you can run it again.

Once the command has finished running, you’ll end up with a file in your home directory called sd_backup.dmg. If you’d like to backup multiple SD cards (or keep multiple backups!) simply replace sd_backup.dmg with a different file name. This will contain a complete disk image of your SD card. If you’d like to restore it, or clone it to another SD card, read on.

Copy the disk image (dmg) to your SD card

You’ll first need to unmount your SD card. Do not click the eject button in finder, but run this command, replacing 4 with whatever number you identified as your sd card

sudo diskutil unmountDisk /dev/disk4

Then to copy the image, run the following command:

sudo gdd of=/dev/rdisk4 if=sd_backup.dmg status=progress bs=16M

Tip: you can experiment with different numbers for the block size by replacing bs=16M with larger or smaller numbers to see if it makes a difference to the speed. I’ve found 16M the best for my hardware.

You should see some progress feedback telling you the transfer speed. If you’d like to experiment with different block sizes, just type ctrl + c to cancel the command, then you can run it again.

Once the command has finished running, your SD card should be an exact copy of the disk image you specified.

Step 1: switch the .bashrc to be running in sim mode. Step 1.1: Go into .bashrc file and uncomment the simulation mode as shown below:

Setting for simulation mode

$(bru mode sim)

$(bru name roba -m $(myvpnip))

Step 1.2: Comment out real mode/robot ip addresses For Example:

Settings for donatello

$(bru mode real)

$(bru name donatello -m 100.106.194.39)

Step 2: run roscore on vnc. To do this type "roscore" into the terminal

Step 3: Now in the terminal do these steps Step 3.1: get vpn ip address: In the terminal type "myvpnipaddress" Step 3.2: Type "$(bru mode real)" Step 3.3: Type "$(bru name robotname -m "vpn ip address from step 4")" Step 3.4: type"multibringup" in each robot terminal

Step 4: Repeat step 3 in a second tab for the other robot(s)

How to create a new message type?

After created a ROS package, our package is constructed by a src folder, a CMakeLists.txt file, and a package.xml file. We need to create a msg folder to hold all of our new msg file. Then in your msg folder, create a new file <new_message>.msg, which contains fields you needed for this message type. For each fields in your msg file, define the name of the field and the type of the field (usually use std_msgs/ or geometry_msgs/). <br > For example, if you want your message to have a list of string and an integer, you msg file should look like:

std_msgs/String[] list
std_msgs/Int16 int

How to let the new message type recognized by ROS?

There are some modifications you need to make to CMakeLists.txt and package.xml in order to let the new message type recognized by ROS. <br > For CMakeLists.txt:

1. Make sure message_generation is in find_package().

2. Uncomment add_message_files() and add your .msg file name to add_message_files().

3. Uncomment generate_messages()

4. Modify catkin_package() to

catkin_package(
  CATKIN_DEPENDS message_runtime
)

1. Uncomment include in include_directories()

For package.xml:

1. Uncomment <build_depend>message_generation</build_depend> on line 40

2. Uncomment <exec_depend>message_runtime</exec_depend> on line 46

How to use the newly created message type?

1. How to import the message?

from package_name.msg import message_name as message_name

If your message type contains a list of buildin message type, also make sure to import that buildin message type:

from std_msgs.msg import String

1. How to use the message? <br > The publisher and subscriber's syntax are the same. However, we want to create a new topic name and make sure the new topic message type is specified. For example in my project I have a message type called see_intruder:

self.detect_intruder_pub = rospy.Publisher('/see_intruder', see_intruder, queue_size=1)
self.detect_intruder_sub=rospy.Subscriber('/see_intruder', see_intruder,self.see_intruder_callback)
  

Since our new message depends on some build in message type, when we try to access the feild of our msg, we need to do msg.<field_name>.data given that the build in message type has a field named data. So in the first example I had, if we want to access the string stored in index int of list, I need to do

msg.list[msg.int.data].data

How to check if the new message type is recognized by ROS?

Do a cm in your vnc. if the message is successfully recognized by ROS, you will see the msg file being successfully generated.

Having an error "No module named <>.msg"?

In your vnc terminal, type the command

source ~/catkin_ws/devel/setup.bash

This should solve the error.

PID is a common and powerful programming tool that can be used to maintain stability and accuracy in many tasks that give feedback. This guide will walk through how to de-mystify PID and teach you to become a confident PID user.

PID stands for _P_roportional, _I_ntegral and _D_erivative. All three mathematical techniques are used in a PID controller.

First, let's think of a PID application. There is a wheel on a motor with a quadrature shaft encoder that you desire to spin at a certain RPM. In this example, imagine you have already written the code to get the speed of the motor by reading the encoder data.

The goal is for the wheel to spin at 60 RPM. so 60 is our Setpoint Variable. the actual speed that the motor encoders report is the Process Variable. By subtracting the Process Variable PV from the Setpoint Variable SP, we get the error

double error = target_speed - get_wheel_speed(); // assume this function has been written and returns motor speed in RPM

We will use this error three ways, and combine them using weights (called Gains) to acheive PID control. The three components answer three questions:

1. How big is my error?

2. How much error have I accumulated?

3. Is my error getting bigger or smaller?

Proportional

Proportional control could not be simpler: multiply the error by some constant Kp

double p = error * Ki;

Integral

If you know about Integrals then you know that they represent the area under a curve, which is a sum of all the points on said curve. Therefore, we can calculate the Integral component by summating the total error over time, like this:

double i;
...
i += Ki * error * change_in_time();

Of course, Ki is the Integral Gain constant, similar to Kp. Error is multiplied by the change in time because remember, the integral is the area under the curve. So if we plot error on the Y axis, time would likely be plotted on the x axis, thus area is error * time.

Derivative

Derivatives are the rate at which things change, so naturally a simple way to get the Derivative component is to compare the current error to the pervious error, and then account for time, like this:

double d = Kd * (error - pervious_error) / change_in_time();
previous_error = error;

Again, Kd is the derivative gain constant, and we divide by the change in time because if the change is 4 units in half a second, then the rate of change is 8 units/second (4 / 0.5)

What to do with P, I and D

Add them up. This is an adjusted representation of your control system's error. Therefore, you can apply it as a change to your previous command to try to get closer to the Setpoint Variable

double pid = p + i + d;
command = previous_command + pid;
previous_command = command;

Then your command can be sent to your actuator. Please note that some additional arithmatic may be required, this is a bare-bones simple example and does by no means serve as a copy+paste solution to all PID applications.

Tuning

A PID system must be tuned with the proper values of Kp, Ki and Kd in order to acheive smooth control. The main goals of tuning are to minimize error and over shooting the Setpoint Goal. There are plenty of guides and theories as to how to tune, so here is the Google Search to get you started

Question

How do I set up the PX-100 arm to work with ROS2?

Answer

Introduction

The PX-100 arm currently supports ROS2 Galactic on Ubuntu Linux 20.04, or ROS2 Humble on Ubuntu Linux 22.04. This is unlikely to change in the future, as Trossen Robotics seems to have stopped working further on the PX-100.

This guide assumes you will be using ROS2 Humble on Ubuntu Linux 22.04. If you need to, it wouldn't be too hard to adapt the instructions here for a setup on ROS2 Galactic.

Install Ubuntu

Follow these official instructions, or others you might find on the web, to install Ubuntu Linux 22.04 on your computer. You might meet with a few hiccups along the way, e.g., about Ubuntu being incompatible with RST, etc. When you do, don't panic, and search for the simplest solution to the problem.

Check for Hardware Compatibility

Plug in the arm to a usb port of your computer. After waiting for a bit, run the lsusb command to see the USB devices connected to your computer. If you see something like:

Bus 001 Device 003: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC

as a line in your output, your computer is probably compatible with the arm.

Install ROS2 Humble, colcon, and rosdep

First, install ROS2 Humble. The official guide is good. The only recommendation I'd make is to add source /opt/ros/humble/setup.bash to your .bashrc file (if you're using a BASH shell. To see whether you are, run echo $SHELL. If the output is /bin/bash, you're using BASH).

Second, install colcon, ROS2's build tool.

sudo apt install python3-colcon-common-extensions

Third, install rosdep. Run:

sudo apt install python3-rosdep

Then execute

sudo rosdep init
rosdep update

Install Interbotix's Software for ROS2

Finally, it's time to install software for the PX-100 arm. Execute:

sudo apt install curl
curl 'https://raw.githubusercontent.com/Interbotix/interbotix_ros_manipulators/main/interbotix_ros_xsarms/install/amd64/xsarm_amd64_install.sh' > xsarm_amd64_install.sh
chmod +x xsarm_amd64_install.sh
./xsarm_amd64_install.sh -d humble

If you're using galactic, replace 'humble' with 'galactic' in the last line.

After installation, follow these Installation Checks from Trossen Robotics, to confirm that your installation was successful.

Checking that Things Work as Expected

Execute the following line:

ros2 launch interbotix_xsarm_control xsarm_control.launch.py robot_model:=px100

This should launch an RVIZ window displaying a graphical representation of your arm. Don't panic if you don't get the arm on the first try, and there are white blocks where components of the robot should be. Try closing rviz, and running the command again.

If this still doesn't work, check to see if the arm is in the sleeping position. If it isn't, unplug the arm, and gently move it into the sleeping position.

Even if the arm is in a sleeping position, unplug, and then replug the arm.

Run the command above again. It should have worked this time. If it still doesn't, there was probably something wrong with the installation process. Or the hardware of the arm is incompatible with your computer for mysterious reasons.

If rviz does work, execute the following command:

ros2 service call /px100/torque_enable interbotix_xs_msgs/srv/TorqueEnable "{cmd_type: 'group', name: 'all', enable: false}"

This command disables the mechanisms on the robot that keep its joints fixed in their positions. This means that you should be able to move the robot's joints manually. Try this, GENTLY, and see the graphical representation of the robot on RVIZ move in tandem.

After you're satisfied with your experiment, place the robot back into its sleeping position, and quit the two ros2 processes you started.

Next Steps

The documentation for Interbotix's ROS2 API is very lacking. If you're curious about something, try searching our lab notes first to see if you can find what you're looking for.

Otherwise, dig into the source code, installed under the interbotix_ws directory. The directory's path is likely ~/interbotix_ws. Don't be afraid to do this! It's what you'll have to do anyway if you join a company or work on a substantial open source project. Don't be afraid to modify the source code either, if you feel it's not working as it should! The developers are only human, and may have made mistakes.

arm setup

Links

• Arm Details

• Quickstart Guide

• Hardware setup

  • only needed to install the grippers

    • the screws would not go through initially - had to go from the other side with the screw first to ensure that the holes were completely open and the screw could be secured and then installed the grippers as the instructions said

  • plug in power supply first and then plug usb into computer and then plug microUSB into arm

• ROS Installation Guide

• Troubleshooting

• DYNAMIXEL Software

  • after installing software and plugging arm in, scan for the arm in the dynamixel software to check that everything is working properly:

    • in options, select baudrates 57600 and 1000000

    • if any link is in 57600, then change to 1000000

  • make sure to disconnect before running ros

  • NOTE: you may not actually need to do this, but it may be good to do the first time you try to connect the arm to your computer to make sure everything is running correctly.

• Arm Control

  • contains command line configs for launch file

• Python-ROS Interface

  • contains details on methods that control the arm using their methods - can find basic overview at the bottom of this file.

Installation:

On Intel/AMD based processor:

sudo apt install curl
curl 'https://raw.githubusercontent.com/Interbotix/interbotix_ros_manipulators/main/interbotix_ros_xsarms/install/amd64/xsarm_amd64_install.sh' > xsarm_amd64_install.sh
chmod +x xsarm_amd64_install.sh
./xsarm_amd64_install.sh -d noetic

Basic Commands:

• move the arm manually:

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100

• disable torque:

  • rosservice call /px100/torque_enable "{cmd_type: 'group', name: 'all', enable: false}"

• re-enable torque to hold a pose:

  • rosservice call /px100/torque_enable "{cmd_type: 'group', name: 'all', enable: true}"

• run with moveit:

  • roslaunch interbotix_xsarm_moveit xsarm_moveit.launch robot_model:=px100 use_actual:=true dof:=4

  • roslaunch interbotix_xsarm_moveit xsarm_moveit.launch robot_model:=px100 use_gazebo:=true dof:=4

• run using ROS-PYTHON API:

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100 use_sim:=true

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100 use_actual:=true

• play with joints:

  • roslaunch interbotix_xsarm_descriptions xsarm_description.launch robot_model:=px100 use_joint_pub_gui:=true

• publish static transforms between two frames:

  • rosrun tf static_transform_publisher x y z yaw pitch roll frame_id child_frame_id period(milliseconds)

Interbotix Python-ROS Interface

• arm.set_ee_pose_components()

  • sets an absolute position relative to the base of the frame

    • ee_gripper_link frame with respect to the base_link frame

• arm.set_single_joint_position()

  • move the specified joint

  • usually used for the waist to turn the robot

• arm.set_ee_cartesian_trajectory()

  • move the end effector the specified value in each direction relative to the current position

  • for a 4dof arm, the y and yaw values cannot be set through this

• arm.go_to_sleep_position()

  • return the arm to the sleep position

• arm.go_to_home_position()

  • return the arm to the home position

• gripper.open()

  • open the gripper

• gripper.close()

  • close the gripper

• arm.set_trajectory_time()

  • moving_time - duration in seconds it should take for all joints in the arm to complete one move.

  • accel_time - duration in seconds it should take for all joints in the arm to accelerate/decelerate to/from max speed.

Overview of ROSBridge and ROSLIBJS

This FAQ is deigned to provide an overview of ROSBridge and ROSLIBJS, two important tools for integrating web applications, such as the command control dashboard, with the Robot Operating System (ROS).

Table of contents

• What are ROSBridge and ROSLIBJS?

• How do ROSBridge and ROSLIBJS work together?

• How do I install and use ROSBridge and ROSLIBJS?

• Are there any limitations or challenges when using ROSBridge and ROSLIBJS?

What are ROSBridge and ROSLIBJS?

ROSBridge is a package for the Robot Operating System (ROS) that provides a JSON-based interface for interacting with ROS through WebSocket protocol (usually through TCP). It allows external applications to communicate with ROS over the web without using the native ROS communication protocol, making it easier to create web-based interfaces for ROS-based robots.

ROSLIBJS is a JavaScript library that enables web applications to communicate with ROSBridge, providing a simple API for interacting with ROS. It allows developers to write web applications that can send and receive messages, subscribe to topics, and call ROS services over websockets.

How do ROSBridge and ROSLIBJS work together?

ROSBridge acts as a bridge between the web application and the ROS system. It listens for incoming WebSocket connections and translates JSON messages to ROS messages, and the other way around. ROSLIBJS, on the other hand, provides an API for web applications to interact with ROSBridge, making it easy to send and receive ROS messages, subscribe to topics, and call services.

In a typical application utilizing ROSBridge and ROSLIBJS, it would have the following flow:

1. Web client uses ROSLIBJS to establish a WebSocket connection to the ROSBridge server, through specifying a specific IP address and port number.

2. The web client sends JSON messages to ROSBridge, which converts them to ROS messages and forwards them to the appropriate ROS nodes.

3. If the node has a reply, the ROS nodes send messages back to ROSBridge, which converts them to JSON and sends them over the WebSocket connection to the web client.

4. ROSLIBJS API processes the incoming JSON messages so it can be displayed/utilized to the web client

How do I install and use ROSBridge and ROSLIBJS?

ROSBridge Installation

The following assumes that you have ROS Noetic installed on your system.

To install ROSBridge you need to install the rosbridge-server package

sudo apt install ros-noetic-rosbridge-server

ROSLIBJS Installation

You can either use the hosted version of ROSLIBJS or download it for use in your project. To use the hosted version, include the following script tag in your HTML file:

<script src="https://static.robotwebtools.org/roslibjs/current/roslib.min.js"></script>

To download ROSLIBJS, visit the Github repository builds and download and save the files. To use the local version, include the following script tag in your HTML file:

<script src="PATH-TO-DOWNLOADED-SCRIPT"></script>

Simple Example

The following example is for React Application

First, download the roslib.min.js file from the ROSLIBJS GitHub repository and place it in your project's public directory.

Next, create a new React component called RosConnect:

RosConnect.jsx

import React, { Component } from 'react';
import { Alert } from 'react-bootstrap';

class ROSConnect extends Component {

    constructor() {
        super()
        this.state = { connected: false, ros: null } 
        
    }
    // run the function as soon as the page renders
    componentDidMount() {
        this.init_connection()
    }

    // a function to connect to the robot using ROSLIBJS
    init_connection() {
        this.state.ros = new window.ROSLIB.Ros()
        this.state.ros.on("connection", () => {
            this.setState({connected: true})
        })

        this.state.ros.on("close", () => {
            this.setState({connected: false})
            // try to reconnect to rosbridge every 3 seconds
            setTimeout(() => {
                try{
                    // ip address of the rosbridge server and port
                    this.state.ros.connect('ws://127.0.0.1:9090')
                }catch (error) {
                    console.log("connection error:", error);
                }
            // if the robot disconnects try to reconnect every 3 seconds (1000 ms = 1 second)
            }, 3000); 
        })

        try{
            // connect to rosbridge using websocket 
            this.state.ros.connect('ws://127.0.0.1:9090')
        }catch (error) {
            console.log("connection error:", error);
        }

    }

    render() { 
        return (
            // a Alert component from react-bootstrap showing if the robot is connected or not
            <div>
                <Alert className='text-center m-3' variant={this.state.connected ? "success" : "danger"}>
                    {this.state.connected ? "Robot Connected" : "Robot Disconnected"}
                </Alert>
            </div>
        );
    }
}
 
export default ROSConnect;

This component can be rendered to any page, for example it can be used in the App.js component which already comes with creat-react-app

Challenges and Limitations of ROSBridge and ROSLIBJS

Even though ROSBridge and ROSLIBJS is has a lot of use cases from being able to view camera feed from a robot to getting its GPS data display on a dashboard, it does have some prominent limitations.

While working on the campus command control project, one of the issues that was encountered was lag. ROSLIBJS uses web socket, which is built on top of Transmission Control Protocol (TCP). While TCP is more reliable, it transfers data more slowly, which led to lag in robot controls and video feed. It is worth mentioning that ROSBridge does support User Datagram Protocol (UDP), which comes at a cost of reliability for speed, but ROSLIBJS current implementation does not support UDP.

Author: Belle Scott

In order to spawn multiple robots in a gazebo launch file (in this example, there are two robots, seeker and hider), you must define their x,y,z positions, as shown below. Also shown is setting the yaw, which is used to spin the robot. For example, if one robot was desired to face forward and the other backward, ones yaw would be 0 while the others would be 3.14.

    <arg name="model" default="burger" doc="model type [burger, waffle, waffle_pi]"/>
    <arg name="seeker_pos_x" default="0"/>
    <arg name="seeker_pos_y" default=" -0.5"/>
    <arg name="seeker_pos_z" default=" 0.0"/>
    <arg name="seeker_yaw" default="0"/>
    <arg name="hider_pos_x" default="0"/>
    <arg name="hider_pos_y" default="0.5"/>
    <arg name="hider_pos_z" default=" 0.0"/>
    <arg name="hider_yaw" default="0" />

After setting their initial positions, you must use grouping for each robot. Here, you put the logic so the launch file knows which python files correspond to which robot. It is also very important to change the all the parameters to match your robots name (ex: $(arg seeker_pos_y)). Without these details, the launch file will not spawn your robots correctly.

    <group ns="seeker">
        <param name="robot_description" command="$(find xacro)/xacro.py $(find turtlebot3_description)/urdf/turtlebot3_$(arg model).urdf.xacro" />
        <node pkg="robot_state_publisher" type="robot_state_publisher" name="robot_state_publisher" output="screen">
            <param name="publish_frequency" type="double" value="50.0" />
            <param name="tf_prefix" value="seeker" />
        </node>
        <node name="spawn_urdf" pkg="gazebo_ros" type="spawn_model" args="-urdf -model seeker -x $(arg seeker_pos_x) -y $(arg seeker_pos_y) -z $(arg seeker_pos_z)          -Y $(arg seeker_yaw) -param robot_description" />
    </group>
    <group ns="hider">
        <param name="robot_description" command="$(find xacro)/xacro.py $(find turtlebot3_description)/urdf/turtlebot3_$(arg model).urdf.xacro" />
        <node pkg="robot_state_publisher" type="robot_state_publisher" name="robot_state_publisher" output="screen">
            <param name="publish_frequency" type="double" value="50.0" />
            <param name="tf_prefix" value="hider" />
        </node>
        <node name="spawn_urdf" pkg="gazebo_ros" type="spawn_model" args="-urdf -model hider -x $(arg hider_pos_x) -y $(arg hider_pos_y) -z $(arg hider_pos_z) -Y               $(arg hider_yaw) -param robot_description" />
    </group>

References

• This is from TheConstruct: Launch Multiple Robots in Gazebo

• And this is from our own Belle Scott: Hide and Seek

Reinforcement Learning and its Applications

Authors: Rachel Lese, Tiff Lyman

A lot of people are intimidated by the idea of machine learning, which is fair. However, machine learning doesn't have to be some complicated multi-layered network. In fact, you can implement a machine learning algorithm with just a dictionary in ROS. Specifically, you can run a reinforcement learning algorithm that replaces PID to steer a robot and have it follow a line.

What is Reinforcement Learning?

Reinforcement learning is exactly what it sounds like, learning by reinforcing behavior. Desired outcomes are rewarded in a way that makes them more likely to occur down the road (no pun intended). There are a few key components to a reinforcement learning algorithm and they are as follows:

• A set of possible behaviors in response to the reinforcement

• A quantifiable observation/state that can be evaluated repeatedly

• A reward function that scores behaviors based on the evaluation

With this, the algorithm takes the highest scoring behavior and applies it.

Application: Line Follower

Lets look at what those required traits would be in a Line Follower.

Behaviors

In this case would be our possible linear and angular movements. At the start we want everything to be equally likely, so we get the following dictionaries.

angular_states_prob =  { -.5: .5, -.45: .5, -.4: .5, -.35: .5, -.3: .5, -.25: .5, -.2: .5, -.15: .5, -.1: .5, -.05: .5, 0 : .5, .05 : .5, .1 : .5, .15 : .5, .2: .5, .25: .5, .3: .5, .35: .5, .4: .5, .45: .5, .5: .5 }
linear_states_prob = { 0.1 : .5, 0.25 : .5, 0.4 : .5 }