Ensure that the camera is enabled and there is enough GPU memory:

1. Add the following lines to the /boot/firmware/config.txt file:

And then reboot. Note that the config.txt file will say that you should not modify it and instead make the changes in a file called userconfig.txt. However I found that the userconfig.txt file is not invoked. So I added the lines above directly to config.txt.

1. Check the amount of gpu_memory

It should show 256 or whatever number you put there.

1. plug in battery and turn on robot with the power switch, give it a moment and wait for the lidar to start spinning.

2. run tailscale status | grep <name> to find the robot’s IP address. Replace with the name of the robot you are trying to connect to.

1. go to .bashrc in my_ros_data folder and get it to look like this with your robot’s name and IP address instead:

1. Open a new terminal and you should see:

1. You can then ssh into the robot, ssh ubuntu@100.117.252.97 (enter your robot’s IP) and enter the password that is in the lab.

2. Once onboard the robot, enter the command bringup which starts roscore and the Turtlebot’s basic functionalities. For the Platform robots, run this: roslaunch platform full_bringup.launch

3. After that, open a new terminal (you’ll be in real mode again) and run your program!

4. To go back to simulation mode, go back to .bashrc and uncomment the settings for simulation mode and comment out the settings for a physical robot. Or type the command sim in the terminal. You will need to do this in every terminal that you open then. To switch to real mode, type command real.

When you're done, run sudo shutdown now onboard the robot (where is ran bringup) and then turn off the robot with the power switch.

by Helen Lin edited by Jeremy Huey

Introduction

This guide shows you how to create launch files for your code so that you can launch multiple nodes at once rather than running each node individually.

Step 1

Open a new terminal window and navigate to a package you want to create a launch file in. Create a folder called 'launch'.

mkdir launch

Step 2

Navigate to the launch directory and create a new launch file ending in .launch. Replace 'name' with the name of your launch file.

touch name.launch

Step 3

Add the following code and fill in the parameters within the double quotes.

Here is an example of what the node will look like filled in, using code from the Mini Scouter project:

The pkg name can be found in the package.xml. Eg.

Step 4

Make sure you have run the following command on all of the files used in the launch file so they can all be found by ROS launch. Replace 'name' with the name of the python file.

chmod +x name.py

Change the permissions of the launch file as well by going to the launch directory and running the following command. Replace 'name' with the name of the launch file.

chmmod +x name.launch

Step 5

Open a new terminal window and run the following command. Replace 'package_name' with the name of the package and 'name' with the name of the launch file.

roslaunch package_name name.launch

For example, to run the Mini Scouter launch file:

roslaunch mini_scouter teleop.launch

All of the nodes you specified in the launch file should now be running.

Optional, launch process in new window

To have a node launch and open in a new window, such as to run things like key_publisher.py, you can modify the line to include this: <node pkg="object_sorter" type="key_publisher.py" name="key" output="screen" launch-prefix="xterm -e"/> You must then run in terminal:

More info on launch files

To continue to get more information on launch files, go to here: labnotebook/faq/launch-files.md

Author: Michael Jiang

How do I download files from my online VSCode to my local computer?

While the online VSCode does not offer a way to download entire folders and their contents with one click, there is a non-obvious functionality that allows you to download individual files directly from VSCode.

Right click the desired file and select 'Download' from the menu. The file will download to your default download location on your computer.

You will be able to now access the files from your local system and submit them to Gradescope. To submit packages like this, you can set up a Git repository and push the files there, or you can individually download each file in the package using this method, arrange them properly, and submit it on Gradescope.

Author - Eyal Cohen

Given two coordinates in a 2-dimensional plane and your robots' current direction, the smart rotation algorithm will calculate your target angle and return whether your robot should to turn left, turn right, or go straight ahead to reach its navigation goal.

How It Works:

Inputs: posex1 is a float that represents the x-axis coordinate of your robot, posey1 is a float that represents your robot's y-axis coordinate. posex2 and posey2 are floats that represent the x and y of your robot's goal. Lastly, theta represents your robot's current pose angle.

from math import atan2
import math


def smartRotate(posex1, posey1, posex2, posey2, theta):
    inc_x = posex2 -posex1
    inc_y = posey2 -posey1
    angle_goal = atan2(inc_y, inc_x)
    # if angle_to_goal < 0:
    #     angle_to_goal = (2* math.pi) + angle_to_goal
    print("angle goal = ",angle_goal)
    goal_range = theta + math.pi
    wrapped = goal_range - (2 * math.pi)
    if abs(angle_goal - theta) > 0.1:
        print(theta)
        print("goal_range = ",goal_range)
        if (goal_range) > (2 * math.pi) and (theta < angle_goal or angle_goal < wrapped):
            print("go left")
        elif (goal_range) < (2 * math.pi) and (theta < angle_goal and angle_goal < goal_range):
            print("go left")
        else:
            print("go right")

    else:
        print("go straight")

Firstly, the angle_goal from your robot's coordinate (not taking its current angle into account) is calculated by finding the arc tangent of the difference between the robot's coordinates and the goal coordinates.

In order to decide whether your robot should go left or right, we must determine where the angle_goal is relative to its current rotational direction. If the angle_goal is on the robot's left rotational hemisphere, the robot should rotate left, otherwise it should rotate right. Since we are working in Radians, π is equivilant to 180 degrees. To check whether the angle_goal is within the left hemisphere of the robot, we must add π to theta (the robot's current direction) to get the upperbound of the range of values we want to check the target may be included in. If the angle_goal is between theta and that upper bound, then the robot must turn in that direction to most efficiently reach its goal.

Consider This Example:

    smartRotation(0,0,2,2,0)

If your robot is at (0,0), its rotational direction is 0, and it's target is at (2,2), then its angle_goal would equal = 0.785. First we check whether its current angle's deviation from the angle_goal is significant by finding the difference and seeing if its larger than 0.1. If the difference between the angles is insignificant the robot should go straight towards its goal. In this case however, angle_goal - theta (0.785 - 0) is greater than 0.1, so we know that we must turn left or right to near our angle_goal. To find out whether this angle is to the left or the right of the robot's current angle, we must add π to its current angle to discover the point between its left and right hemispheres. In this case, if the angle_goal is between theta and its goal_range, 3.14 (0(theta) + π), then we would know that the robot must turn left to reach its goal.

However, if theta (your robot's current direction) + π is greater than 2π (maximum radians in a circle) then the left hemisphere of your robot is partially across the 0 radian point of the circle. To account for that case, we must calculate how far the goal range wraps around the circle passed the origin. If there is a remainder, we check whether the angle_goal is between theta and 2π or if the angle_goal is present within the remainder of the range that wraps around the origin. If either of these conditions are met then we know that your robot should turn left to most efficiently arrive at its goal, otherwise it should turn right.

Notes

You really need to know what you are doing when you are dealing at this level.

Radical Cleanup

• It turns out that it is safe to delete the build and devel subdirectories in ROS.

• Sometimes this helps:

cd ~/catkin_ws
rm -rf build/ devel/
catkin_make
source devel/setup.bash

Another way

• First: lean your build by running "catkin_make clean" in the root of your workspace.

• Second: remake your project with "catkin_make"

• Third: re-source the devel/setup.bash in your workspace.

Brandon J. Lacy

Overview

The utilization of GPS data on a robot is a common requirement within projects. However, a majority of hardware components that can be configured with the robot produce lackluster results. The iPhone uses sophisticated technology to produce more accurate GPS data, which makes it a prime candidatee for situation in which a robot is in need of accurate information. The application, GPS2IP, uses the technology of the iPhone and communicates it over the internet. Through this application and the iPhone technology an accurate vehicle to produce GPS data is obtained.

GPS2IP

The application is solely available on iOS. No research was conducted on applications on Android that produce similar functionality. There are two versions of the application on the App Store: GPS2IP ($8.99) and GPS2IP Lite (Free). The free version only allows transmission for 4 minutes before automatically turning off. The paid version has no restrictions.

Walkthrough

iPhone Configuration

Turn Off Auto-Lock

Settings > Display & Brightness > Auto-Lock > Never

GPS2IP Configuration

Enable GPS2IP

Open GPS2IP > Toggle On "Enable GPS2IP" Switch

NMEA Message Type

Open GPS2IP > Settings > NMEA Messages to Send > Only Toggle On "GLL" Switch

Python Code

#!/usr/bin/env python

'''
A module with a GPS node.

GPS2IP: http://www.capsicumdreams.com/gps2ip/
'''

import json
import re
import rospy
import socket

from std_msgs.msg import String

class GPS:
    '''A node which listens to GPS2IP Lite through a socket and publishes a GPS topic.'''
    def __init__(self):
        '''Initialize the publisher and instance variables.'''
        # Instance Variables
        self.HOST = rospy.get_param('~HOST', '172.20.38.175')
        self.PORT = rospy.get_param('~PORT', 11123)

        # Publisher
        self.publisher = rospy.Publisher('/gps', String, queue_size=1)

    def get_coords(self):
        '''A method to receive the GPS coordinates from GPS2IP Lite.'''
        # Instantiate a client object
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.connect((self.HOST, self.PORT))
            # The data is received in the RMC data format
            gps_data = s.recv(1024)

        # Transform data into dictionary
        gps_keys = ["message_id", "latitude", "ns_indicator", "longitude", "ew_indicator"]
        gps_values = re.split(',|\*', gps_data.decode())[:5]
        gps_dict = dict(zip(gps_keys, gps_values))

        # Cleanse the coordinate data
        for key in ['latitude', 'longitude']:
            # Identify the presence of a negative number indicator
            neg_num = False

            # The GPS2IP application transmits a negative coordinate with a zero prepended
            if gps_dict[key][0] == '0':
                neg_num = True
            
            # Transform the longitude and latitude into a format that can be utilized by the front-end web-client
            gps_dict[key] = float(gps_dict[key]) / 100

            # Apply the negative if the clause was triggered
            if neg_num:
                gps_dict[key] = -1 * gps_dict[key]

        # Publish the decoded GPS data
        self.publisher.publish(json.dumps(gps_dict))


if __name__ == '__main__':
    # Initialize a ROS node named GPS
    rospy.init_node("gps")

    # Initialize a GPS instance with the HOST and PORT
    gps_node = GPS()

    # Continuously publish coordinated until shut down
    while not rospy.is_shutdown():
        gps_node.get_coords()

Introduction

Amazon Web Services RoboMaker is an online service which allows users to develop and test robotic applications online. Robomaker has a number of advanced features, but this notebook page will focus on developing nodes and simulating them in Gazebo. To use RoboMaker, you will also need to use AWS S3 and Cloud9.

Getting started

Create an AWS root account if you do not already have one. A free tier account will suffice for getting started, though make note that under a free tier membership you will be limited to 25 Simulation Units (hours) for the first twelve months.

Once you have an AWS account, you should create an IAM for your account. AWS recommends not using your root user account when using services like RoboMaker. To learn how to set up IAM, click here. Remember the username and password of the account you create. Additionally, save the link to where you can log in with those credentials.

Going forward, you should be logged in with your IAM account. Log into AWS with your IAM, then proceed.

Amazon Documentation and Mini Tutorial

RoboMaker has a limited documentation set that can help you use the software. The “getting Started” section can help familiarize yourself with the software by working with a sample application. You can find this tutorial by clicking here.

Creating an S3 bucket

From the AWS Management Console, type “S3” into the “find services” field and click on S3 in the autofill list below the entry box. From the S3 Management Console, click “Create Bucket”

• On the first page, enter a name for your bucket. Under region, make sure that it is US East (N Virginia), NOT US East (Ohio), as RoboMaker does not work in the Ohio region.

• Skip step 2, “configure options”

• In step 3, “Set Permissions”, uncheck all four boxes

• Click “Create Bucket”

This bucket is where your bundled robotic applications will be stored.

Creating a Development Environment

Beck at the AWS Management Console, type “robomaker” in to the same entry field as S3 in the last part to go to the RoboMaker Management Console. In the left hand menu, under Development, select “Development environments” then click “Create Environment”

• Give your environment a name

• Keep the instance type as its default, m4 large

• Under networking, select the default VPC and any subnet, then click “create” to finish creating your environment

In your Cloud9 environment, use the bash command line at the bottom of the screen and follow these instructions: how to create a new RoboMaker application to create the directories needed to work with ROS.

At the end, you will have both a robot workspace and a simulation workspace. The robot workspace (robot_ws) contains source files which are to be run on the robot. The simulation workspace (simulation_ws) contains the launch files and the world files needed to run gazebo simulations. Going forward this guide assumes that you have directories set up exactly as described in the walkthrough linked above, especially that the folders robot_app and simulation_app exist inside the src folders of robot_ws and simulation_ws, respectively.

Adding new scripts to robot_ws

Python ROS node files should be stored in the scripts folder inside robot_app. When you add a new node, you must also add it to the CMakeLists.txt inside robot_app. In the section labelled “install” you will see scripts/rotate.py, below that line is where you should list all file names that you add (with scripts/ preceding the name).

Modifying the Simulation

When creating your directories, two files were put inside simulation_app/launch: example.launch and spawn_turtlebot.launch

• Inside spawn_turtlebot.launch, you will see a line that looks like this (it should be on line 3): <arg name="model" default="$(optenv TURTLEBOT3_MODEL waffle_pi)" doc="model type [burger, waffle, waffle_pi]"/> In this section: default="$(optenv TURTLEBOT3_MODEL waffle_pi)" you can replace waffle_pi with burger or waffle to change the model of turtlebot3 used in the simulation

• Inside example.launch you will see this line (it should be on line 8): <arg name="world_name" value="$(find simulation_app)/worlds/example.world"/> You can replace example.world with the name of another world file to change the world used for the simulation. Note that the world file must be present in the folder simulation_app/worlds. You can copy world files from the folder catkin_ws/src/turtlebot3_simulations/turtlebot3_gazebo/worlds (which is on your personal machine if you have installed ROS Kinetic)

Building and Bundling your applications

In order to use your applications in simulation, they must first be built and bundled to a format that RoboMaker likes to use. At the top of the IDE, click: RoboMaker Run → Add or Edit configurations, then click Colcon Build.

Give your configuration a name. I suggest it be robot or simulation>. For working directory, select the path to either robot_ws or simulation_ws (you will have to do this twice, once for each workspace).

Do the same, but this time for Colcon Bundle.

You now have shortcuts to build and bundle your robot and simulation applications.

Next, in the configuration window that you have been using, select workflow to create a shortcut which will build and bundle both applications with one click. Give your workflow a name, then put your actions in order. It is important that the builds go before the bundles.

Once you’ve made your workflow, go to: RoboMaker Run → workflow → your workflow to build and bundle your applications. This will create a bundle folder inside both robot_ws and simulation_ws. Inside bundle, there is a file called output.tar.gz. You can rename it if you like, but remember where it is.

Finally, we will go back to the configuration window to configure a simulation launcher.

• Give it a name

• Give it a duration (in seconds)

• Select “fail” for failure behavior

• Select an IAM role - it doesn’t necessarily matter which one, but AWSServiceRoleForRoboMaker is recommended

• Skip to the robot application section

• Give it a name

• Bundle path is the path to output.tar.gz (or whatever you renamed it) inside robot_ws/bundle

• S3 bucket should be the bucket you created at the beginning of this guide

• Launch package name is robot_app

• Launch file is the launch file you wish to use

NOTE: the name of the robot application and the launch file should related in some way The simulation application section is much of the same, except everything that was “robot” should be replaced with “simulation.”

OPTIONAL: Once your simulation launch configuration has been saved, you can add it as the final action of the workflow you made earlier.

Running a simulation

These are the steps that have been found to work when you want to run a simulation in RoboMaker. They are Kludgy, and perhaps a more elegant solution exists, but for now this is what has been found:

1. Make sure your applications have been built and bundled. Then from the IDE, go to RoboMaker Run → Simulation Launch → your simulation config. This will upload your application bundles to the S3 bucket you specified, then try to start the simulation. IT WILL PROBABLY FAIL. This is okay, the main goal of this step was to upload the bundles.

2. Go back to the RoboMaker Management Console, and in the left menu Select Simulations → Simulation Jobs, then click “Create simulation job”

3. Now we will configure the simulation job again:

   • Set the failure behavior to fail

   • For IAM role, select “create new role”, then give your role a name. Each simulation job will have its own IAM role, so make the name related to the simulation job.

   • Click next

   • For robot application, select the name you gave when you configured the simulation in the IDE. The launch package name and launch file will be the same too, but you must type those in manually.

   • Click next, the next page is for configuring the simulation application, the same rules apply here as the robot application

   • Click next, review the configuration then click create.

4. RoboMaker will begin preparing to launch your simulation, in a few minutes it will be ready and start automatically. Once it is running, you will be able to monitor it through gazebo, rqt, rviz and the terminal.

5. Be sure that once you are done with your simulation (if it is before the duration expires) to cancel the simulation job under the action menu in the upper right of the simulation job management screen.

Author: Ken Kirio

This guide will show how to set up an external camera by connecting it to a computer running ROS. This guide assumes your computer is running ROS on Ubuntu natively, not via the VNC, in order to access the computer's USB port. (The lab has computers with Ubuntu preinstalled if you need one.)

1. Installation

   • Install the ros package usb_cam: sudo apt install ros-noetic-usb-cam

   • Install guvcview for the setup: sudo apt install guvcview

2. Edit the launch file, usb_cam-test.launch

   • Find the location of the file inside the usb_cam package: roscd usb_cam

   • Set video_device parameter to the port of your camera

     • Check which ports are in use: ls /dev and look for video0, video1, etc.

     • Check the output of each port: guvcview /dev/<port>

     • If you unplug the camera between uses or restart your computer, the camera may be on a different port. Check every time!

3. Run the node! roslaunch usb_cam usb_cam-test.launch

Question

I want to run a computer vision algorithm on my robot, but I'm told that I need to calibrate my camera(s) first. What is camera calibration, and how can I do it?

What is Camera Calibration?

Camera calibration is the act of determining the intrinsic parameters of your camera. Roughly speaking, the intrinsic parameters of your camera are constants that, in a mathematical model of your camera, describe how your camera (via its interior mechanisms) converts a 3D point in the world coordinate frame to a 2D point on its image plane.

Intrinsic parameters are distinct from extrinsic parameters, which describe where your camera is in the world frame.

So, since calibration deals with the intrinsic parameters of your camera, it practically doesn't matter where you place your camera during calibration.

To hear more about the basics of camera calibration, watch the following 5-minute videos by Cyrill Stachniss in order:

1. Camera Intrinsics and Extrinsics

2. Mapping the 3D World to an Image

3. Intrinsic Camera Calibration

This video, also by Cyrill Stachniss, is a deep dive into Zhang's method, which is what the camera_calibration package we discuss below uses under the hood.

How Can I Calibrate my Camera?

This note describes two ways you can calibrate your camera. The first is by using the camera_calibration ROS package. This is the easier approach, since it basically does almost all of the work for you. The second is by using OpenCV's library directly, and writing your own calibration code (or using one in circulation).

The camera_calibration Package

This guide assumes you've already got your camera working on ROS, and that you're able to publish camera_info and image_raw topics for the camera. If you need to set up a new usb camera, see this entry in our lab notebook.

First, let's install the package:

sudo apt-get update
sudo apt-get install ros-noetic-camera-calibration

Second, print out this checkerboard on a letter-sized piece of paper.

Third, tape the corners of the paper to a firm, flat surface, like the surface of a piece of cardboard.

Fourth, measure a side of a single square, convert your measurement to millimeters, and divide the result by 1000. Let's call your result, RESULT.

Now, let the number of rows of your checkerboard be M and its number of columns N. Finally, let's say your camera node's name is CAM, such that, when you connect it with ROS, it publishes the /CAM/camera_info and /CAM/image_raw topics. Now, after ensuring that these two topics are being published, execute:

rosrun camera_calibration cameracalibrator.py --size MxN --square
RESULT image:=/CAM/image_raw camera:=CAM

Next, follow the instructions under section 4 "Moving the Checkboard", and section 5 Calibration Results of the official camera_calibration tutorial.

WARNING The two sections stated above are the only ones you actually want to follow in the official tutorial. Much of the rest of the material there is outdated or misleading.

Here's a video of what a successful calibration process might look like.

OpenCV

Sometimes, you might want to use object detection or use certain algorithms that require a camera such as VSLAM. These algorithms usually require a very good calibration of the camera to work properly. The calibration fixes things like distortion by determining the camera’s true parameters such as focal length, format size, principal point, and lens distortion. If you see lines that are curved but are supposed to be straight, then you should probably calibrate your camera.

Usually this is done with some kind of checkerboard pattern. This can be a normal checkerboard or a Charuco/Aruco board which has some patterns that look like fiducials or QR codes on it to further help with calibration. In this tutorial, we’ll be using a 7x9 checkerboard with 20x20mm squares: checkerboard pdf.

The most ideal way to do is to print the checkerboard on a large matte and sturdy piece of paper so that the checkerboard is completely flat and no reflections can be seen on it. However, it’s okay to just print it on a normal piece of paper as well and put it on a flat surface. Then, take at least ten photos with your camera from a variety of angles and positions so that the checkerboard is in all corners of the photos. Make sure the whole checkerboard is seen in each picture. Save those photos in an easy to find place and use the following to get your intrinsic calibration matrix.

The code I used was this opencv calibration. It also has more notes and information about what the information you are getting is.

Step by step:

• print out checkerboard pattern

• take at least 10 photos of the checkerboard at a variety of angles and positions (see image 1 for examples) and save in an easy to access place

• download/copy the opencv calibration code and run it after changing the folder path

• get the intrinsic matrix and distortion and enter it into whatever you need

Image 1: some examples of having the checkerboard cover all corners of the image

Potential intrinsic matrix:

[[688.00030953 0. 307.66412893]

[ 0. 689.47629485 274.053018 ]

[ 0. 0. 1. ]]

Pastable into python:

fx, fy, cx, cy = (688.00030953, 689.47629485, 307.66412893, 274.053018)

Distortion coefficients:

[9.39260444e-03, 4.90535732e-01, 1.48962564e-02, 4.68503188e-04, -1.77954077e+00]

By Harris Ripp

The detection of edges in image processing is very important when needing to find straight lines in pictures using a camera. One of the most popular ways to do so is using an algorithm called a Canny Edge Detector. This algorithm was developed by John F. Canny in 1986 and there are 5 main steps to using it. Below are examples of the algorithm in use:

The second image displays the result of using canny edge detection on the first image.

Steps to Edge Detection:

• Apply Gaussian filter to smooth image

• Find intensity gradients of image

• Apply gradient magnitude thresholding or lower bound cut-off suppression to remove false results

• Track edge by surprisessing weak edges so only strong ones appear

Application of Canny Edge Detection

The below function demonstrates how to use this algorithm:

def img_callback(self, msg):

        # Canny Edge Detection
        # Blurs image and find intensity gradients
        img = self.bridge.imgmsg_to_cv2(msg, 'bgr8')

        gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

        kernelSize = 31

First, the image is converting into something usable by cv. It is then grayed and the intensity gradient for the kernel is found

grayBlur = cv2.GaussianBlur(gray, (kernelSize, kernelSize), 0)

The image is then blurred for canny preparation.

lowEnd = 30
        highEnd = 100
        edges = cv2.Canny(grayBlur, lowEnd, highEnd)

        # Publish for use in finding centroid 
        self.img_pub.publish(self.bridge.cv2_to_imgmsg(edges))

The lower and upper bounds are decided and the Canny algorithm is run on the image. In the case of this function, the new image is then published to a topic called "canny mask" for use by another node.

The above code was created for use in a project completed by myself and fellow student Adam Ring

This tutorial is largely based on what I have learnt here: . Please refer to this official tutorial if you need more details.

1. Open a Gazebo simulation

First, open Gazebo - either search for gazebo in the Unity Launcher GUI or simply type gazebo onto the terminal. Click on Edit --> Building Editor and you should see the following page. Note there are three areas:

• Platte: You can choose models that you wish to add into the map here.

• 2D View: The only place you make changes to the map.

• 3D View: View only.

2. Import a floor plan

You may create a scene from scratch, or use an existing image as a template to trace over. On the Platte, click on import and selet a 2D map plan image in the shown prompt and click on next.

To make sure the walls you trace over the image come up in the correct scale, you must set the image's resolution in pixels per meter (px/m). To do so, click/release on one end of the wall. As you move the mouse, an orange line will appear as shown below. Click/release at the end of the wall to complete the line. Once you successfully set the resolution, click on Ok and the 2D map plan image you selected should show up in the 2D-View area.

3. Add & Edit walls

• Select Wall from Platte.

• On the 2D View, click/release anywhere to start the wall. As you move the mouse, the wall's length is displayed.

• Click again to end the current wall and start an adjacent wall.

• Double-click to finish a wall without starting a new one.

• Double-clicking on an existing wall allows you to modify it.

4. Prepare a package

You need to create a package for your Gazebo world so that you can use roslaunch to launch your it later.

• Go to your catkin workspace

  $ cd ~/catkin_ws/src

• Create a package using the following command.

  $ catkin_create_pkg ${your_package_name}

• Go to your package and create three folders launch, worlds and models.

5.Save your map

Once you finish editing the map, give a name to your model on the top on the Platte and click on File -> Save As to save the model you just created into ../${your_package_name}/models.

Click on File -> Exit Building Editor to exit. Please note that once you exit the editor, you are no longer able to make changes to the model. Click on File -> Save World As into ../${your_package_name}/worlds.

I will refer to this world file as ${your_world_file_name}.world from now on.

6.Create a launch file for your gazebo map

Go to ../${your_package_name}/launch and make a new file ${your_launch_file} Copy and paste the following code into your launch file and substitute ${your_package_name} and {your_world_file_name} with their actual names.

7. Test

Go to the workspace where your new package was created e.g. cd ~/catkin_ws

run catkin_make and then roslaunch ${your_package_name} ${your_launch_file}

You should see the Gazebo map you just created along with a turtlebot loaded.

Using the Model Editor instead

The building editor is a faster, easier to use tool than the model editor, as it can create a map in mere minutes. With the model editor, you have more technical control over the world, with the trade off being a more tedious process. The model editor can help make more detailed worlds, as you can import .obj files that can be found on the internet or made in 3d modeling software such as Blender. For the purposes of use in this class, USE THE BUILDING EDITOR For your own recreational robotic experimentation purposes, of course, do whatever you like.

If you do wish to use the model editor, here are two tips that will help you to get started making basic, serviceable worlds.

1. Change the Color

The basic shapes that gazebo has are a greyish-black by default- which is difficult to see on gazebo's greyish-black background. To change the color, follow these steps: 1. Right click on the model 1. select "open link inspector" 1. go to the "visual" tab 1. scroll down to "material" and open that section 1. use the RGB values labeled "ambient" to alter the color - set them all to 1 to make it white.

2. Alter the shape

use the shortcut s to open the scaling tool - grab the three axis to stretch the shape. Hold ctrl to snap it to the grid. use the shortcut t to switch to the translation tool - this moves the model around. Hold ctrl to snap it to the grid. use the shortcut r to open the rotation tool. grab the rings to rotate the object.

3. Make it Static

If an object isn't static, it will fall over/ obey the laws of physics if the robot collides with it - to avoid this, click the object in the left hand menu and click the is_static field.

4. Use the Building Editor instead

Does the model editor seem like a hassle already? Then just use the building editor.

Muthhukumar Malaiiyyappan (Malai)

Building a gazebo world might be a little daunting of a task when you are getting started. One might want to edit existing gazebo worlds but I will save you the trouble and state that its not going to work.

1. Open a vnc terminal

This would open a gazebo terminal and once you get in there you would want to bring your cursor to the top left hand corner and find the Building Editor in the Edit tab.

Once in the building editor click on the wall to create the boundaries on the top half of the editor. Left click to get out of the building mode. If you would like to create walls without standard increments, press shift while dragging the wall.

If you would like to increase or decrease the thickness of the wall. Click on the walls you would like to change and it will open up a modal with options to change.

After you are satisfied with their boundaries, save it as a model into your models folder within your project.

When the models have been saved. You would be brought back to gazebo with the model that you have built.

1. Change the pose of the model if need be according to your needs then save the world.

2. Upload your new model into this part of the launch file

If you find that your robot is not in the right place. Open the launch file, make the changes to the model accordingly and save it again as the world again.

Thats how you can build a world from scratch, hope this helped.

ImageNet file xml format to text format.

Full Repo here:

Installation

Usage

Example

Input xml file.

Output text file.

Motivation

Deploying a Pretrained Pytorch Model in Ubuntu 18.04 Virtual Environment

By Adam Ring

Using a pre-trained deep learning model from a framework such as Pytorch has myriad applications in robotics, from computer vision to speech recognition, and many places inbetween. Sometimes you have a model that you want to train on another system with more powerful hardware, and then deploy the model elsewhere on a less powerful system. For this task, it is extremely useful to be able to transfer the weights of your trained model into another system, such as a virtual machine running Ubuntu 18.04. These methods for model transfer will also run on any machine with pytorch installed.

Note

It is extremely discouraged to mix versions of Pytorch between training and deployment. If you train your model on Pytorch 1.8.9, and then try to load it using Pytorch 1.4.0, you may encounter some errors due to differences in the modules between versions. For this reason it is encouraged that you load your Pytorch model using the same version that is was trained on.

Saving and loading a trained model

Let's assume that you have your model fully trained and loaded with all of the necessary weights.

model = MyModel()

model.train()

For instructions on how to train a machine learning model, see in the lab notebook. There are multiple ways to save this model, and I will be covering just a few in this tutorial.

Saving the state_dict

This is reccommended as the best way to save the weights of your model as its state_dict, however it does require some dependencies to work. Once you have your model, you must specify a PATH to the directory in which you want to save your model. This is where you can name the file used to store your model.

PATH = "path/to/directory/my_model_state_dict.pt"

or

PATH = "path/to/directory/my_model_state_dict.pth"

You can either specify that the state_dict be saved using .pt or .pth format.

Then, to save the model to a path, simply call this line of code.

torch.save(model.state_dict(), PATH)

Loading the state_dict

Download the my_model_state_dict.pt/pth into the environment in which you plan on deploying it. Note the path that the state dict is placed in. In order to load the model weights from the state_dict file, you must first initialize an untrained istance of your model.

loaded_model = MyModel()

Keep in mind that this step requires you to have your model architecture defined in the environment in which you are deploying your model.

Next, you can simply load your model weights from the state dict using this line of code.

loaded_model.load_state_dict(torch.load("path/to/state/dict/my_model_state_dict.pt/pth"))

The trained weights of the model are now loaded into the untrained model, and you are ready to use the model as if it is pre-trained.

Saving and loading the model using TorchScript

TorchScript is a framework built into Pytorch which is used for model deployment in many different types of environments without having the model defined in the deployment environment. The effect of this is that you can save a model using tracing and load it from a file generated by tracing it.

What tracing does is follow the operations done on an input tensor that is run through your model. Note that if your model has conditionals such as if statements or external dependencies, then the tracing will not record these. Your model must only work on tensors as well.

Saving the trace of a model

In order to trace your trained model and save the trace to a file, you may run the following lines of code.

PATH = "path/to/traced/model/traced_model.pt/pth" dummy_input = torch.ones(typical_input_size, dtype=dype_of_typical_input) traced_model = torch.jit.trace(model, dummy_input)

torch.jit.save(traced_model, PATH)

The dummy_input can simply be a bare tensor that is the same size as a typical input for your model. You may also use one of the training or test inputs. The content of the dummy input does not matter, as long as it is the correct size.

Loading the trace of a model

In order to load the trace of a model, you must download the traced model .pt or .pth file into your deployment environment and note the path to it.

All you need to do to load a traced model for deployment in Pytorch is use the following line of code.

loaded_model = torch.jit.load("path/to/traced/model/traced_model.pt/pth")

Keep in mind that the traced version of your model will only work for torch tensors, and will not mimic the behavior of any conditional statements that you may have in your model.

Data Annotation

Authors: Julian Ho, Cass Wang

What is a BLDC motor?

BLDC motor stands for Brushless DC motor, as their name implies, brushless DC motors do not use brushes. With brushed motors, the brushes deliver current through the commutator into the coils on the rotor.

With a BLDC motor, it is the permanent magnet that rotates; rotation is achieved by changing the direction of the magnetic fields generated by the surrounding stationary coils. To control the rotation, you adjust the magnitude and direction of the current into these coils.

By switching on/off each pairs of stators really quickly, the BLDC motor can achieve a high rotational speed.

This is a simple table comparing a brushed DC motor, AC induction motor, and a BLDC motor:

BLDC motors are commonly found in drones, electric cars, even robots!

Types of BLDC motor

There are a couple different types of BLDC motor on the market for different applications. Here are some examples,

Small motor

• <150g weight

• <5cm diameter

• 11.1-22.2v operational voltage

• ~0.3 NM torque

• application: small drones

Mid-size motor

• 400-1000g weight

• 5-10cm diameter

• 22.2-44.4v operational voltage

• ~4 NM torque

• application: RC cars, electric skateboard, robot actuator

Stepper motor

• ~400g weight

• 5-8cm diameter

• 11.1-22.2v operational voltage

• ~0.9 NM torque

• application: 3D printers, servos

BLDC motor lingos

When shopping for a BLDC motor, there are a couple motor specific terms to consider.

• Max RPM (KV - RPM per volt)

  • 2200KV @ 10v = KV x V = 22,000 RPM max speed

• • 1 NM = able to lift 1 KG weight attached to the end of a 1 meter long stick

• Max Power (W - Watts)

  • 835w @ 10v = W/V = 83.5Amp max power draw

• Motor Efficiency (%)

  • 90% efficiency = 90% of theoretical power

• Input Volt (S - li-ion Cells)

  • 6S = S x 3.7V = 22.2v

• Max Current (A - Amps)

  • 50A @ 10v = A x V = 500W max power

• Motor poles (N-P)

  • 24N40P = 24 stator poles, 40 permanent magnet poles

• Outrunner/Inrunner

  • Outrunner = motor body spin with output shaft

  • Inrunner = only output shaft will spin, body is stationary

• Motor numbering

  • 6355 = 63mm diameter, 55mm length

BLDC electric speed controllers (ESC)

To drive a BLDC motor, you need a dedicated speed controller (ESC) to control it. Here are different types of ESC for different applications. These ESCs (like the motors above) are considered hobbyist-use, but they are quite sufficient for building small/mid-size robots.

Drone ESC

• very light ~9g

• very small footprint (size of a dollar coin)

• 1-6S input voltage

• ~40A max continuous current

• cheap

• application: small drone, small fighter robot, RC helicopter

• downside: cannot handle big motors, heat up very quickly, only simple motor control algorithms available

RC car ESC

• 3-12S input voltage

• ~50A max continuous current

• can handle medium size motors

• have active cooling

• affordable

• application: RC car, RC boat, electric skateboard

• downside: limited control protocol (PWM only), only simple motor control algorithms available

Robot ESC

• commonly used in robotic arm, actuator control

• more expensive

• ~120A max continuous current

• can handle large motors

• offer fine positional control of motor

• offer programmatic control (serial/USB/CANbus)

• application: robot, robotic arm

• downside: quite pricey, not plug-and-play, need to tune the motor manually before use

BLDC control algorithms

There are 2 most common motor control algorithm used in hobbyist ESCs.

• Sensorless BLDC Motor Control

  • Advantage: No need for dedicated encoder on the motor

  • Downside: Weak low speed control, less speed less torque

• Field Oriented Control (FOC)

  • Advantage: Full torque at any speed

  • Downside: Require fine motor tuning (PID), and dedicated encoder

Brendon Lu and Benjamin Blinder

Make sure you have the following packages imported: tkinter and rospy. The tkinter module is a basic and simple, yet effective way of implementing a usable GUI.

Because tkinter has a hard time recognizing functions created at strange times, you should next create any functions you want to use for your node. For this example, I recommend standard functions to publish very simple movement commands.

You still need to initialize the ros node, as well as any publishers and subscribers, which should be the next step in your code.

Now it is time to create a basic window for your GUI. First, write [window name] = tk.Tk() to create a window, then set the title and size of the window. Not declaring a window size will create a window that adapts automatically to the size of whatever widgets you create on the window.

The step is to populate your window with the actual GUI elements, which tkinter calls "Widgets". Here we will be making two basic buttons, but there are other common widget types such as the canvas, entry, label, and frame widgets.

And now that you have created widgets, you will notice that if you run your code, it is still blank. This is because the widgets need to be added to the window. You can use "grid", "place", or "pack" to put the widget on the screen, each of which have their own strengths and weaknesses. For this example, I will be using "pack".

And now finally, you are going to run the tkinter mainloop. Please note that you cannot run a tkinter loop and the rospy loop in the same node, as they will conflict with each other.

To run the node we have created here, you should have your robot already running either in the simulator or in real life, and then simply use rosrun to run your node. Here is the code for the example tkinter node I created, with some more notes on what different parts of the code does

Although this code does technically move the robot and with some serious work it could run a much more advanced node, I do not recommend doing this. I would recommend that you create two nodes: A GUI node and a robot node. In the GUI node, create a custom publisher such as command_pub=rospy.Publisher('command', Twist, queue_size=1) and use this to send messages for movement to the robot node. This way, the robot node can handle things like LiDAR or odometry without issues, since the tkinter update loop will not handle those kinds of messages very efficiently.

Overall, tkinter is an industry staple for creating simple GUIs in Python, being fast, easy to implement, versatile, and flexible, all with an intuitive syntax. For more information, check out the links below.

Problem

The InterbotixPincherX100 can rotate, move up and down, and extend. To get the arm to go to a specific point in space given (x, y, z) coordinates, the x and y compenents must be converted to polar coordinates.

Solution

Moving the arm to point (x, y), top-down view

Since the arm can move out/back and up/down to specific points without conversion, consider only a top-down view of the arm (x-y axis). To get the rotation and extension of the arm at a specific point, consider the triangle shown above. For rotation θ: $θ=atan(x/y)$. For extension r: $r=y/cos(θ)$. The up-down movement of the arm remains the same as the given z coordinate.

The arm can be moved to the desired point using set_ee_cartesian_trajectory. In this method, the extension and motion up/down is relative so the current extension and vertical position of the arm must be known as current_r and current_z.

Setting up apriltags to work with your program is pretty simple.

You can run the following two lines to install apriltags and its ros version onto your ubuntu:

sudo apt install ros-noetic-apriltag

sudo apt install ros-noetic-apriltag-ros

Afterwards, connect your camera

roslaunch usb_cam usb_cam-test.launch

• changed the video_device to /dev/video2 (or whichever video device you figure out it is) to take from usbcam

• created head_camera.yaml file from this and added the information:

roslaunch apriltag_ros continuous_detection.launch publish_tag_detections_image:=true

• can remap input topics to what you need:

  • image_rect:=usb_cam/image_raw

  • camera_info:=usb_cam/camera_info

Junhao Wang

Read CompressedImage type

When using Turtlebot in the lab, it only publishs the CompressedImage from '/raspicam_node/image/compressed'. Here's how to read CompressedImage and Raw if you need.

Limit frame rate

When the computation resource and the bandwidtn of robot are restricted, you need limit the frame rate.

Republish the image

Again, when you have multiple nodes need image from robot, you can republish it to save bandwidth.

by Veronika Belkina

This is a guide to installing the Astra Pro Depth Camera onto a robot and the various problems and workarounds that were experienced along the way.

Setup

To start, try to follow the instructions given on the .

If this goes well, then you're pretty much all set and should skip down to the usb_cam section. If this doesn't go well, then keep reading to see if any of the errors that you received can be resolved here.

Possible errors and ways to solve them

• make sure to run sudo apt update on the robot

• if you are getting an error that mentions a lock when you are installing dependencies, try to reboot the robot: sudo reboot now

If you have tried to install the dependencies using this:

and this is not working, then here is an alternative to try:

If this is successful, then within ~/catkin_ws/src, git clone https://github.com/orbbec/ros_astra_camera and run the following commands:

After this, run catkin_make on the robot. This might freeze on you. Restart the robot and run catkin_make -j1 to run on a single core. This will be slower, but will finish.

At this point, hopefully, all the errors have been resolved and you are all set with the main astra_camera package installation.

There is one more step that needs to be done.

usb_cam

Test it by running rosrun usb_cam usb_cam_node

Afterwards, when you need to run the depth camera and need the rgb stream as well, you will need to run the following instructions onboard the robot:

If this is something that you will be needing often, then it might be worth it to add the usb_cam node into the astra launch file as you do need to ssh onto the robot for each instruction. The usb_cam_node publishes to the topic /usb_cam/image_raw. You can check rostopic list to see which one suits your needs.

If you want to just explore the depth camera part of the camera, then just run the astra.launch file.

Then there will be a view topics that the camera publishes:

If you open rviz, click add, go to the by topic tab and open the PointCloud2 topic under /camera/depth/points:

If you’re working with just the camera, you might need to fix the frame and can just pick any random one for now other than map. A point cloud of what’s in front of you should appear:

If you're looking through rqt, then you might see something like this:

From there, you could use colour detection, object detection, or whatever other detector you want to get the pixels of your object and connect them with the pixels in the point cloud. It should output a distance from the camera to the object. However, I can’t speak for how reliable it is. It can’t see objects right in front of it - for example when I tried to wave my hand in front of it, it wouldn’t detect it until it was around 40 or so cm away.

The Google Coral TPU is a USB machine learning accelerator which can plugged into a computer to increase machine learning performance on Tensorflow Lite models. The following documentation will detail how to install the relevant libraries and use the provided PyCoral library in Python to make use of the Coral TPU.

Requirements

• Linux Debian 10, or a derivative thereof (such as Ubuntu 18.04)

• A system architecture of either x86-64, Armv7 (32-bit), or Armv8 (64-bit), Raspberry Pi 3b+ or later

• One available USB port (for the best performance, use a USB 3.0 port)

• Python 3.6-3.9

• Note: For use with ROS, you will want to have ROS Noetic installed on Ubuntu 20.04.

For details on how to set up for Windows or Mac OS, see Get started with the USB Accelerator on the Coral website.

Installing Required Packages

Follow the following steps in order to get your environment configured for running models on the Coral TPU

1. Open up a terminal window and run the following commands:

   • echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list

   • curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

   • sudo apt-get update

2. Plug in your Coral TPU USB Accelerator into a USB 3.0 port on your computer. If you already have it plugged in, unplug it and then plug it back in.

3. Install one of the following Edge TPU Runtime libraries:

   To install the reduced clock-speed library, run the following command:

   • sudo apt-get install libedgetpu1-std

   Or run this command to install the maximum clock-speed library:

   • sudo apt-get install libedgetpu1-max

   Note: If you choose to install the maximum clock-speed library, an excessive heat warning message will display in your terminal. To close this window, simply use the down arrow to select OK and press enter and your installation will continue.

   In order to switch runtime libraries, just run the command corresponding to the runtime library that you wish to install. Your previous runtime installation will be deleted automatically.

4. Install the PyCoral Python library with the following command:

   • sudo apt-get install python3-pycoral

You are now ready to begin using the PyCoral TPU to run Tensorflow Lite models.

Running a pretrained TFLite model

The following section will detail how to download and execute a TFLite model that has been compiled for the Edge TPU

Downloading a model

Pretrained TFLite models that have been precompiled for the Coral TPU can be found on the models section of the Coral website.

• Once you have downloaded your model, place it into a folder within your workspace.

Step 1: switch the .bashrc to be running in sim mode. Step 1.1: Go into .bashrc file and uncomment the simulation mode as shown below:

Setting for simulation mode

$(bru mode sim)

$(bru name roba -m $(myvpnip))

Step 1.2: Comment out real mode/robot ip addresses For Example:

Settings for donatello

$(bru mode real)

$(bru name donatello -m 100.106.194.39)

Step 2: run roscore on vnc. To do this type "roscore" into the terminal

Step 3: Now in the terminal do these steps Step 3.1: get vpn ip address: In the terminal type "myvpnipaddress" Step 3.2: Type "$(bru mode real)" Step 3.3: Type "$(bru name robotname -m "vpn ip address from step 4")" Step 3.4: type"multibringup" in each robot terminal

Step 4: Repeat step 3 in a second tab for the other robot(s)

How to create a new message type?

After created a ROS package, our package is constructed by a src folder, a CMakeLists.txt file, and a package.xml file. We need to create a msg folder to hold all of our new msg file. Then in your msg folder, create a new file <new_message>.msg, which contains fields you needed for this message type. For each fields in your msg file, define the name of the field and the type of the field (usually use std_msgs/ or geometry_msgs/). <br > For example, if you want your message to have a list of string and an integer, you msg file should look like:

std_msgs/String[] list
std_msgs/Int16 int

How to let the new message type recognized by ROS?

There are some modifications you need to make to CMakeLists.txt and package.xml in order to let the new message type recognized by ROS. <br > For CMakeLists.txt:

1. Make sure message_generation is in find_package().

2. Uncomment add_message_files() and add your .msg file name to add_message_files().

3. Uncomment generate_messages()

4. Modify catkin_package() to

catkin_package(
  CATKIN_DEPENDS message_runtime
)

1. Uncomment include in include_directories()

For package.xml:

1. Uncomment <build_depend>message_generation</build_depend> on line 40

2. Uncomment <exec_depend>message_runtime</exec_depend> on line 46

How to use the newly created message type?

1. How to import the message?

from package_name.msg import message_name as message_name

If your message type contains a list of buildin message type, also make sure to import that buildin message type:

from std_msgs.msg import String

1. How to use the message? <br > The publisher and subscriber's syntax are the same. However, we want to create a new topic name and make sure the new topic message type is specified. For example in my project I have a message type called see_intruder:

self.detect_intruder_pub = rospy.Publisher('/see_intruder', see_intruder, queue_size=1)
self.detect_intruder_sub=rospy.Subscriber('/see_intruder', see_intruder,self.see_intruder_callback)
  

Since our new message depends on some build in message type, when we try to access the feild of our msg, we need to do msg.<field_name>.data given that the build in message type has a field named data. So in the first example I had, if we want to access the string stored in index int of list, I need to do

msg.list[msg.int.data].data

How to check if the new message type is recognized by ROS?

Do a cm in your vnc. if the message is successfully recognized by ROS, you will see the msg file being successfully generated.

Having an error "No module named <>.msg"?

In your vnc terminal, type the command

source ~/catkin_ws/devel/setup.bash

This should solve the error.

Author: Shuo Shi

Overview

This tutorial describes the details of a SDF Model Object.

SDF Models can range from simple shapes to complex robots. It refers to the SDF tag, and is essentially a collection of links, joints, collision objects, visuals, and plugins. Generating a model file can be difficult depending on the complexity of the desired model. This page will offer some tips on how to build your models.

Components of SDF Models

Links: A link contains the physical properties of one body of the model. This can be a wheel, or a link in a joint chain. Each link may contain many collision and visual elements. Try to reduce the number of links in your models in order to improve performance and stability. For example, a table model could consist of 5 links (4 for the legs and 1 for the top) connected via joints. However, this is overly complex, especially since the joints will never move. Instead, create the table with 1 link and 5 collision elements.

Collision: A collision element encapsulates a geometry that is used for collision checking. This can be a simple shape (which is preferred), or a triangle mesh (which consumes greater resources). A link may contain many collision elements.

Visual: A visual element is used to visualize parts of a link. A link may contain 0 or more visual elements.

Inertial: The inertial element describes the dynamic properties of the link, such as mass and rotational inertia matrix.

Sensor: A sensor collects data from the world for use in plugins. A link may contain 0 or more sensors.

Light: A light element describes a light source attached to a link. A link may contain 0 or more lights.

Joints: A joint connects two links. A parent and child relationship is established along with other parameters such as axis of rotation, and joint limits.

Plugins: A plugin is a shared library created by a third party to control a model.

Building a Model

Step 1: Collect your meshes

This step involves gathering all the necessary 3D mesh files that are required to build your model. Gazebo provides a set of simple shapes: box, sphere, and cylinder. If your model needs something more complex, then continue reading.

Meshes come from a number of places. Google's 3D warehouse is a good repository of 3D models. Alternatively, you may already have the necessary files. Finally, you can make your own meshes using a 3D modeler such as Blender or Sketchup.

Gazebo requires that mesh files be formatted as STL, Collada or OBJ, with Collada and OBJ being the preferred formats.

Step 2: Make your model SDF file

Start by creating an extremely simple model file, or copy an existing model file. The key here is to start with something that you know works, or can debug very easily.

Here is a very rudimentary minimum box model file with just a unit sized box shape as a collision geometry and the same unit box visual with unit inertias:

Create the box.sdf model file

gedit box.sdf Copy the following contents into box.sdf:

<?xml version='1.0'?>
<sdf version="1.4">
  <model name="my_model">
    <pose>0 0 0.5 0 0 0</pose>
    <static>true</static>
    <link name="link">
      <inertial>
        <mass>1.0</mass>
        <inertia> <!-- inertias are tricky to compute -->
          <!-- http://gazebosim.org/tutorials?tut=inertia&cat=build_robot -->
          <ixx>0.083</ixx>       <!-- for a box: ixx = 0.083 * mass * (y*y + z*z) -->
          <ixy>0.0</ixy>         <!-- for a box: ixy = 0 -->
          <ixz>0.0</ixz>         <!-- for a box: ixz = 0 -->
          <iyy>0.083</iyy>       <!-- for a box: iyy = 0.083 * mass * (x*x + z*z) -->
          <iyz>0.0</iyz>         <!-- for a box: iyz = 0 -->
          <izz>0.083</izz>       <!-- for a box: izz = 0.083 * mass * (x*x + y*y) -->
        </inertia>
      </inertial>
      <collision name="collision">
        <geometry>
          <box>
            <size>1 1 1</size>
          </box>
        </geometry>
      </collision>
      <visual name="visual">
        <geometry>
          <box>
            <size>1 1 1</size>
          </box>
        </geometry>
      </visual>
    </link>
  </model>
</sdf>

Note that the origin of the Box-geometry is at the geometric center of the box, so in order to have the bottom of the box flush with the ground plane, an origin of 0 0 0.5 0 0 0 is added to raise the box above the ground plane.

Step 3: Add to the model SDF file

With a working .sdf file, slowly start adding in more complexity.

Under the <geometry> label, add your .stl model file

<geometry>
        <mesh filename="package://package_name/model_path/model.stl" scale="0.01 0.01 0.01"/>
</geometry>

The <geometry> label can be added below <collision> and <visual> label.

Import the model in you .world file

In your .world file, import the sdf file using < include > tag

<include>
      <uri>model://my_model</uri>
</include>

Then open termial to add the model path to Gazebo variable.

export GAZEBO_MODEL_PATH=~/catkin_ws/src/package_name/model_path/

To use a sigmoid function instead of a PID controller, you need to replace the PID control algorithm with a sigmoid-based control algorithm. Here is a step-by-step guide to do this:

1. Understand the sigmoid function: A sigmoid function is an S-shaped curve, mathematically defined as: f(x) = 1 / (1 + exp(-k * x))

   where x is the input, k is the steepness factor, and exp() is the exponential function. The sigmoid function maps any input value to a range between 0 and 1.

2. Determine the error: Just like in a PID controller, you need to calculate the error between the desired setpoint and the current value (process variable). The error can be calculated as: error = setpoint - process_variable

3. Apply the sigmoid function: Use the sigmoid function to map the error to a value between 0 and 1. You can adjust the steepness factor (k) to control the responsiveness of the system: sigmoid_output = 1 / (1 + exp(-k * error))

4. Scale the output: Since the sigmoid function maps the error to a range between 0 and 1, you need to scale the output to match the actual range of your control signal (e.g., motor speed or actuator position). You can do this by multiplying the sigmoid_output by the maximum control signal value: control_signal = sigmoid_output * max_control_signal

5. Apply the control signal: Send the control_signal to your system (e.g., motor or actuator) to adjust its behavior based on the error.

Note that a sigmoid function-based controller may not provide the same level of performance as a well-tuned PID controller, especially in terms of overshoot and settling time. However, it can be useful in certain applications where a smooth, non-linear control response is desired.

Ben Soli

Before you use this tutorial, consult with the Campus Rover Packages which outline setting up ngrok with flask and getting to the Alexa Developer Console.

The Flask Alexa Skills Kit module allows you to define advanced voice functionality with your robotic application.

In your VNC terminal

pip3 install flask-ask

Then in your rospy node file

from flask_ask import Ask, statement, question, elicit_slot, confirm_slot

declare your Flask application as

app = Flask(__name__)

and connect it to the Flask ASK

ask = Ask(app, '[endpoint]')

So, if your ngrok subdomain is campusrover and your Alexa endpoint is /commands

your code would be

ask = Ask(app, '/commands')

On the Alexa Developer Consoles define your intents and determine which slots your intents will use. Think of intents as functions, and slots as parameters you need for that function. If a slot can have multiple items in a slot mark that in the developer console. You do not need to provide the Alexa responses on the Developer Console because you will be writing them with Flask-ASK. The advantage of doing this is you can write responses to the user that take into account the robot's state and publish information to other nodes as you receive input with Alexa.

Let's assume we have an intent called 'bring_items' that can contain multiple items to fetch. Assume that the robot needs at least one item to fulfill this intent and that the robot has a weight capacity for how much it can carry. Let's also assume we have some kind of lookup table for these items which tells us their weight. With Flask-ASK we can quickly build this.

You are required to have a response for launching the intent marked as

@ask.launch
def start_skill():
  return question('hello, what you need?)

This intent is not called every single time you use the skill, but its a good way to tell the user what the bot can do or tell the user what information the bot needs from them. There are a few different response types,

statement(str: response) which returns a voice response and closes the session. question(str: response) which you can use to ask the user a question and keep the session open. elicit_slot(str: slot_name, str: response) which you can use to prompt the user for a specific type of information needed to fulfill the intent. confirm_slot(str: slot_name, str: response) which you can use to confirm with the user that Alexa heard the slot correctly.\

It is important to note, that to use elicit_slot and confirm_slot you must have a Dialog Model enabled on the Alexa Developer Console. The easiest way I have to found to enable this is to create an intent on the console that requires confirmation. To avoid this intent being activated, set its activation phrase to gibberish like 'asdlkfjaslfh'. You can check a switch marking that this intent requires confirmation.

Now let's build out an intent in our flask application.

Start with a decorator for your skill response marking which intent you are programming a response for.

@ask.intent('bring_items')
def bring_items():

First, let's assure that the user has provided the robot with some items. When you talk to Alexa, it essentially just publishes a JSON dictionary to your endpoints which your flask app can read. To get a list of the slots for this intent use the line:

  slots = request.get_json()['request']['intent']['slots']

Let's call our items slots 'items'. To check that the user has provided at least one item, write.

if 'value' not in slots['items'].keys() 
  return elicit_slot('items', 'what do you want me to bring?)

This will check that there is an item, and if there is not, prompt the user for some items. The string python 'value' is a key in the dictionary if and only if the user has provided the information for this slot, so this is the best way to check.

Let's assume that our robot has a carrying capacity of 15 pounds and has been asked to carry some items that weigh more than 15 pounds. Once we've checked that the items are too heavy for the robot, you can elicit the slot again with a different response like

elicit_slot('items','those are too heavy, give me something else to bring')

and the user can update the order. Returning elicit_slots keeps the Alexa session open with that intent, so even though each call is a return statement, you are essentially making a single function call and updating a single JSON dictionary.

Once the robot is given a list of items that it can carry, you can use a rospy publisher to send a message to another node to execute whatever robotic behavior you've implemented.

You should also include

@ask.intent('AMAZON.FallbackIntent')

This is an intent you can use for any phrase that you have not assigned to an intent that your robot will use. If you do not implement a response for this, you will very likely get error messages from Alexa.

Skills come prebuilt with some intents such as these. If a user activates one of these intents, and you don't define a response, you will get a skills response error. You should thoroughly test possible utterances a user might use to avoid errors. Skills response errors do not crash the application, but it makes for bad user-experience.

Another default intent you will likely need is

@ask.intent('AMAZON.CancelIntent')

This handles negative responses from the user. An example of this is:

User: Alexa launch campus rover Alexa: hello, what do you need User: nothing Alexa: AMAZON.CancelIntent response\

I hope this tutorial has made clear the advantages of using the flask-ASK for your Alexa integration. It is a great way to rapidly develop different voice responses for your robot and quickly integrate those with robotic actions while avoiding the hassle of constantly rebuilding your Alexa skill in the Developer Console.

arm setup

Links

• Arm Details

• Quickstart Guide

• Hardware setup

  • only needed to install the grippers

    • the screws would not go through initially - had to go from the other side with the screw first to ensure that the holes were completely open and the screw could be secured and then installed the grippers as the instructions said

  • plug in power supply first and then plug usb into computer and then plug microUSB into arm

• ROS Installation Guide

• Troubleshooting

• DYNAMIXEL Software

  • after installing software and plugging arm in, scan for the arm in the dynamixel software to check that everything is working properly:

    • in options, select baudrates 57600 and 1000000

    • if any link is in 57600, then change to 1000000

  • make sure to disconnect before running ros

  • NOTE: you may not actually need to do this, but it may be good to do the first time you try to connect the arm to your computer to make sure everything is running correctly.

• Arm Control

  • contains command line configs for launch file

• Python-ROS Interface

  • contains details on methods that control the arm using their methods - can find basic overview at the bottom of this file.

Installation:

On Intel/AMD based processor:

sudo apt install curl
curl 'https://raw.githubusercontent.com/Interbotix/interbotix_ros_manipulators/main/interbotix_ros_xsarms/install/amd64/xsarm_amd64_install.sh' > xsarm_amd64_install.sh
chmod +x xsarm_amd64_install.sh
./xsarm_amd64_install.sh -d noetic

Basic Commands:

• move the arm manually:

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100

• disable torque:

  • rosservice call /px100/torque_enable "{cmd_type: 'group', name: 'all', enable: false}"

• re-enable torque to hold a pose:

  • rosservice call /px100/torque_enable "{cmd_type: 'group', name: 'all', enable: true}"

• run with moveit:

  • roslaunch interbotix_xsarm_moveit xsarm_moveit.launch robot_model:=px100 use_actual:=true dof:=4

  • roslaunch interbotix_xsarm_moveit xsarm_moveit.launch robot_model:=px100 use_gazebo:=true dof:=4

• run using ROS-PYTHON API:

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100 use_sim:=true

  • roslaunch interbotix_xsarm_control xsarm_control.launch robot_model:=px100 use_actual:=true

• play with joints:

  • roslaunch interbotix_xsarm_descriptions xsarm_description.launch robot_model:=px100 use_joint_pub_gui:=true

• publish static transforms between two frames:

  • rosrun tf static_transform_publisher x y z yaw pitch roll frame_id child_frame_id period(milliseconds)

Interbotix Python-ROS Interface

• arm.set_ee_pose_components()

  • sets an absolute position relative to the base of the frame

    • ee_gripper_link frame with respect to the base_link frame

• arm.set_single_joint_position()

  • move the specified joint

  • usually used for the waist to turn the robot

• arm.set_ee_cartesian_trajectory()

  • move the end effector the specified value in each direction relative to the current position

  • for a 4dof arm, the y and yaw values cannot be set through this

• arm.go_to_sleep_position()

  • return the arm to the sleep position

• arm.go_to_home_position()

  • return the arm to the home position

• gripper.open()

  • open the gripper

• gripper.close()

  • close the gripper

• arm.set_trajectory_time()

  • moving_time - duration in seconds it should take for all joints in the arm to complete one move.

  • accel_time - duration in seconds it should take for all joints in the arm to accelerate/decelerate to/from max speed.

PID is a common and powerful programming tool that can be used to maintain stability and accuracy in many tasks that give feedback. This guide will walk through how to de-mystify PID and teach you to become a confident PID user.

PID stands for _P_roportional, _I_ntegral and _D_erivative. All three mathematical techniques are used in a PID controller.

First, let's think of a PID application. There is a wheel on a motor with a quadrature shaft encoder that you desire to spin at a certain RPM. In this example, imagine you have already written the code to get the speed of the motor by reading the encoder data.

The goal is for the wheel to spin at 60 RPM. so 60 is our Setpoint Variable. the actual speed that the motor encoders report is the Process Variable. By subtracting the Process Variable PV from the Setpoint Variable SP, we get the error

double error = target_speed - get_wheel_speed(); // assume this function has been written and returns motor speed in RPM

We will use this error three ways, and combine them using weights (called Gains) to acheive PID control. The three components answer three questions:

1. How big is my error?

2. How much error have I accumulated?

3. Is my error getting bigger or smaller?

Proportional

Proportional control could not be simpler: multiply the error by some constant Kp

double p = error * Ki;

Integral

If you know about Integrals then you know that they represent the area under a curve, which is a sum of all the points on said curve. Therefore, we can calculate the Integral component by summating the total error over time, like this:

double i;
...
i += Ki * error * change_in_time();

Of course, Ki is the Integral Gain constant, similar to Kp. Error is multiplied by the change in time because remember, the integral is the area under the curve. So if we plot error on the Y axis, time would likely be plotted on the x axis, thus area is error * time.

Derivative

Derivatives are the rate at which things change, so naturally a simple way to get the Derivative component is to compare the current error to the pervious error, and then account for time, like this:

double d = Kd * (error - pervious_error) / change_in_time();
previous_error = error;

Again, Kd is the derivative gain constant, and we divide by the change in time because if the change is 4 units in half a second, then the rate of change is 8 units/second (4 / 0.5)

What to do with P, I and D

Add them up. This is an adjusted representation of your control system's error. Therefore, you can apply it as a change to your previous command to try to get closer to the Setpoint Variable

double pid = p + i + d;
command = previous_command + pid;
previous_command = command;

Then your command can be sent to your actuator. Please note that some additional arithmatic may be required, this is a bare-bones simple example and does by no means serve as a copy+paste solution to all PID applications.

Tuning

A PID system must be tuned with the proper values of Kp, Ki and Kd in order to acheive smooth control. The main goals of tuning are to minimize error and over shooting the Setpoint Goal. There are plenty of guides and theories as to how to tune, so here is the Google Search to get you started

Evalyn Berleant, Kelly Duan

Arguments and parameters are important tags for roslaunch files that are similar, but not quite the same.

What are parameters?

Parameters are either set within a launch file or taken from the command line and passed to the launch file, and then used within scripts themselves.

Getting parameters

Parameters can be called inside their nodes by doing

# get a global parameter
rospy.get_param('/global_param_name')

# get a parameter from our parent namespace
rospy.get_param('param_name')

# get a parameter from our private namespace
rospy.get_param('~private_param_name')

Example from ROS parameter tutorial.

Adding parameters to launch files

Setting Parameters

Parameters can be set inside nodes like such (python):

rospy.set_param('some_numbers', [1., 2., 3., 4.])
rospy.set_param('truth', True)
rospy.set_param('~private_bar', 1+2)

For instance, if you wanted to generate a random number for some parameter, you could do as follows:

<param name="robot_position" command="$(find some_package)/scripts/generate_random_position.py"/>

which would generate a random position for the parameter.

Be careful that if you are setting parameters in more than one place that they are set in order correctly, or one file may overwrite the parameter’s value set by another file. (See links in resources for more detail).

What are arguments?

While parameters can pass values from a launch file into a node, arguments (that look like <arg name=”name”/> in the launch file) are passed from the terminal to the launch file, or from launch file to launch file. You can put arguments directly into the launch file like such and give it a value (or in this case a default value):

<launch>
  <arg name="x_pos" default="0.0" />
  <arg name="y_pos" default="0.0" />
  <arg name="z_pos" default="0.0" />
...

Or you can pass arguments into “included” files (launch files included in other launch files that will run):

<!-- start world and launch gazebo -->
  <include file="$(find gazebo_ros)/launch/empty_world.launch">
    <arg name="world_name" value="$(find swarmbots)/worlds/$(arg world).world"/>
    <arg name="paused" value="true"/>
    <arg name="use_sim_time" value="true"/>
    <arg name="gui" value="true"/>
    <arg name="headless" value="false"/>
    <arg name="debug" value="false"/>
  </include>

Substitution args

Substitution args, recognized by the $ and parentheses surrounding the value, are used to pass values between arguments. Setting the value of a parameter or argument as value=”$(arg argument_name)” will get the value of argument_name in the same launch file. Using $(eval some_expression) will set the value to what the python expression at some_expression evaluates to. Using $(find pkg) will get the location of a package recognized by the catkin workspace (very often used).

The if attribute can be used on the group tag, node tag, or include tag and work like an if statement that will execute what is inside the tag if true. By using eval and if together, it is possible to create loops to run files recursively. For example, running a launch file an arbitrary number of times can be done by specifying the number of times to be run in the launch file, including the launch file within itself, and decrementing the number of times to be run for each recursive include launch, stopping at some value checked by the if attribute. Here is an example of a recursive launch file called follower.launch to spawn in robots.

<launch>
  <arg name="followers" />
  <arg name="ns" />

  <!-- BEGIN robot[#] -->
  <group ns="$(arg ns)">

    <param name="tf_prefix" value="$(arg ns)" />
    <include file="$(find swarmbots)/launch/one_robot.launch">
      <arg name="robot_name" value="$(arg ns)" />
      <arg name="followers" value="$(arg followers)" />
    </include>
  </group>

  <!-- recursively start robot[#-1] -->
  <arg name="new_followers" value="$(eval arg('followers') - 1)" />
  <include file="$(find swarmbots)/launch/follower.launch" if="$(eval arg('new_followers') >= 0)">
    <arg name="followers" value="$(arg new_followers)" />
    <arg name="ns" value="robot$(arg new_followers)" />
  </include>
</launch>

followers here will impact the number of times the launch file is recursively called. $(eval arg('followers') - 1) will decrement the value of followers inside each recursive launch, and the if attribute

 if="$(eval arg('new_followers') >= 0)"

checks that once the new number is below 0, it will not call the launch file again.

Differences between arguments and parameters (important!)

Both arguments and parameters can make use of substitution args. However, arguments cannot be changed by nodes like parameters are with rospy.set_param(). Because of the limits of substitution, you cannot take the value of a parameter and bring it to an argument. If you want to use the same value between two params that require generating a specific value with rospy.set_param() then you should create another node that sets both parameters at once.

For example, this script

#!/usr/bin/env python
import random, rospy, roslib

rospy.init_node("generate_random_x")
pos_range = float(rospy.get_param('pos_range', 3))

x_pos = random.uniform(-pos_range / 2, pos_range / 2)
rospy.set_param('map_merge/init_pose_x',x_pos)

print(x_pos)

is called within a parameter using the command attribute.

<param name="x_pos" command="$(find swarmbots)/src/generate_random_x.py" />

The command attribute sets the value of the parameter to whatever is printed by stdout in the script. In this case, the script generates a random number for x_pos. In the same file, rospy.setparam is called to set another parameter to the same value of x_pos. In that way, both parameters can be set at once.

Too many parameters? Use rosparam!

If you have too many parameters and/or groups of parameters, not only is it inefficient to write them into a launch file, but is also prone to many more errors. That is when a rosparam file comes in handy--a rosparam file is a YAML file that stores parameters in an easier-to-read format. A good example of the utility of rosparam is the parameters for move_base, which uses the command

<rosparam file="$(find turtlebot3_navigation)/param/local_costmap_params.yaml" command="load" />

which loads the parameters from the yaml file here:

local_costmap:
  global_frame: odom
  robot_base_frame: base_footprint

  update_frequency: 10.0
  publish_frequency: 10.0
  transform_tolerance: 0.5  

  static_map: false  
  rolling_window: true
  width: 3
  height: 3
  resolution: 0.05

References

• Ros Parameters Tutorial

• Substitution args

• Order of launch files

Question

How do I set up the PX-100 arm to work with ROS2?

Answer

Introduction

The PX-100 arm currently supports ROS2 Galactic on Ubuntu Linux 20.04, or ROS2 Humble on Ubuntu Linux 22.04. This is unlikely to change in the future, as Trossen Robotics seems to have stopped working further on the PX-100.

This guide assumes you will be using ROS2 Humble on Ubuntu Linux 22.04. If you need to, it wouldn't be too hard to adapt the instructions here for a setup on ROS2 Galactic.

Install Ubuntu

Follow these official instructions, or others you might find on the web, to install Ubuntu Linux 22.04 on your computer. You might meet with a few hiccups along the way, e.g., about Ubuntu being incompatible with RST, etc. When you do, don't panic, and search for the simplest solution to the problem.

Check for Hardware Compatibility

Plug in the arm to a usb port of your computer. After waiting for a bit, run the lsusb command to see the USB devices connected to your computer. If you see something like:

Bus 001 Device 003: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC

as a line in your output, your computer is probably compatible with the arm.

Install ROS2 Humble, colcon, and rosdep

First, install ROS2 Humble. The official guide is good. The only recommendation I'd make is to add source /opt/ros/humble/setup.bash to your .bashrc file (if you're using a BASH shell. To see whether you are, run echo $SHELL. If the output is /bin/bash, you're using BASH).

Second, install colcon, ROS2's build tool.

sudo apt install python3-colcon-common-extensions

Third, install rosdep. Run:

sudo apt install python3-rosdep

Then execute

sudo rosdep init
rosdep update

Install Interbotix's Software for ROS2

Finally, it's time to install software for the PX-100 arm. Execute:

sudo apt install curl
curl 'https://raw.githubusercontent.com/Interbotix/interbotix_ros_manipulators/main/interbotix_ros_xsarms/install/amd64/xsarm_amd64_install.sh' > xsarm_amd64_install.sh
chmod +x xsarm_amd64_install.sh
./xsarm_amd64_install.sh -d humble

If you're using galactic, replace 'humble' with 'galactic' in the last line.

After installation, follow these Installation Checks from Trossen Robotics, to confirm that your installation was successful.

Checking that Things Work as Expected

Execute the following line:

ros2 launch interbotix_xsarm_control xsarm_control.launch.py robot_model:=px100

This should launch an RVIZ window displaying a graphical representation of your arm. Don't panic if you don't get the arm on the first try, and there are white blocks where components of the robot should be. Try closing rviz, and running the command again.

If this still doesn't work, check to see if the arm is in the sleeping position. If it isn't, unplug the arm, and gently move it into the sleeping position.

Even if the arm is in a sleeping position, unplug, and then replug the arm.

Run the command above again. It should have worked this time. If it still doesn't, there was probably something wrong with the installation process. Or the hardware of the arm is incompatible with your computer for mysterious reasons.

If rviz does work, execute the following command:

ros2 service call /px100/torque_enable interbotix_xs_msgs/srv/TorqueEnable "{cmd_type: 'group', name: 'all', enable: false}"

This command disables the mechanisms on the robot that keep its joints fixed in their positions. This means that you should be able to move the robot's joints manually. Try this, GENTLY, and see the graphical representation of the robot on RVIZ move in tandem.

After you're satisfied with your experiment, place the robot back into its sleeping position, and quit the two ros2 processes you started.

Next Steps

The documentation for Interbotix's ROS2 API is very lacking. If you're curious about something, try searching our lab notes first to see if you can find what you're looking for.

Otherwise, dig into the source code, installed under the interbotix_ws directory. The directory's path is likely ~/interbotix_ws. Don't be afraid to do this! It's what you'll have to do anyway if you join a company or work on a substantial open source project. Don't be afraid to modify the source code either, if you feel it's not working as it should! The developers are only human, and may have made mistakes.

Overview of ROSBridge and ROSLIBJS

This FAQ is deigned to provide an overview of ROSBridge and ROSLIBJS, two important tools for integrating web applications, such as the command control dashboard, with the Robot Operating System (ROS).

Table of contents

• What are ROSBridge and ROSLIBJS?

• How do ROSBridge and ROSLIBJS work together?

• How do I install and use ROSBridge and ROSLIBJS?

• Are there any limitations or challenges when using ROSBridge and ROSLIBJS?

What are ROSBridge and ROSLIBJS?

ROSBridge is a package for the Robot Operating System (ROS) that provides a JSON-based interface for interacting with ROS through WebSocket protocol (usually through TCP). It allows external applications to communicate with ROS over the web without using the native ROS communication protocol, making it easier to create web-based interfaces for ROS-based robots.

ROSLIBJS is a JavaScript library that enables web applications to communicate with ROSBridge, providing a simple API for interacting with ROS. It allows developers to write web applications that can send and receive messages, subscribe to topics, and call ROS services over websockets.

How do ROSBridge and ROSLIBJS work together?

ROSBridge acts as a bridge between the web application and the ROS system. It listens for incoming WebSocket connections and translates JSON messages to ROS messages, and the other way around. ROSLIBJS, on the other hand, provides an API for web applications to interact with ROSBridge, making it easy to send and receive ROS messages, subscribe to topics, and call services.

In a typical application utilizing ROSBridge and ROSLIBJS, it would have the following flow:

1. Web client uses ROSLIBJS to establish a WebSocket connection to the ROSBridge server, through specifying a specific IP address and port number.

2. The web client sends JSON messages to ROSBridge, which converts them to ROS messages and forwards them to the appropriate ROS nodes.

3. If the node has a reply, the ROS nodes send messages back to ROSBridge, which converts them to JSON and sends them over the WebSocket connection to the web client.

4. ROSLIBJS API processes the incoming JSON messages so it can be displayed/utilized to the web client

How do I install and use ROSBridge and ROSLIBJS?

ROSBridge Installation

The following assumes that you have ROS Noetic installed on your system.

To install ROSBridge you need to install the rosbridge-server package

sudo apt install ros-noetic-rosbridge-server

ROSLIBJS Installation

You can either use the hosted version of ROSLIBJS or download it for use in your project. To use the hosted version, include the following script tag in your HTML file:

<script src="https://static.robotwebtools.org/roslibjs/current/roslib.min.js"></script>

To download ROSLIBJS, visit the Github repository builds and download and save the files. To use the local version, include the following script tag in your HTML file:

<script src="PATH-TO-DOWNLOADED-SCRIPT"></script>

Simple Example

The following example is for React Application

First, download the roslib.min.js file from the ROSLIBJS GitHub repository and place it in your project's public directory.

Next, create a new React component called RosConnect:

RosConnect.jsx

import React, { Component } from 'react';
import { Alert } from 'react-bootstrap';

class ROSConnect extends Component {

    constructor() {
        super()
        this.state = { connected: false, ros: null } 
        
    }
    // run the function as soon as the page renders
    componentDidMount() {
        this.init_connection()
    }

    // a function to connect to the robot using ROSLIBJS
    init_connection() {
        this.state.ros = new window.ROSLIB.Ros()
        this.state.ros.on("connection", () => {
            this.setState({connected: true})
        })

        this.state.ros.on("close", () => {
            this.setState({connected: false})
            // try to reconnect to rosbridge every 3 seconds
            setTimeout(() => {
                try{
                    // ip address of the rosbridge server and port
                    this.state.ros.connect('ws://127.0.0.1:9090')
                }catch (error) {
                    console.log("connection error:", error);
                }
            // if the robot disconnects try to reconnect every 3 seconds (1000 ms = 1 second)
            }, 3000); 
        })

        try{
            // connect to rosbridge using websocket 
            this.state.ros.connect('ws://127.0.0.1:9090')
        }catch (error) {
            console.log("connection error:", error);
        }

    }

    render() { 
        return (
            // a Alert component from react-bootstrap showing if the robot is connected or not
            <div>
                <Alert className='text-center m-3' variant={this.state.connected ? "success" : "danger"}>
                    {this.state.connected ? "Robot Connected" : "Robot Disconnected"}
                </Alert>
            </div>
        );
    }
}
 
export default ROSConnect;

This component can be rendered to any page, for example it can be used in the App.js component which already comes with creat-react-app

Challenges and Limitations of ROSBridge and ROSLIBJS

Even though ROSBridge and ROSLIBJS is has a lot of use cases from being able to view camera feed from a robot to getting its GPS data display on a dashboard, it does have some prominent limitations.

While working on the campus command control project, one of the issues that was encountered was lag. ROSLIBJS uses web socket, which is built on top of Transmission Control Protocol (TCP). While TCP is more reliable, it transfers data more slowly, which led to lag in robot controls and video feed. It is worth mentioning that ROSBridge does support User Datagram Protocol (UDP), which comes at a cost of reliability for speed, but ROSLIBJS current implementation does not support UDP.

Reinforcement Learning and its Applications

Authors: Rachel Lese, Tiff Lyman

A lot of people are intimidated by the idea of machine learning, which is fair. However, machine learning doesn't have to be some complicated multi-layered network. In fact, you can implement a machine learning algorithm with just a dictionary in ROS. Specifically, you can run a reinforcement learning algorithm that replaces PID to steer a robot and have it follow a line.

What is Reinforcement Learning?

Reinforcement learning is exactly what it sounds like, learning by reinforcing behavior. Desired outcomes are rewarded in a way that makes them more likely to occur down the road (no pun intended). There are a few key components to a reinforcement learning algorithm and they are as follows:

• A set of possible behaviors in response to the reinforcement

• A quantifiable observation/state that can be evaluated repeatedly

• A reward function that scores behaviors based on the evaluation

With this, the algorithm takes the highest scoring behavior and applies it.

Application: Line Follower

Lets look at what those required traits would be in a Line Follower.

Behaviors

In this case would be our possible linear and angular movements. At the start we want everything to be equally likely, so we get the following dictionaries.

angular_states_prob =  { -.5: .5, -.45: .5, -.4: .5, -.35: .5, -.3: .5, -.25: .5, -.2: .5, -.15: .5, -.1: .5, -.05: .5, 0 : .5, .05 : .5, .1 : .5, .15 : .5, .2: .5, .25: .5, .3: .5, .35: .5, .4: .5, .45: .5, .5: .5 }
linear_states_prob = { 0.1 : .5, 0.25 : .5, 0.4 : .5 }

Here the key is the possible behavior (in radians/sec or m/sec) and the value is the associated weight.

Observation to Evaluate

As for the observation that we evaluate at regular intervals, we have camera data in CV callback. Just like with the original line follower, we create a mask for the camera data so that we see just the line and have everything else black. With this we can get the "center" of the line and see how far it is from being at the center of the camera. We do this by getting the center x and y coordinates as shown below.

moments = cv2.moments(mask_yellow)
        if moments['m00'] > 0:
            cx = int(moments['m10']/moments['m00'])
            cy = int(moments['m01']/moments['m00'])

Here cx and cy represent pixel indecies for the center, so if we know the resolution of the camera we can use this to establish preferable behaviors. To do this, we want our angular movement to be proportional to how far we deviate from the camera's center. For the Waffle model in ROS, we determined that it's roughly 1570x1000, meaning that cx has a range of (0,1570) and cy has a range of (0,1000).

Reward Function

This next part might also be intimidating, only because it involves some math. What we want to do is convert our pixel range to the range of possible outcomes and reward velocities based on how close they are to the converted index. For example, right turns are negative and further right means greater cx, so the line of code below does the transformation (0,1570) => (0.5, -0.5) for cx:

angle = (float(cx)/1570.0 - 0.5) * -1.0

With this, we check which keys are close to our value angle and add weight to those values as follows:

for value in angular_states_prob.keys():
            angleSum += angular_states_prob[value]
            if abs(value - angle) < 0.05:
                angular_states_prob[value] = angular_states_prob[value]*5
            elif abs(value - angle) < 0.1:
                angular_states_prob[value] = angular_states_prob[value]*2

So if the key is closest to our angle we multiply the weight by 5, and if it's somewhat close we multiply it by 2. To make sure weights don't go off to infinity, we have the value angleSum which keeps track of the sum of the weights. If it exceeds a certain limit, we scale all of the weights down. Similarly, we dont scale a weight down if it is less than 0.05 so that they don't become infinitesimally small.

Conclusion

This is just one example of reinforcement learning, but it's remarkably ubiquitous. Hopefully this demonstrates that reinforcement learning is a straightforward introduction to machine learning.

This FAQ section assumes understanding of creating a basic Gazebo world and how to manipulate a XML file. This tutorial relies on the assets native to the Gazebo ecosystem.

Setting up empty Gazebo world

By Nathan Cai

(If you have a prexisting Gazebo world you want to place an actor you can skip this part) Empty Gazebo worlds often lack a proper ground plane so it must be added in manually. You can directly paste this code into the world file.

<?xml version="1.0" ?>
<sdf version="1.6">
   <world name="default">
      <!-- A ground plane -->
      <include>
         <uri>model://ground_plane</uri>
      </include>
      <!-- A global light source -->
      <include>
         <uri>model://sun</uri>
      </include>

Placing an actor in the world

TL:DR Quick Setup

Here is the quick setup of everything, one can simply copy and paste this code and change the values to suit the need:

(If you do not have a Plugin for the model, please delete the Plugin section)

<actor name="actor1">
    <pose>0 0 0 0 0 0</pose>
    <skin>
        <filename>PERSON_MESH.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="NAME">
        <filename>ANIMATION_FILE_NAME.dae</filename>
        <scale>1.000000</scale>
        <interpolate_x>true</interpolate_x>
    </animation>
    <plugin name="PLUGIN NAME" filename="PLUGIN_FILE_NAME.so">
        ...
        ...
        ...
    </plugin>
    <script>
        <trajectory id="0" type="NAME">
            <waypoint>
                <time>0</time>
                <pose>0 2 0 0 0 -1.57</pose>
            </waypoint>
            <waypoint>
                <time>2</time>
                <pose>0 -2 0 0 0 -1.57</pose>
            </waypoint>
            ...
                ...
                ...
            ...
        </trajectory>
      </script>
</actor>

Defining an actor

Human or animated models in Gazebo are called actors, which contains all the information of an actor. The information can include: pose, skin, animation, or any plugins. Each actor needs a unique name. It uses the syntax:

<actor name="actor">
    ...
    ...
    ...
</actor>

Change the actor pose

The pose of the is determined using the pose parameter of an actor. The syntax is: (x_pos, y_pos, z_pos, x_rot, y_rot, z_rot)

<actor name="actor">
    <pose>0 0 0 0 0 0</pose>
    ...
    ...
</actor>

Add in Skins

The skin is the mesh of the actor or model that you want to place into the world. it is placed in the actor group and takes in the input of the filename. The syntax is:

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
</actor>

The mesh scale is also adjustable by changing the scale parameter.

Add in Animations

Though the actor can operate without animations, it is preferable for you to add one, especially if the model is to move, as it would make the enviorment more interesting and realistic.

To add an animation to the actor, all it needs is a name fore the animation, and the file that contains the animation. The syntax for this is: NOTE: THE FILE BECOMES BUGGY OR WILL NOT WORK IF THERE IS NO SKIN.

IMPORTANT: IN ORDER FOR THE ANIMATION TO WORK, THE SKELETON OF THE SKIN MUST BE COMPATABLE WITH THE ANIMATION!!!!

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="walking">
        <filename>walk.dae</filename>
        <scale>1.000000</scale>
    </animation>
</actor>

The animation can also be scaled.

Scripts

Scripts are tasks that you can assign an actor to do, in this case it is to make the actor walk around to specific points at specific times. The syntax for this is:

<actor name="actor">
    <skin>
        <filename>moonwalk.dae</filename>
        <scale>1.0</scale>
    </skin>
    <animation name="walking">
        <filename>walk.dae</filename>
        <scale>1.000000</scale>
    </animation>
    <script>
        <trajectory id="0" type="NAME">
            <waypoint>
                <time>0</time>
                <pose>0 2 0 0 0 -1.57</pose>
            </waypoint>
            <waypoint>
                <time>2</time>
                <pose>0 -2 0 0 0 -1.57</pose>
            </waypoint>
            ...
                ...
                ...
            ...
        </trajectory>
    </script>
</actor>