For new contributors of Percetion_CV team, please first create your own branch and make sure all your work is done within your branch. Do PR (pull request) only if your team leader asks you to do so.
For new team leaders of Perception_CV, the master branch should only contain stable code that has been confirmed working. Master branch will be the source we use for integration with other teams when the time is ready.
Full CV Repo here: https://github.com/campusrover/Robotics_Computer_Vision
This repo is originally forked from https://github.com/ultralytics/yolov3 but heavily modified for our own use. The purpose of this repo is to achieve custom object detection for Brandeis Autonomous Robotics Course. Changes were made based on our object of deploying CV on ROS.
To download our most recent best trained weights, please go to https://drive.google.com/file/d/1DquRwpNDaXgkON2gj9Oks8N2ZWgN2Z9q/view?usp=sharing
Then unzip the file and copy coco and weights directory in this repo and replace everything.
Notes:
I've put a low of useful tools inside the ./utils directory, please feel free to use them whenever you need it.
./utils/vid_2_frm.py : The python script that extracts all frames out of a video, you can control the extracting rate by reading the comment and do small modification. This script will also tell you the fps of the source video which will be useful for later converting frames back to video.
./utils/frm_2_vid.py : The python script that is converting frames by its name into a video, you better know the original/target video's fps to get the optimal output.
./utils/xml_2_txt : The repo that converts .xml format annotation into our desired .txt format (Yolo format), read and follow the README file inside.
./utils/labelimg : The repo that we use for labelling images, great tool! Detailed README inside.
./utils/check_missing_label.py : The python script that can be used for checking if there's any missing label in the annotation/image mixed directory.
./utils/rename_dataset.py : The python script doing mass rename in case different datasets' images names and annotations are the same and need to be distinguished.
./list_img_path.py : The python script that splits the datasets (images with its corresponding annotations) into training set and validation set in the ratio of 6:1 (you can modify the ratio).
./utils/img_2_gif.py : The python script that converts images to gif.
./coco/dataset_clean.py : The python script that cleans the uneven images and labels that is going to be trained and make sure they are perfectly parallel.
./utils/video_recorder_pi.py : The python script that records videos on pi camera. This script should be located in the robot and run under SSH
Here are links to download our datasets (images and annotations) by certain class:
Doorplate Recognition:
custom_volen(provided by Haofan): https://drive.google.com/file/d/1A9yI5PdLeAlKEVQww2NJgQLRDxq9tcOJ/view?usp=sharing
custom_doorplate(provided by Haofan): https://drive.google.com/file/d/1jITWceHYYFXjUyaJ1bp4Wdb_tKhtylQJ/view?usp=sharing
Facial Recognition:
All the CV subscriber and publisher are located at ./utils/ directory, they are:
./utils/image_subscriber.py : The python script that subscribe image from raspicam_node/image rostopic.
./utils/string_publisher.py : The python script that publishes a string on rostopic of /mutant/face_detection which is generated from detect.py, the format is explained below:
CV Publisher example: "['sibo', -4.34, 1.63]"
[ <"class name">, <"angle of target to front in degree (negative -> left, positive -> right")>, <"rough distance in meter"> ]
Detailed official user guide here: http://emanual.robotis.com/docs/en/platform/turtlebot3/appendix_raspi_cam/
Some useful commands:
raspivid -vf -hf -t 30000 -w 640 -h 480 -fps 25 -b 1200000 -p 0,0,640,480 -o pivideo.h264 recording 30 seconds video on 25 fps.
MP4Box -add pivideo.h264 pivideo.mp4 converting .h264 video to .mp4
scp donatello@129.64.243.61:~/pivideo.mp4 ~/Downloads/ downloading video from ssh to local machine
rqt_image_view getting vision from camera, requires bringup which is conflict to the video recording function
rosrun rqt_reconfigure rqt_reconfigure edit camera configuration
Pipeline of recording video on DONATELLO:
ssh donatello@129.64.243.61
If you want to see preview images, roslaunch turtlebot3_bringup turtlebot3_rpicamera.launch, then on remote computer, do rqt_image_view
when you recording video, shut down the rpicamera bringup in advance
Do raspivid -vf -hf -t 30000 -w 640 -h 480 -fps 25 -b 1200000 -p 0,0,640,480 -o pivideo.h264 on DONATELLO to record video
Get image_view
ssh <Robot_name_space>@
plug in the USB camera
On slave, do lsusb and ls /dev |grep video to check if camera was recognized by system
On slave, install usb_cam ROS node sudo apt install ros-kinetic-usb-cam
On slave, check the usb camera launch file cat /opt/ros/kinetic/share/usb_cam/launch/usb_cam-test.launch
(Optional) On local client machine (master machine), run roscore (Usually it's constantly running on the desktop of Robotics Lab so you won't need to do this line)
On slave, start usb_cam node roslaunch usb_cam usb_cam-test.launch
(Optional) On slave, bring running process to background with CTRL+Z and execute bg command to continue execution it in background
(Optional) On slave, check the topic of usb camera rostopic list
(Optional) On master, check the topics in GUI rqt_graph
On master, read camera data with image_view rosrun image_view image_view image:=/<name_space>/usb_cam/image_raw
On slave, to bring background task to foreground fg
Web Streaming
On slave, install web-video-server ROS node sudo apt install ros-kinetic-web-video-server
On slave, to make it right, create catkin workspace for our custom launch file mkdir -p ~/rosvid_ws/src
On slave,cd ~/rosvid_ws
On slave, catkin_make
On slave, source devel/setup.bash
On slave, create ROS package cd src then catkin_create_pkg vidsrv std_msgs rospy roscpp
On slave, create launch file using nano, vim, etc mkdir -p vidsrv/launch then nano vidsrv/launch/vidsrv.launch. Then copy and paste the code below
https://github.com/campusrover/Perception_CV/blob/master/utils/vidsrv.launch
On slave, build package cd.. then catkin_make
On master, Make sure roscore is running
On slave, run created launch file roslaunch vidsrv vidsrv.launch
On your client machine, open web browser and go to <Robot IP address>:8080 . Under /usb_cam/ categoryand and click image_raw .
Enjoy the web streaming
The https://github.com/ultralytics/yolov3 repo contains inference and training code for YOLOv3 in PyTorch. The code works on Linux, MacOS and Windows. Training is done on the COCO dataset by default: https://cocodataset.org/#home. Credit to Joseph Redmon for YOLO: https://pjreddie.com/darknet/yolo/.
Python 3.7 or later with the following pip3 install -U -r requirements.txt packages:
numpy
torch >= 1.0.0
opencv-python
tqdm
Start Training: Run train.py to begin training after downloading COCO data with data/get_coco_dataset.sh.
Resume Training: Run train.py --resume resumes training from the latest checkpoint weights/latest.pt.
Each epoch trains on 117,263 images from the train and validate COCO sets, and tests on 5000 images from the COCO validate set. Default training settings produce loss plots below, with training speed of 0.6 s/batch on a 1080 Ti (18 epochs/day) or 0.45 s/batch on a 2080 Ti.
Here we see training results from coco_1img.data, coco_10img.data and coco_100img.data, 3 example files available in the data/ folder, which train and test on the first 1, 10 and 100 images of the coco2014 trainval dataset.
datasets.py applies random OpenCV-powered (https://opencv.org/) augmentation to the input images in accordance with the following specifications. Augmentation is applied only during training, not during inference. Bounding boxes are automatically tracked and updated with the images. 416 x 416 examples pictured below.
Translation
+/- 10% (vertical and horizontal)
Rotation
+/- 5 degrees
Shear
+/- 2 degrees (vertical and horizontal)
Scale
+/- 10%
Reflection
50% probability (horizontal-only)
HSV Saturation
+/- 50%
HSV Intensity
+/- 50%
https://cloud.google.com/deep-learning-vm/ Machine type: n1-standard-8 (8 vCPUs, 30 GB memory) CPU platform: Intel Skylake GPUs: K80 ($0.198/hr), P4 ($0.279/hr), T4 ($0.353/hr), P100 ($0.493/hr), V100 ($0.803/hr) HDD: 100 GB SSD Dataset: COCO train 2014
GPUs
batch_size
batch time
epoch time
epoch cost
(images)
(s/batch)
1 K80
16
1.43s
175min
$0.58
1 P4
8
0.51s
125min
$0.58
1 T4
16
0.78s
94min
$0.55
1 P100
16
0.39s
48min
$0.39
2 P100
32
0.48s
29min
$0.47
4 P100
64
0.65s
20min
$0.65
1 V100
16
0.25s
31min
$0.41
2 V100
32
0.29s
18min
$0.48
4 V100
64
0.41s
13min
$0.70
8 V100
128
0.49s
7min
$0.80
Run detect.py to apply trained weights to an image, such as zidane.jpg from the data/samples folder:
Run detect.py with webcam=True to show a live webcam feed.
Darknet *.weights format: https://pjreddie.com/media/files/yolov3.weights
PyTorch *.pt format: https://drive.google.com/drive/folders/1uxgUBemJVw9wZsdpboYbzUN4bcRhsuAI
Use test.py --weights weights/yolov3.weights to test the official YOLOv3 weights.
Use test.py --weights weights/latest.pt to test the latest training results.
Compare to darknet published results https://arxiv.org/abs/1804.02767.
YOLOv3 320
51.8
51.5
YOLOv3 416
55.4
55.3
YOLOv3 608
58.2
57.9
YOLOv3-spp 320
52.4
-
YOLOv3-spp 416
56.5
-
YOLOv3-spp 608
60.7
60.6
Issues should be raised directly in the repository. For additional questions or comments please contact your CV Team Leader or Sibo Zhu at siboz1995@gmail.com
from utils import utils; utils.plot_results()
YOLOv3: python3 detect.py --cfg cfg/yolov3.cfg --weights weights/yolov3.weights
YOLOv3-tiny: python3 detect.py --cfg cfg/yolov3-tiny.cfg --weights weights/yolov3-tiny.weights
YOLOv3-SPP: python3 detect.py --cfg cfg/yolov3-spp.cfg --weights weights/yolov3-spp.weights
