Workshop 10 ‐ 3D Vision

Task 1 - World-to-image projections

• Launch the simulator with the default environment: ros2 launch limo_gazebosim limo_gazebo_diff.launch.py.

• Run the image_projection_1 node (ros2 run cmp9767_tutorial image_projection_1) which projects a 3D point from the (front) camera to image coordinates. Inspect the code and experiment with different 3d point positions. What camera coordinates (x,y) correspond to the image borders of the camera sensor, assuming we are operating on the plane 1 m ahead of the camera (z=1.0)?

• Run the image_projection_2 node (ros2 run cmp9767_tutorial image_projection_1). The node visualizes the coordinates of a ground point 5 m in front of the robot specified in the robot frame (base_link) transformed into the camera frame (depth_frame). Place the robot outside the enclosure and find the distance value in front of the camera which will mark exactly the horizon line.

Task 2 - Object detection in 3D

The repository contains a new node detector_3d.py which demonstrates how to detect colour objects in images and then perform image -> camera -> global coordinates transformations. The node subscribes to colour and depth images and utilises the camera_info topic which includes camera geometry information (focal point, resolution, etc.). The colour detector is very similar to the basic one used in previous workshops. The additional functionality includes transforming the image coordinates to a camera frame and subsequently from a camera frame to a global frame. Study the file first so that you familiarise yourself with the basic workflow. Ask for questions if something is not quite clear.

• Now, launch the simulator with the default environment: ros2 launch limo_gazebosim limo_gazebo_diff.launch.py.
• Insert a coloured red object in Gazebo and place the object in front of the robot. You might need to move the robot and the ball manually into a location with more free space around.
• Run the detector_3d node by issuing ros2 run cmp9767_tutorial detector_3d. You should see the debug windows visualising the image processing pipeline with colour and depth images. The node outputs the detected objects in 3d as the /limo/object_location topic.
• Move the colour object around and note its position in Gazebo (right-click on the object and inspect its pose property in the left panel of the simulator) and compare to the output of the detector. Why is there a discrepancy between the calculated location and values read from the simulator?

Task 3 - Object localisation in a map

• Change the detector so that the global_frame corresponds now to the map frame.
  • You may want to create your own copy inside the main folder of your own ROS package (the one you will use for your submission).
• In addition to the simulator, run the navigation node together with rviz visualisation.
• Set up the simulation as above with the colour object in front of the robot.
• Run the updated detector, and add /object_location topic to rviz.
• Note the calculated object location in relation to its real location.
• Try the setup out with multiple objects of the same colour.

Task 4 - Real robot

• Try the detector on the real robot with real colour objects. You will need to set the real_robot variable to True and adjust the ranges of the colour filters. Try it out with the odometry first and then use map-based navigation.