Chapter 9: Isaac ROS & VSLAM

Building on Isaac Sim from Chapter 8, we now explore NVIDIA's GPU-accelerated perception stack. Isaac ROS brings the power of NVIDIA GPUs to robot perception, enabling real-time Visual SLAM, 3D reconstruction, and AI-powered sensing that runs orders of magnitude faster than CPU alternatives.

Learning Objectives

By the end of this chapter, you will be able to:

Explain the Isaac ROS architecture and its advantages
Set up Isaac ROS packages in a ROS 2 workspace
Implement Visual SLAM using cuVSLAM for robot localization
Build 3D maps in real-time using nvblox
Integrate depth cameras and stereo vision with Isaac ROS
Optimize perception pipelines for GPU acceleration
Test SLAM systems in Isaac Sim before real-world deployment

Introduction to Isaac ROS

What is Isaac ROS?

Isaac ROS is NVIDIA's collection of GPU-accelerated ROS 2 packages designed for high-performance robot perception:

┌─────────────────────────────────────────────────────────────┐
│                    Isaac ROS Ecosystem                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │                  Isaac ROS Packages                  │   │
│  ├─────────────────────────────────────────────────────┤   │
│  │                                                       │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │   │
│  │  │  cuVSLAM    │  │   nvblox    │  │  DNN        │  │   │
│  │  │ Visual SLAM │  │  3D Mapping │  │  Inference  │  │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  │   │
│  │                                                       │   │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │   │
│  │  │  AprilTag   │  │   Stereo    │  │  Freespace  │  │   │
│  │  │  Detection  │  │   Depth     │  │  Segmentat. │  │   │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  │   │
│  │                                                       │   │
│  └──────────────────────────┬──────────────────────────┘   │
│                             │                               │
│  ┌──────────────────────────▼──────────────────────────┐   │
│  │              NITROS (NVIDIA ROS)                     │   │
│  │         Zero-copy GPU memory transport               │   │
│  └──────────────────────────┬──────────────────────────┘   │
│                             │                               │
│  ┌──────────────────────────▼──────────────────────────┐   │
│  │                    ROS 2 Humble                      │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Why GPU-Accelerated Perception?

Task	CPU Performance	Isaac ROS (GPU)	Speedup
Visual SLAM	10-15 FPS	60+ FPS	4-6x
3D Reconstruction	2-5 Hz	30+ Hz	6-15x
Stereo Depth	15 FPS	90+ FPS	6x
Object Detection	5-10 FPS	60+ FPS	6-12x
Semantic Segmentation	2-5 FPS	30+ FPS	6-15x

NITROS: Zero-Copy Data Transport

Isaac ROS uses NITROS (NVIDIA ROS) for efficient GPU data handling:

┌─────────────────────────────────────────────────────────────┐
│            Traditional ROS 2 vs NITROS Pipeline              │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Traditional ROS 2:                                         │
│  ┌──────┐    ┌──────┐    ┌──────┐    ┌──────┐             │
│  │Camera│───▶│ CPU  │───▶│ GPU  │───▶│ CPU  │───▶Output   │
│  └──────┘    │Copy  │    │Proc  │    │Copy  │             │
│              └──────┘    └──────┘    └──────┘             │
│                 ▲           │           ▲                  │
│                 └───────────┴───────────┘                  │
│                   Memory copies = SLOW                     │
│                                                             │
│  NITROS (Zero-Copy):                                       │
│  ┌──────┐    ┌──────────────────────────┐                 │
│  │Camera│───▶│      GPU Memory          │───▶Output       │
│  └──────┘    │  (stays on GPU)          │                 │
│              │  cuVSLAM → nvblox → DNN  │                 │
│              └──────────────────────────┘                 │
│                 No memory copies = FAST                    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Setting Up Isaac ROS

System Requirements

Component	Requirement
OS	Ubuntu 22.04
ROS 2	Humble Hawksbill
GPU	NVIDIA GPU with Compute Capability 7.0+
CUDA	11.8 or 12.x
Docker	20.10+ with NVIDIA Container Toolkit

Installation Option 1: Docker (Recommended)

# Clone Isaac ROS Common
mkdir -p ~/workspaces/isaac_ros-dev/src
cd ~/workspaces/isaac_ros-dev/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git

# Clone required packages
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam.git
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox.git
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_pipeline.git

# Launch the Isaac ROS development container
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common
./scripts/run_dev.sh

Installation Option 2: Native Build

# Install dependencies
sudo apt-get update
sudo apt-get install -y \
  ros-humble-isaac-ros-visual-slam \
  ros-humble-isaac-ros-nvblox \
  ros-humble-isaac-ros-image-pipeline

# Or build from source
cd ~/ros2_ws/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam.git
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox.git

cd ~/ros2_ws
colcon build --packages-up-to isaac_ros_visual_slam isaac_ros_nvblox
source install/setup.bash

Verifying Installation

# Check CUDA availability
nvidia-smi

# List Isaac ROS packages
ros2 pkg list | grep isaac

# Check cuVSLAM node
ros2 run isaac_ros_visual_slam isaac_ros_visual_slam_node --help

Understanding Visual SLAM

What is Visual SLAM?

Visual Simultaneous Localization and Mapping (VSLAM) uses camera images to:

Localize: Determine the robot's position and orientation
Map: Build a representation of the environment
Track: Follow visual features across frames

┌─────────────────────────────────────────────────────────────┐
│                    VSLAM Pipeline                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐                                           │
│  │   Camera    │                                           │
│  │   Frames    │                                           │
│  └──────┬──────┘                                           │
│         │                                                   │
│         ▼                                                   │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    │
│  │  Feature    │───▶│   Feature   │───▶│    Pose     │    │
│  │  Detection  │    │   Matching  │    │  Estimation │    │
│  └─────────────┘    └─────────────┘    └──────┬──────┘    │
│         │                                      │            │
│         ▼                                      ▼            │
│  ┌─────────────┐                      ┌─────────────┐      │
│  │    Map      │◀─────────────────────│   Bundle    │      │
│  │   Points    │                      │ Adjustment  │      │
│  └─────────────┘                      └─────────────┘      │
│         │                                      │            │
│         └──────────────────┬───────────────────┘            │
│                            ▼                                │
│                   ┌─────────────┐                          │
│                   │   Output:   │                          │
│                   │ Pose + Map  │                          │
│                   └─────────────┘                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

VSLAM vs Traditional SLAM

Aspect	Lidar SLAM	Visual SLAM
Sensor Cost	$1,000 - $10,000+	$50 - $500
Power Consumption	High	Low
Information Density	Sparse geometry	Rich texture + geometry
Lighting Dependency	None	Requires adequate light
Feature Types	Distance measurements	Visual features
Best For	Outdoor, industrial	Indoor, robotics

Types of Visual SLAM

┌─────────────────────────────────────────────────────────────┐
│                 Visual SLAM Approaches                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Feature-Based SLAM                                   │   │
│  │ • Extracts keypoints (ORB, SIFT, SURF)              │   │
│  │ • Sparse map representation                          │   │
│  │ • Fast, works on CPU                                 │   │
│  │ • Examples: ORB-SLAM, VINS                          │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Direct SLAM                                          │   │
│  │ • Uses pixel intensities directly                    │   │
│  │ • Dense/semi-dense maps                              │   │
│  │ • More accurate, computationally expensive           │   │
│  │ • Examples: LSD-SLAM, DSO                           │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ Deep Learning SLAM (cuVSLAM)                        │   │
│  │ • Learned feature extraction                         │   │
│  │ • GPU-accelerated processing                         │   │
│  │ • Robust to challenging conditions                   │   │
│  │ • Examples: cuVSLAM, DROID-SLAM                     │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

cuVSLAM: GPU-Accelerated Visual SLAM

What is cuVSLAM?

cuVSLAM is NVIDIA's GPU-accelerated Visual SLAM implementation that provides:

Real-time visual odometry at 60+ FPS
Loop closure detection and correction
Multi-camera support (stereo, RGB-D)
IMU fusion for improved accuracy
Relocalization after tracking loss

cuVSLAM Architecture

┌─────────────────────────────────────────────────────────────┐
│                   cuVSLAM Architecture                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Inputs:                                                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                 │
│  │  Stereo  │  │  RGB-D   │  │   IMU    │                 │
│  │  Camera  │  │  Camera  │  │  (opt.)  │                 │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                 │
│       │             │             │                        │
│       └─────────────┼─────────────┘                        │
│                     ▼                                       │
│  ┌──────────────────────────────────────────────────────┐  │
│  │                GPU Processing Pipeline                │  │
│  │  ┌────────────┐  ┌────────────┐  ┌────────────────┐ │  │
│  │  │  Feature   │─▶│   Stereo   │─▶│ Visual Odometry│ │  │
│  │  │ Extraction │  │  Matching  │  │   Estimation   │ │  │
│  │  └────────────┘  └────────────┘  └────────────────┘ │  │
│  │         │                               │            │  │
│  │         ▼                               ▼            │  │
│  │  ┌────────────┐                 ┌────────────────┐  │  │
│  │  │   Loop     │────────────────▶│     Pose       │  │  │
│  │  │  Closure   │                 │    Graph       │  │  │
│  │  └────────────┘                 └────────────────┘  │  │
│  └──────────────────────────────────────────────────────┘  │
│                            │                                │
│                            ▼                                │
│  Outputs:                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐ │
│  │ Visual Odom  │  │  Pose Graph  │  │ Landmark Cloud   │ │
│  │ (30-60 Hz)   │  │  (on demand) │  │  (sparse map)    │ │
│  └──────────────┘  └──────────────┘  └──────────────────┘ │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Launching cuVSLAM

Basic Launch (Stereo Camera)

ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam_realsense.launch.py

Custom Launch File

# File: cuVSLAM_launch.py
from launch import LaunchDescription
from launch_ros.actions import Node
from launch.actions import DeclareLaunchArgument
from launch.substitutions import LaunchConfiguration

def generate_launch_description():
    # Parameters
    enable_imu_fusion = LaunchConfiguration('enable_imu_fusion')
    enable_debug_mode = LaunchConfiguration('enable_debug_mode')

    return LaunchDescription([
        # Arguments
        DeclareLaunchArgument(
            'enable_imu_fusion',
            default_value='false',
            description='Enable IMU sensor fusion'
        ),
        DeclareLaunchArgument(
            'enable_debug_mode',
            default_value='false',
            description='Enable debug visualization'
        ),

        # cuVSLAM Node
        Node(
            package='isaac_ros_visual_slam',
            executable='isaac_ros_visual_slam_node',
            name='visual_slam',
            parameters=[{
                'denoise_input_images': True,
                'rectified_images': True,
                'enable_imu_fusion': enable_imu_fusion,
                'gyro_noise_density': 0.000244,
                'gyro_random_walk': 0.000019393,
                'accel_noise_density': 0.001862,
                'accel_random_walk': 0.003,
                'calibration_frequency': 200.0,
                'image_jitter_threshold_ms': 34.0,
                'enable_debug_mode': enable_debug_mode,
            }],
            remappings=[
                ('stereo_camera/left/image', '/camera/left/image_raw'),
                ('stereo_camera/left/camera_info', '/camera/left/camera_info'),
                ('stereo_camera/right/image', '/camera/right/image_raw'),
                ('stereo_camera/right/camera_info', '/camera/right/camera_info'),
                ('visual_slam/imu', '/imu'),
            ]
        ),
    ])

cuVSLAM Topics and Services

# Published Topics
/visual_slam/tracking/odometry          # nav_msgs/Odometry (high-rate pose)
/visual_slam/tracking/vo_pose           # geometry_msgs/PoseStamped
/visual_slam/tracking/vo_pose_covariance # Pose with uncertainty
/visual_slam/vis/landmarks_cloud        # sensor_msgs/PointCloud2 (sparse map)
/visual_slam/vis/observations_cloud     # Current observations

# Services
/visual_slam/reset                      # Reset SLAM system
/visual_slam/save_map                   # Save current map
/visual_slam/load_map                   # Load existing map

# Example: Save map
ros2 service call /visual_slam/save_map isaac_ros_visual_slam_interfaces/srv/SaveMap \
  "{map_url: '/tmp/my_slam_map'}"

Integrating with Robot Localization

# Fuse cuVSLAM with wheel odometry using robot_localization
from launch import LaunchDescription
from launch_ros.actions import Node

def generate_launch_description():
    return LaunchDescription([
        # EKF for sensor fusion
        Node(
            package='robot_localization',
            executable='ekf_node',
            name='ekf_filter_node',
            parameters=[{
                'frequency': 50.0,
                'sensor_timeout': 0.1,
                'two_d_mode': False,
                'odom_frame': 'odom',
                'base_link_frame': 'base_link',
                'world_frame': 'odom',

                # Wheel odometry
                'odom0': '/wheel_odom',
                'odom0_config': [
                    True, True, False,   # x, y, z
                    False, False, True,  # roll, pitch, yaw
                    True, True, False,   # vx, vy, vz
                    False, False, True,  # vroll, vpitch, vyaw
                    False, False, False  # ax, ay, az
                ],

                # Visual SLAM odometry
                'odom1': '/visual_slam/tracking/odometry',
                'odom1_config': [
                    True, True, True,    # x, y, z
                    True, True, True,    # roll, pitch, yaw
                    False, False, False, # vx, vy, vz
                    False, False, False, # vroll, vpitch, vyaw
                    False, False, False  # ax, ay, az
                ],
            }]
        ),
    ])

nvblox: Real-Time 3D Reconstruction

What is nvblox?

nvblox is NVIDIA's GPU-accelerated 3D reconstruction library that creates:

Signed Distance Fields (SDF): Continuous surface representation
Occupancy Maps: For navigation and collision avoidance
Mesh Reconstruction: Real-time surface meshes
ESDF (Euclidean Signed Distance Field): For path planning

┌─────────────────────────────────────────────────────────────┐
│                   nvblox Pipeline                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────┐    ┌─────────────┐                        │
│  │ Depth Image │    │  Robot Pose │                        │
│  │  (Camera)   │    │  (cuVSLAM)  │                        │
│  └──────┬──────┘    └──────┬──────┘                        │
│         │                  │                                │
│         └────────┬─────────┘                                │
│                  ▼                                          │
│  ┌──────────────────────────────────────────────────────┐  │
│  │               nvblox GPU Pipeline                     │  │
│  │                                                       │  │
│  │  ┌─────────────┐   ┌─────────────┐   ┌───────────┐  │  │
│  │  │   Depth     │──▶│   TSDF      │──▶│   Mesh    │  │  │
│  │  │ Integration │   │  Voxel Grid │   │ Extraction│  │  │
│  │  └─────────────┘   └─────────────┘   └───────────┘  │  │
│  │         │                │                           │  │
│  │         ▼                ▼                           │  │
│  │  ┌─────────────┐   ┌─────────────┐                  │  │
│  │  │ Occupancy   │   │    ESDF     │                  │  │
│  │  │    Grid     │   │ (Planning)  │                  │  │
│  │  └─────────────┘   └─────────────┘                  │  │
│  │                                                       │  │
│  └──────────────────────────────────────────────────────┘  │
│                            │                                │
│                            ▼                                │
│  Outputs:                                                   │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐           │
│  │ 3D Mesh    │  │ Costmap    │  │ ESDF for   │           │
│  │ (visual)   │  │ (Nav2)     │  │ Planning   │           │
│  └────────────┘  └────────────┘  └────────────┘           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

nvblox Concepts

TSDF (Truncated Signed Distance Field)

┌─────────────────────────────────────────────────────────────┐
│                    TSDF Representation                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Surface                                                   │
│      │                                                      │
│      ▼                                                      │
│  ────────────────────────────────────                       │
│      │◀──────────────────────────▶│                        │
│      │     Truncation Distance     │                        │
│      │                             │                        │
│  ┌───┴───┐                     ┌───┴───┐                   │
│  │ -1.0  │  Negative values    │ +1.0  │  Positive values  │
│  │ Inside│  (inside surface)   │Outside│  (outside surface)│
│  └───────┘                     └───────┘                   │
│                                                             │
│   Voxel value = signed distance to nearest surface         │
│   Zero-crossing = surface location                          │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Launching nvblox

Basic Launch

# With RealSense camera
ros2 launch nvblox_examples_bringup realsense_example.launch.py

# With Isaac Sim
ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py

Custom Configuration

# File: nvblox_mapping.launch.py
from launch import LaunchDescription
from launch_ros.actions import Node

def generate_launch_description():
    return LaunchDescription([
        Node(
            package='nvblox_ros',
            executable='nvblox_node',
            name='nvblox',
            parameters=[{
                # Voxel parameters
                'voxel_size': 0.05,  # 5cm voxels
                'esdf_update_rate_hz': 10.0,
                'mesh_update_rate_hz': 5.0,

                # Map parameters
                'global_frame': 'odom',
                'map_clearing_radius_m': 5.0,

                # TSDF parameters
                'tsdf_integrator_max_integration_distance_m': 5.0,
                'tsdf_integrator_truncation_distance_vox': 4.0,

                # Mesh parameters
                'mesh_integrator_min_weight': 0.0001,
                'mesh_integrator_weld_vertices': True,

                # ESDF parameters
                'esdf_integrator_max_distance_m': 2.0,
                'esdf_integrator_min_weight': 0.0001,

                # Occupancy parameters
                'occupancy_integrator_free_region_occupancy_probability': 0.3,
                'occupancy_integrator_occupied_region_occupancy_probability': 0.7,
            }],
            remappings=[
                ('depth/image', '/camera/depth/image_rect_raw'),
                ('depth/camera_info', '/camera/depth/camera_info'),
                ('color/image', '/camera/color/image_raw'),
                ('color/camera_info', '/camera/color/camera_info'),
            ]
        ),
    ])

nvblox Topics

# Published Topics
/nvblox_node/mesh              # nvblox_msgs/Mesh (3D reconstruction)
/nvblox_node/static_occupancy  # nav_msgs/OccupancyGrid (for Nav2)
/nvblox_node/esdf_pointcloud   # sensor_msgs/PointCloud2 (ESDF visualization)
/nvblox_node/mesh_marker       # visualization_msgs/Marker (RViz mesh)
/nvblox_node/slice_bounds      # Bounding box of map slice

# Subscribed Topics
/depth/image                   # sensor_msgs/Image (depth)
/depth/camera_info            # sensor_msgs/CameraInfo
/color/image                  # sensor_msgs/Image (RGB, optional)
/tf                           # Transform tree

Visualizing nvblox Output

# RViz2 configuration for nvblox
# Add these displays:
# 1. Marker - topic: /nvblox_node/mesh_marker
# 2. OccupancyGrid - topic: /nvblox_node/static_occupancy
# 3. PointCloud2 - topic: /nvblox_node/esdf_pointcloud

ros2 run rviz2 rviz2 -d $(ros2 pkg prefix nvblox_rviz_plugin)/share/nvblox_rviz_plugin/rviz/nvblox.rviz

Complete VSLAM + Mapping Pipeline

System Architecture

┌─────────────────────────────────────────────────────────────┐
│           Complete Isaac ROS Perception Pipeline             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │                     Sensors                          │   │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌───────┐  │   │
│  │  │ Stereo  │  │  RGB-D  │  │  IMU    │  │ Wheel │  │   │
│  │  │ Camera  │  │ Camera  │  │         │  │ Odom  │  │   │
│  │  └────┬────┘  └────┬────┘  └────┬────┘  └───┬───┘  │   │
│  └───────┼────────────┼────────────┼───────────┼──────┘   │
│          │            │            │           │           │
│          └────────────┼────────────┴───────────┘           │
│                       │                                     │
│  ┌────────────────────▼────────────────────────────────┐   │
│  │                   cuVSLAM                            │   │
│  │              Visual Odometry + SLAM                  │   │
│  └────────────────────┬────────────────────────────────┘   │
│                       │                                     │
│          ┌────────────┼────────────┐                       │
│          ▼            ▼            ▼                       │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐              │
│  │   Pose    │  │  Sparse   │  │   Loop    │              │
│  │ Estimate  │  │    Map    │  │  Closures │              │
│  └─────┬─────┘  └───────────┘  └───────────┘              │
│        │                                                   │
│        ▼                                                   │
│  ┌────────────────────────────────────────────────────┐   │
│  │                    nvblox                           │   │
│  │         3D Reconstruction + Occupancy Grid          │   │
│  └────────────────────────────────────────────────────┘   │
│        │                    │                              │
│        ▼                    ▼                              │
│  ┌───────────┐        ┌───────────┐                       │
│  │ 3D Mesh   │        │ Costmap   │                       │
│  │           │        │ (Nav2)    │                       │
│  └───────────┘        └─────┬─────┘                       │
│                             │                              │
│                             ▼                              │
│                      ┌───────────┐                        │
│                      │   Nav2    │                        │
│                      │ Planning  │                        │
│                      └───────────┘                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Complete Launch File

# File: full_perception_pipeline.launch.py
from launch import LaunchDescription
from launch_ros.actions import Node
from launch.actions import IncludeLaunchDescription
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch_ros.substitutions import FindPackageShare
import os

def generate_launch_description():
    # Package paths
    isaac_ros_visual_slam_dir = FindPackageShare('isaac_ros_visual_slam')
    nvblox_dir = FindPackageShare('nvblox_ros')

    return LaunchDescription([
        # cuVSLAM Node
        Node(
            package='isaac_ros_visual_slam',
            executable='isaac_ros_visual_slam_node',
            name='visual_slam',
            parameters=[{
                'denoise_input_images': True,
                'rectified_images': True,
                'enable_imu_fusion': True,
                'enable_debug_mode': False,
                'map_frame': 'map',
                'odom_frame': 'odom',
                'base_frame': 'base_link',
            }],
            remappings=[
                ('stereo_camera/left/image', '/camera/infra1/image_rect_raw'),
                ('stereo_camera/left/camera_info', '/camera/infra1/camera_info'),
                ('stereo_camera/right/image', '/camera/infra2/image_rect_raw'),
                ('stereo_camera/right/camera_info', '/camera/infra2/camera_info'),
                ('visual_slam/imu', '/camera/imu'),
            ]
        ),

        # nvblox Node
        Node(
            package='nvblox_ros',
            executable='nvblox_node',
            name='nvblox',
            parameters=[{
                'voxel_size': 0.05,
                'global_frame': 'odom',
                'esdf_update_rate_hz': 10.0,
                'mesh_update_rate_hz': 5.0,
            }],
            remappings=[
                ('depth/image', '/camera/depth/image_rect_raw'),
                ('depth/camera_info', '/camera/depth/camera_info'),
                ('color/image', '/camera/color/image_raw'),
                ('color/camera_info', '/camera/color/camera_info'),
            ]
        ),

        # Static Transform: base_link to camera
        Node(
            package='tf2_ros',
            executable='static_transform_publisher',
            arguments=[
                '0.1', '0', '0.2',  # x, y, z
                '0', '0', '0', '1',  # qx, qy, qz, qw
                'base_link', 'camera_link'
            ]
        ),
    ])

Testing in Isaac Sim

Setting Up Isaac Sim for VSLAM Testing

# File: isaac_sim_vslam_test.py
from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": False})

from omni.isaac.core import World
from omni.isaac.core.utils.extensions import enable_extension
from omni.isaac.sensor import Camera
from omni.isaac.ros2_bridge import CameraROS2Publisher
import numpy as np

# Enable ROS 2 bridge
enable_extension("omni.isaac.ros2_bridge")

# Create world
world = World(stage_units_in_meters=1.0)
world.scene.add_default_ground_plane()

# Create stereo camera rig
left_camera = Camera(
    prim_path="/World/Robot/StereoRig/LeftCamera",
    position=np.array([0.0, 0.05, 0.3]),  # 10cm baseline
    frequency=30,
    resolution=(640, 480)
)

right_camera = Camera(
    prim_path="/World/Robot/StereoRig/RightCamera",
    position=np.array([0.0, -0.05, 0.3]),
    frequency=30,
    resolution=(640, 480)
)

# Create depth camera for nvblox
depth_camera = Camera(
    prim_path="/World/Robot/DepthCamera",
    position=np.array([0.1, 0.0, 0.3]),
    frequency=30,
    resolution=(640, 480)
)

world.scene.add(left_camera)
world.scene.add(right_camera)
world.scene.add(depth_camera)

# Create ROS 2 publishers
left_pub = CameraROS2Publisher(left_camera, "camera_left", "/camera/infra1/image_rect_raw")
right_pub = CameraROS2Publisher(right_camera, "camera_right", "/camera/infra2/image_rect_raw")
depth_pub = CameraROS2Publisher(depth_camera, "camera_depth", "/camera/depth/image_rect_raw")

# Simulation loop
world.reset()
while simulation_app.is_running():
    world.step(render=True)

simulation_app.close()

Creating Test Environments

def create_slam_test_environment(world):
    """Create an environment with features for VSLAM testing."""
    from omni.isaac.core.objects import VisualCuboid, DynamicCuboid

    # Textured walls (important for visual features)
    wall_configs = [
        {"pos": [0, 5, 1.5], "scale": [10, 0.1, 3], "color": [0.8, 0.2, 0.2]},
        {"pos": [0, -5, 1.5], "scale": [10, 0.1, 3], "color": [0.2, 0.8, 0.2]},
        {"pos": [5, 0, 1.5], "scale": [0.1, 10, 3], "color": [0.2, 0.2, 0.8]},
        {"pos": [-5, 0, 1.5], "scale": [0.1, 10, 3], "color": [0.8, 0.8, 0.2]},
    ]

    for i, cfg in enumerate(wall_configs):
        world.scene.add(
            VisualCuboid(
                prim_path=f"/World/Walls/Wall_{i}",
                name=f"wall_{i}",
                position=np.array(cfg["pos"]),
                scale=np.array(cfg["scale"]),
                color=np.array(cfg["color"])
            )
        )

    # Add distinctive objects for loop closure
    landmarks = [
        {"pos": [2, 2, 0.5], "scale": [0.5, 0.5, 1], "color": [1, 0, 0]},
        {"pos": [-2, 2, 0.5], "scale": [0.5, 0.5, 1], "color": [0, 1, 0]},
        {"pos": [2, -2, 0.5], "scale": [0.5, 0.5, 1], "color": [0, 0, 1]},
        {"pos": [-2, -2, 0.5], "scale": [0.5, 0.5, 1], "color": [1, 1, 0]},
    ]

    for i, lm in enumerate(landmarks):
        world.scene.add(
            VisualCuboid(
                prim_path=f"/World/Landmarks/Landmark_{i}",
                name=f"landmark_{i}",
                position=np.array(lm["pos"]),
                scale=np.array(lm["scale"]),
                color=np.array(lm["color"])
            )
        )

Performance Optimization

Tuning cuVSLAM

# cuVSLAM performance parameters
visual_slam_params:
  # Feature detection
  num_features: 500              # Reduce for speed, increase for accuracy
  feature_threshold: 20          # Lower = more features

  # Tracking
  max_frame_rate: 60             # Match camera FPS
  image_jitter_threshold_ms: 34  # For 30 FPS camera

  # Map management
  max_landmarks: 10000           # Limit for memory
  landmark_culling_threshold: 3  # Remove landmarks seen < N times

  # IMU fusion (if available)
  enable_imu_fusion: true
  imu_integration_sigma: 0.01

Tuning nvblox

# nvblox performance parameters
nvblox_params:
  # Resolution trade-offs
  voxel_size: 0.05               # Larger = faster, less detail

  # Update rates
  esdf_update_rate_hz: 10.0      # Reduce for lower GPU load
  mesh_update_rate_hz: 5.0       # Reduce for lower GPU load

  # Integration limits
  tsdf_integrator_max_integration_distance_m: 5.0  # Limit range
  max_integration_images_per_sec: 30               # Throttle input

  # Memory management
  map_clearing_radius_m: 10.0    # Clear distant voxels
  max_blocks: 100000             # Limit memory usage

Monitoring Performance

# Monitor GPU usage
watch -n 1 nvidia-smi

# Monitor topic rates
ros2 topic hz /visual_slam/tracking/odometry
ros2 topic hz /nvblox_node/mesh

# Profile with ROS 2
ros2 run ros2_tracing trace --session my_session

Practical Exercise: Building a SLAM Pipeline

Goal

Create a complete perception system that:

Runs cuVSLAM for localization
Builds 3D maps with nvblox
Outputs costmaps for navigation
Tested first in Isaac Sim

Step 1: Create the Launch File

# File: perception_pipeline.launch.py
from launch import LaunchDescription
from launch_ros.actions import Node, ComposableNodeContainer
from launch_ros.descriptions import ComposableNode

def generate_launch_description():
    return LaunchDescription([
        # cuVSLAM
        Node(
            package='isaac_ros_visual_slam',
            executable='isaac_ros_visual_slam_node',
            name='visual_slam',
            parameters=[{
                'enable_imu_fusion': False,
                'map_frame': 'map',
                'odom_frame': 'odom',
                'base_frame': 'base_link',
            }]
        ),

        # nvblox
        Node(
            package='nvblox_ros',
            executable='nvblox_node',
            name='nvblox',
            parameters=[{
                'voxel_size': 0.05,
                'global_frame': 'odom',
            }]
        ),

        # Robot State Publisher
        Node(
            package='robot_state_publisher',
            executable='robot_state_publisher',
            parameters=[{'robot_description': '...'}]
        ),
    ])

Step 2: Test Workflow

# Terminal 1: Launch Isaac Sim with robot and cameras
cd ~/isaac_ros_ws
python3 src/my_robot_sim/scripts/isaac_sim_vslam_test.py

# Terminal 2: Launch perception pipeline
ros2 launch my_robot_perception perception_pipeline.launch.py

# Terminal 3: Visualize in RViz2
rviz2 -d config/slam_visualization.rviz

# Terminal 4: Teleoperate the robot
ros2 run teleop_twist_keyboard teleop_twist_keyboard

Step 3: Evaluate SLAM Quality

# Record trajectory
ros2 bag record /visual_slam/tracking/odometry /tf -o slam_test

# Compare to ground truth (if available)
evo_ape tum ground_truth.txt estimated.txt --align --plot

# Check map consistency
ros2 service call /nvblox_node/save_map std_srvs/srv/Empty

Summary

In this chapter, you learned:

Isaac ROS Architecture: GPU-accelerated perception with NITROS zero-copy transport
cuVSLAM: Real-time visual odometry at 60+ FPS with loop closure
nvblox: GPU-accelerated 3D reconstruction with TSDF and ESDF
Sensor Fusion: Combining visual SLAM with IMU and wheel odometry
Integration: Building complete perception pipelines with ROS 2
Testing: Validating SLAM systems in Isaac Sim before deployment
Optimization: Tuning parameters for performance vs accuracy

GPU-accelerated perception fundamentally changes what's possible in real-time robotics, enabling capabilities that were previously too computationally expensive.

Next Week Preview

In Chapter 10, we explore Nav2 Path Planning:

ROS 2 Navigation Stack (Nav2) architecture
Costmap configuration with nvblox
Path planning algorithms (NavFn, Smac, TEB)
Behavior trees for complex navigation
Integration with Isaac ROS perception

Learning Objectives​

Introduction to Isaac ROS​

What is Isaac ROS?​

Why GPU-Accelerated Perception?​

NITROS: Zero-Copy Data Transport​

Setting Up Isaac ROS​

System Requirements​

Installation Option 1: Docker (Recommended)​

Installation Option 2: Native Build​

Verifying Installation​

Understanding Visual SLAM​

What is Visual SLAM?​

VSLAM vs Traditional SLAM​

Types of Visual SLAM​

cuVSLAM: GPU-Accelerated Visual SLAM​

What is cuVSLAM?​

cuVSLAM Architecture​

Launching cuVSLAM​

Basic Launch (Stereo Camera)​

Custom Launch File​

cuVSLAM Topics and Services​

Integrating with Robot Localization​

nvblox: Real-Time 3D Reconstruction​

What is nvblox?​

nvblox Concepts​

TSDF (Truncated Signed Distance Field)​

Launching nvblox​

Basic Launch​

Custom Configuration​

nvblox Topics​

Visualizing nvblox Output​

Complete VSLAM + Mapping Pipeline​

System Architecture​

Complete Launch File​

Testing in Isaac Sim​

Setting Up Isaac Sim for VSLAM Testing​

Creating Test Environments​

Performance Optimization​

Tuning cuVSLAM​

Tuning nvblox​

Monitoring Performance​

Practical Exercise: Building a SLAM Pipeline​

Goal​

Step 1: Create the Launch File​

Step 2: Test Workflow​

Step 3: Evaluate SLAM Quality​

Summary​

Further Reading​

Next Week Preview​