Hammerhead ZMQ

A comprehensive Python and C++ client library for interfacing with the Hammerhead stereo vision system via ZeroMQ

Table of Contents

• Overview
• Quick Start
• Message Types & Ports
• Project Structure
• Examples & Tutorials
• 3D Coordinate System & Point Cloud Conversion
• API Reference
• Best Practices & Tips

Overview

The Hammerhead system is a high-performance stereo vision processing unit that publishes various types of data over ZeroMQ. This library provides easy-to-use APIs for receiving and processing:

• Stereo Images - Raw and rectified left/right camera feeds
• Depth Data - Disparity maps and color-blended depth images
• Point Clouds - 3D point cloud data with or without RGB information
• Obstacle Detection - Real-time obstacle data with bounding boxes
• Camera Control - Parameter adjustment and scheduling

Key Features

Quick Start

Repository Setup

# Get the Hammerhead ZMQ repository
git clone git@github.com:nodarhub/hammerhead_zmq.git

# If you received the HDK with version X.X.X, you can check out the corresponding tag (skip this step if you want the latest version):
git checkout X.X.X

# Make sure that the submodules are up to date
git submodule update --init --recursive

Python Installation & Usage

# Make a virtual environment (here we use ~/venvs/nodarenv, but you can choose any location)
mkdir -p ~/venvs && cd ~/venvs
python3 -m venv nodarenv

# Source the enviroment and install the package (from the root of the repository)
source ~/venvs/nodarenv/bin/activate
pip install -e .

# View live left raw image from Hammerhead device at 10.10.1.10
python examples/python/image_viewer/image_viewer/image_viewer.py 10.10.1.10 nodar/left/image_raw

# Record point clouds to PLY files from Hammerhead device at 10.10.1.10
python examples/python/point_cloud_recorder/point_cloud_recorder/point_cloud_recorder.py 10.10.1.10

# Record obstacle detection data from Hammerhead device at 10.10.1.10
python examples/python/obstacle_data_recorder/obstacle_data_recorder/obstacle_data_recorder.py 10.10.1.10

C++ Installation & Usage

Installing dependencies

Ubuntu:

# Install build tools
sudo apt install build-essential cmake

# Install OpenCV (optional but recommended)
sudo apt install libopencv-dev

Windows:

• Visual Studio Community - Download
• CMake 3.11+ - Download
• OpenCV 4.6.0+ - Download

Building the Examples & Usage

# Build all examples
mkdir build && cd build
cmake .. && cmake --build . --config Release

# View live left raw image from Hammerhead device at 10.10.1.10
./examples/cpp/image_viewer/image_viewer 10.10.1.10 nodar/left/image_raw

# Record point clouds to PLY files from Hammerhead device at 10.10.1.10
./examples/cpp/point_cloud_recorder/point_cloud_recorder 10.10.1.10

# Record obstacle detection data from Hammerhead device at 10.10.1.10
./examples/cpp/obstacle_data_recorder/obstacle_data_recorder 10.10.1.10

Message Types & Ports

Hammerhead publishes data using structured message types over predefined ZMQ ports:

Image Streams

3D Data Streams

Detection & Control

Project Structure

The zmq_msgs folder contains the code for the zmq_msgs target, which defines how objects are sent and received via ZeroMQ on the network. It also defines other important networking information, such as which ports are used for which topics.

The examples folder contains comprehensive examples that demonstrate how to interact with Hammerhead. We envision that you will use these examples as a jumping-off point for your application.

We suggest that you start by examining the code and README's in the individual example directories for more details about what each example does.

Examples & Tutorials

🐍 Python Examples

Python examples provide easy-to-use scripts for common Hammerhead integration tasks.

Visualization Examples

• Image Viewer - Real-time OpenCV viewer for stereo images, disparity maps, and depth data

Data Capture Examples

• Image Recorder - Record images from any Hammerhead stream to disk as TIFF files
• Point Cloud Recorder - Subscribe to point cloud messages and save them as PLY files
• Point Cloud Soup Recorder - Stream the reduced-bandwidth PointCloudSoup messages, convert to point clouds, and save as PLY files
• Obstacle Data Recorder - Record real-time obstacle detection data

Processing Examples

• Depth to Disparity Converter - Convert depth images to disparity format
• Disparity to Point Cloud - Convert stored disparity images to ordered 3D point clouds

Control Examples

• Hammerhead Scheduler - Control Hammerhead's processing schedule
• Topbot Publisher - Publish images to Hammerhead's top/bottom camera topic

⚡ C++ Examples

High-performance C++ implementations for real-time applications and system integration.

Visualization Examples

• Image Viewer - Real-time OpenCV viewer for stereo images, disparity maps, and depth data

Data Capture Examples

• Image Recorder - Record images from any Hammerhead stream to disk as TIFF files
• Point Cloud Recorder - Subscribe to point cloud messages and save them as PLY files
• Point Cloud Soup Recorder - Stream the reduced-bandwidth PointCloudSoup messages, convert to point clouds, and save as PLY files
• Obstacle Data Recorder - Record real-time obstacle detection data

Processing Examples

• Offline Point Cloud Generator - Batch processing of disparity images
• Depth to Disparity Converter - Convert depth images to disparity format
• Legacy Obstacle Data Converter - Convert legacy obstacle data formats

Control Examples

• Hammerhead Scheduler - Control Hammerhead's processing schedule
• Camera Parameter Control - Real-time camera parameter adjustment
• Topbot Publisher - Publish images to Hammerhead's top/bottom camera topic

Common Integration Workflows

🎥 Image Processing Pipeline

1. Start with Image Viewer to verify camera feeds
2. Use Image Recorder to capture datasets
3. Process images with custom algorithms

🌐 3D Reconstruction Workflow

1. Subscribe to PointCloudSoup messages to reduce network bandwidth
2. Reconstruct point clouds
3. Process images with custom algorithms
4. Integrate with 3D processing frameworks

🚗 Obstacle Detection Integration

1. Use Obstacle Data Recorder to understand data format
2. Implement real-time processing of obstacle messages
3. Integrate with path planning or control systems
4. Add custom filtering or tracking algorithms

3D Coordinate System & Point Cloud Conversion

Hammerhead follows standard stereo reconstruction principles for converting disparity to 3D point clouds:

Disparity Scaling

The disparity is in Q12.4 format. We scale the disparity by 1 / 16.0 to get the disparity in float32 format:

disparity_scaled = disparity.astype(np.float32) / 16.0

3D Reprojection

The scaled disparity map is reprojected into 3D space using OpenCV's cv2.reprojectImageTo3D() and a 4×4 reprojection matrix Q:

# Important: Negate the last row for correct coordinate frame
Q_corrected = Q.copy()
Q_corrected[3, :] = -Q_corrected[3, :]

# Reproject to 3D
points_3d = cv2.reprojectImageTo3D(disparity_scaled, Q_corrected)

A negative translation vector (Tx < 0) is used when creating the Q matrix to conform to the definition in OpenCV. This ensures that the point cloud is generated in a consistent right-handed coordinate frame. As a result, the entire last row of Q must be negated before passing to the cv2.reprojectImageTo3D() call.

This conversion scheme has been used in the following examples:

• Offline Point Cloud Generator - C++ batch processing
• Point Cloud Soup Recorder - C++ real-time recording
• Point Cloud Soup Recorder - Python real-time recording
• Disparity to Point Cloud - Python offline processing

API Reference

Message Types

All Hammerhead messages use a versioned protocol with the MessageInfo header structure.

StampedImage

Used for all image data including raw stereo images, rectified images, and disparity maps.

from zmq_msgs.image import StampedImage
import zmq

# Create ZMQ subscriber
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect(f"tcp://{ip_address}:{port}")
socket.setsockopt(zmq.SUBSCRIBE, b"")

# Receive and decode image
buffer = socket.recv()
stamped_image = StampedImage()
stamped_image.read(buffer)

# Access image data
cv2.imshow("Image", stamped_image.img)
print(f"Frame ID: {stamped_image.frame_id}")
print(f"Timestamp: {stamped_image.time}")

ObstacleData

Contains real-time obstacle detection information with bounding boxes and velocity vectors.

from zmq_msgs.obstacle_data import ObstacleData
import zmq

# Create ZMQ subscriber
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect(f"tcp://{ip_address}:9812")
socket.setsockopt(zmq.SUBSCRIBE, b"")

# Receive and decode obstacle data
buffer = socket.recv()
obstacle_data = ObstacleData()
obstacle_data.read(buffer)

# Process obstacles
for obstacle in obstacle_data.obstacles:
    print(f"Bounding box: {obstacle.bounding_box.points}")
    print(f"Velocity: ({obstacle.velocity.x:.2f}, {obstacle.velocity.z:.2f}) m/s")

Coordinate System: Obstacle data is represented in the XZ plane (bird's eye view):

• X axis: Left/right relative to camera
• Z axis: Forward/backward from camera
• No Y component: Height information not included

Best Practices & Tips

🔧 Performance

• Use C++ for real-time applications
• Monitor network bandwidth with point clouds
• Implement connection timeouts
• Consider message buffering for high-frequency data

🛡️ Reliability

• Always validate message types and versions
• Implement automatic reconnection logic
• Handle partial message reception
• Add logging for debugging network issues

🌐 Networking

• Test with different network conditions
• Use appropriate QoS settings if available
• Monitor for packet loss
• Consider compression for bandwidth-limited scenarios

🔍 Debugging

• Start with simple viewers before custom code
• Check Hammerhead configuration files
• Verify port availability and firewall settings
• Use network monitoring tools