INTRODUCTION TO COMPUTER VISION AT THE EDGE
The revolution in artificial intelligence has fundamentally transformed how machines perceive visual information. Object detection, tracking, and segmentation represent three pillars of modern computer vision, enabling systems to identify objects, follow them through time, and understand their precise spatial boundaries. What makes this particularly exciting in the current technological landscape is the democratization of these capabilities across an unprecedented range of hardware platforms.
This guide provides practical implementation details for building computer vision systems across three distinct hardware tiers. We examine high- performance desktop and server computers equipped with dedicated graphics processing units where computational resources enable complex models to process high-resolution video in real time. We then progress to IoT edge devices such as the NVIDIA Jetson Nano and Raspberry Pi, which balance computational capability with power efficiency and form factor constraints. Finally, we explore the frontier of truly embedded AI on microcontrollers like the ESP32, where extreme resource constraints demand innovative optimization techniques.
Throughout this guide, we emphasize practical implementation with working code examples, real-world deployment considerations, and engineering trade-offs inherent in adapting computer vision systems to different hardware platforms.
PART ONE: OBJECT DETECTION ON DESKTOP COMPUTERS
Understanding YOLO Architecture
Object detection solves the dual challenge of classification and localization. The breakthrough came with single-stage detectors, particularly the YOLO family. YOLO treats object detection as a single regression problem, dividing the input image into a grid where each cell predicts bounding boxes and class probabilities directly.
The network processes the entire image through convolutional layers that extract features at multiple scales. Early layers detect low-level features like edges, while deeper layers identify complex patterns. Each predicted bounding box includes center coordinates, dimensions, and a confidence score. Non-maximum suppression eliminates redundant detections by sorting predictions by confidence and removing overlapping boxes.
Building Your First YOLO Detection System
Let us implement a complete detection pipeline using Python and the Ultralytics YOLO library. First, install the necessary dependencies:
pip install ultralytics opencv-python numpy
Now create the basic detection system:
import cv2
import numpy as np
from ultralytics import YOLO
# Load pre-trained YOLOv11 nano model trained on COCO dataset
# Contains 80 object categories including person, car, dog, etc.
model = YOLO('yolo11n.pt')
# Configure detection parameters
# Confidence threshold: minimum probability to consider detection valid
# IOU threshold: controls non-maximum suppression aggressiveness
confidence_threshold = 0.5
iou_threshold = 0.45
The model weights download automatically on first use. The nano variant balances speed and accuracy, processing images quickly even on CPU. Other variants include small, medium, large, and extra-large models trading inference speed for improved accuracy.
For single image detection, implement the following function:
def detect_objects_in_image(image_path, model, conf_threshold=0.5):
"""
Perform object detection on a single image and return results.
Arguments:
image_path: File path to input image
model: Loaded YOLO model instance
conf_threshold: Minimum confidence for valid detections
Returns:
annotated_image: Image with detection boxes drawn
detections: List of detection dictionaries
"""
# Load image using OpenCV BGR format
image = cv2.imread(image_path)
if image is None:
raise ValueError(f"Cannot read image from {image_path}")
# Execute inference with confidence filtering
results = model(image, conf=conf_threshold, verbose=False)
# Extract first result (single image)
result = results[0]
# Get annotated image with boxes and labels
annotated_image = result.plot()
# Parse detection information
boxes = result.boxes
detections = []
for box in boxes:
# Extract bounding box coordinates in xyxy format
x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
# Get confidence score and class information
confidence = float(box.conf[0])
class_id = int(box.cls[0])
class_name = model.names[class_id]
# Store structured detection information
detection_info = {
'bbox': [int(x1), int(y1), int(x2), int(y2)],
'confidence': confidence,
'class_id': class_id,
'class_name': class_name,
'center_x': int((x1 + x2) / 2),
'center_y': int((y1 + y2) / 2),
'width': int(x2 - x1),
'height': int(y2 - y1)
}
detections.append(detection_info)
return annotated_image, detections
Use this function to process images and display results:
# Load and process image
image_path = 'test_image.jpg'
annotated, detections = detect_objects_in_image(image_path, model)
# Print detection results
print(f"Found {len(detections)} objects:")
for det in detections:
print(f" {det['class_name']}: {det['confidence']:.2f}")
# Display annotated image
cv2.imshow('Detections', annotated)
cv2.waitKey(0)
cv2.destroyAllWindows()
Real-Time Video Detection
Video detection requires continuous frame capture and processing. Implement real-time detection with frame rate monitoring:
def detect_objects_in_video(video_path, model, conf_threshold=0.5):
"""
Perform real-time object detection on video stream.
Arguments:
video_path: Path to video file or camera index (0 for webcam)
model: Loaded YOLO model
conf_threshold: Minimum detection confidence
"""
import time
# Open video source (use 0 for default webcam)
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
raise ValueError(f"Cannot open video source: {video_path}")
# Get video properties
fps = int(cap.get(cv2.CAP_PROP_FPS))
frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
print(f"Video: {frame_width}x{frame_height} at {fps} FPS")
frame_count = 0
start_time = time.time()
while True:
ret, frame = cap.read()
if not ret:
print("End of video or read error")
break
frame_count += 1
# Run detection on current frame
results = model(frame, conf=conf_threshold, verbose=False)
annotated_frame = results[0].plot()
# Calculate and display performance metrics
elapsed = time.time() - start_time
current_fps = frame_count / elapsed if elapsed > 0 else 0
# Add performance information to frame
info_text = f"Frame: {frame_count} | FPS: {current_fps:.1f}"
cv2.putText(annotated_frame, info_text,
(10, 30), cv2.FONT_HERSHEY_SIMPLEX,
0.7, (0, 255, 0), 2)
# Display result
cv2.imshow('YOLO Real-Time Detection', annotated_frame)
# Exit on 'q' key
if cv2.waitKey(1) & 0xFF == ord('q'):
break
# Cleanup and report statistics
cap.release()
cv2.destroyAllWindows()
total_time = time.time() - start_time
avg_fps = frame_count / total_time
print(f"Processed {frame_count} frames in {total_time:.2f}s")
print(f"Average FPS: {avg_fps:.2f}")
This implementation provides frame rate monitoring and graceful resource cleanup. On modern desktop GPUs, the nano model typically achieves over 100 FPS at 640x640 resolution. The medium model balances at 50-70 FPS, while the extra-large model may drop to 20-30 FPS with superior accuracy.
PART TWO: OBJECT TRACKING SYSTEMS
Understanding Multi-Object Tracking
While detection identifies objects frame-by-frame, tracking maintains identities across time. This temporal continuity enables understanding object behavior, counting unique entities, and analyzing trajectories. Multi-object tracking faces challenges including occlusions, appearance changes, and crowded scenes where objects interact closely.
The ByteTrack algorithm represents current state-of-the-art tracking. Its key innovation lies in utilizing both high and low confidence detections. Traditional trackers discard low-confidence detections entirely, losing objects under partial occlusion or motion blur. ByteTrack recovers these objects by associating low-confidence detections with existing tracklets after high-confidence matching.
Implementing ByteTrack with YOLO
The Ultralytics library provides integrated tracking capabilities. Enable tracking with minimal code changes:
def track_objects_in_video(video_path, model_path='yolo11n.pt'):
"""
Multi-object tracking with persistent IDs and trajectory visualization.
Arguments:
video_path: Video file path or camera index
model_path: YOLO model weights file
"""
model = YOLO(model_path)
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
raise ValueError(f"Cannot open video: {video_path}")
# Store trajectory history for each tracked object
# Key: track_id, Value: list of (x, y) center points
track_history = {}
track_colors = {} # Consistent colors per track
frame_count = 0
while True:
ret, frame = cap.read()
if not ret:
break
frame_count += 1
# Run tracking (persist=True maintains IDs across frames)
results = model.track(frame, persist=True, verbose=False)
# Check if tracking results exist
if results[0].boxes is None or results[0].boxes.id is None:
cv2.imshow('Tracking', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
continue
# Extract tracking information
boxes = results[0].boxes.xyxy.cpu().numpy()
track_ids = results[0].boxes.id.cpu().numpy().astype(int)
class_ids = results[0].boxes.cls.cpu().numpy().astype(int)
# Get base annotated frame
annotated_frame = results[0].plot()
# Process each tracked object
for box, track_id, cls_id in zip(boxes, track_ids, class_ids):
x1, y1, x2, y2 = box
# Calculate bounding box center
center_x = int((x1 + x2) / 2)
center_y = int((y1 + y2) / 2)
# Initialize track history if new track
if track_id not in track_history:
track_history[track_id] = []
# Assign random color for this track
track_colors[track_id] = tuple(
np.random.randint(0, 255, 3).tolist()
)
# Append current position
track_history[track_id].append((center_x, center_y))
# Maintain last 50 points
if len(track_history[track_id]) > 50:
track_history[track_id].pop(0)
# Draw trajectory trail
points = track_history[track_id]
color = track_colors[track_id]
for i in range(1, len(points)):
# Progressive thickness for recent points
thickness = max(1, int(3 * (i / len(points))))
cv2.line(annotated_frame, points[i-1], points[i],
color, thickness)
# Draw track ID label
label = f"ID: {track_id}"
cv2.putText(annotated_frame, label,
(int(x1), int(y1) - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.6,
color, 2)
# Add statistics overlay
active_tracks = len([t for t in track_history
if len(track_history[t]) > 0])
stats_text = f"Frame: {frame_count} | Active Tracks: {active_tracks}"
cv2.putText(annotated_frame, stats_text,
(10, 30), cv2.FONT_HERSHEY_SIMPLEX,
0.8, (255, 255, 255), 2)
cv2.imshow('Multi-Object Tracking', annotated_frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
print(f"Tracked across {frame_count} frames")
print(f"Total unique tracks: {len(track_history)}")
This implementation maintains consistent object identities, visualizes trajectories with colored trails, and provides real-time statistics. The trajectory visualization helps analyze movement patterns and understand scene dynamics.
Custom Tracking with Kalman Filters
ByteTrack internally uses Kalman filtering for motion prediction. Understanding this mechanism provides deeper insight into tracking robustness. The Kalman filter combines motion model predictions with actual measurements to produce optimal state estimates, handling temporary occlusions gracefully.
Here is a simplified Kalman tracker implementation:
import numpy as np
class SimpleKalmanTracker:
"""
Basic Kalman filter tracker for bounding boxes.
State: [x_center, y_center, width, height,
vx_center, vy_center, vwidth, vheight]
"""
def __init__(self, initial_bbox, track_id):
"""
Initialize tracker with bounding box.
Arguments:
initial_bbox: Initial box [x1, y1, x2, y2]
track_id: Unique track identifier
"""
self.track_id = track_id
self.age = 0
self.hits = 0
self.time_since_update = 0
# Convert bbox to center format
x1, y1, x2, y2 = initial_bbox
cx = (x1 + x2) / 2
cy = (y1 + y2) / 2
w = x2 - x1
h = y2 - y1
# State vector: position and velocity
# [cx, cy, w, h, vcx, vcy, vw, vh]
self.state = np.array([cx, cy, w, h, 0, 0, 0, 0],
dtype=np.float32)
# Measurement matrix (we only observe position)
self.measurement_matrix = np.array([
[1, 0, 0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0, 0, 0],
[0, 0, 1, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 0]
], dtype=np.float32)
# State transition matrix (constant velocity model)
dt = 1.0 # Time delta (one frame)
self.transition_matrix = np.array([
[1, 0, 0, 0, dt, 0, 0, 0],
[0, 1, 0, 0, 0, dt, 0, 0],
[0, 0, 1, 0, 0, 0, dt, 0],
[0, 0, 0, 1, 0, 0, 0, dt],
[0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 0, 1]
], dtype=np.float32)
# Process noise covariance
self.process_noise = np.eye(8, dtype=np.float32) * 0.01
# Measurement noise covariance
self.measurement_noise = np.eye(4, dtype=np.float32) * 1.0
# Error covariance matrix
self.error_covariance = np.eye(8, dtype=np.float32) * 10.0
def predict(self):
"""
Predict next state using motion model.
Returns:
predicted_bbox: Predicted box [x1, y1, x2, y2]
"""
# Predict state: x_pred = F * x
self.state = self.transition_matrix @ self.state
# Predict error covariance: P_pred = F * P * F^T + Q
self.error_covariance = (
self.transition_matrix @
self.error_covariance @
self.transition_matrix.T +
self.process_noise
)
self.age += 1
self.time_since_update += 1
# Convert state to bbox format
cx, cy, w, h = self.state[:4]
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
return np.array([x1, y1, x2, y2])
def update(self, measurement_bbox):
"""
Update state with new measurement.
Arguments:
measurement_bbox: Observed box [x1, y1, x2, y2]
"""
# Convert measurement to center format
x1, y1, x2, y2 = measurement_bbox
cx = (x1 + x2) / 2
cy = (y1 + y2) / 2
w = x2 - x1
h = y2 - y1
measurement = np.array([cx, cy, w, h], dtype=np.float32)
# Calculate Kalman gain: K = P * H^T * (H * P * H^T + R)^-1
innovation_covariance = (
self.measurement_matrix @
self.error_covariance @
self.measurement_matrix.T +
self.measurement_noise
)
kalman_gain = (
self.error_covariance @
self.measurement_matrix.T @
np.linalg.inv(innovation_covariance)
)
# Update state: x = x_pred + K * (z - H * x_pred)
predicted_measurement = (
self.measurement_matrix @ self.state
)
innovation = measurement - predicted_measurement
self.state = self.state + kalman_gain @ innovation
# Update error covariance: P = (I - K * H) * P
identity = np.eye(8, dtype=np.float32)
self.error_covariance = (
(identity - kalman_gain @ self.measurement_matrix) @
self.error_covariance
)
self.hits += 1
self.time_since_update = 0
def get_current_bbox(self):
"""
Get current bounding box estimate.
Returns:
bbox: Current box [x1, y1, x2, y2]
"""
cx, cy, w, h = self.state[:4]
x1 = cx - w / 2
y1 = cy - h / 2
x2 = cx + w / 2
y2 = cy + h / 2
return np.array([x1, y1, x2, y2])
This Kalman filter implementation predicts object positions between frames and updates estimates when measurements arrive. The constant velocity motion model assumes objects move smoothly, making predictions during brief occlusions. The Kalman gain balances trust between predictions and measurements based on their respective uncertainties.
PART THREE: IMAGE SEGMENTATION WITH MASK R-CNN
Understanding Instance Segmentation
Segmentation extends detection by providing pixel-level masks defining exact object boundaries. Instance segmentation treats each object as a distinct entity, producing separate masks even for objects of the same class. This granularity enables precise spatial reasoning required for applications like robotic manipulation, medical image analysis, and augmented reality.
Mask R-CNN extends the Faster R-CNN object detector by adding a mask prediction branch. The architecture has two stages. First, a region proposal network suggests candidate object locations. Second, these regions are refined through classification, bounding box regression, and mask generation occurring in parallel branches.
The critical innovation is ROI Align, which replaces the coarse ROI pooling used in Faster R-CNN. ROI Align uses bilinear interpolation to compute feature values at non-integer locations, preserving spatial alignment and enabling accurate mask prediction. This careful spatial handling proves essential for pixel-level accuracy.
Implementing Mask R-CNN Detection
Install the required libraries for segmentation:
pip install torch torchvision
# For visualization
pip install matplotlib pillow
Load a pre-trained Mask R-CNN model:
import torch
import torchvision
from torchvision import transforms
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np
def load_maskrcnn_model(device='cuda' if torch.cuda.is_available() else 'cpu'):
"""
Load pre-trained Mask R-CNN model.
Arguments:
device: Computation device ('cuda' or 'cpu')
Returns:
model: Loaded Mask R-CNN model in evaluation mode
device: Device where model is loaded
"""
# Load pre-trained model trained on COCO dataset
model = torchvision.models.detection.maskrcnn_resnet50_fpn(
weights='DEFAULT'
)
model.to(device)
model.eval() # Set to evaluation mode
print(f"Model loaded on {device}")
return model, device
Perform instance segmentation on images:
def segment_image(image_path, model, device, conf_threshold=0.5):
"""
Perform instance segmentation on image.
Arguments:
image_path: Path to input image
model: Loaded Mask R-CNN model
device: Computation device
conf_threshold: Minimum confidence for detections
Returns:
image: Original image as numpy array
results: Dictionary with boxes, labels, scores, and masks
"""
# Load and preprocess image
image = Image.open(image_path).convert('RGB')
image_tensor = transforms.ToTensor()(image)
image_tensor = image_tensor.to(device)
# Run inference
with torch.no_grad():
predictions = model([image_tensor])[0]
# Filter by confidence threshold
scores = predictions['scores'].cpu().numpy()
valid_indices = scores >= conf_threshold
# Extract results
boxes = predictions['boxes'].cpu().numpy()[valid_indices]
labels = predictions['labels'].cpu().numpy()[valid_indices]
scores = scores[valid_indices]
masks = predictions['masks'].cpu().numpy()[valid_indices]
# Convert masks to binary (threshold at 0.5)
masks = masks > 0.5
results = {
'boxes': boxes,
'labels': labels,
'scores': scores,
'masks': masks
}
# Convert image to numpy for visualization
image_np = np.array(image)
return image_np, results
Visualize segmentation results:
def visualize_segmentation(image, results, class_names=None):
"""
Visualize instance segmentation results.
Arguments:
image: Original image as numpy array
results: Segmentation results dictionary
class_names: Optional list of class names
"""
import matplotlib.patches as mpatches
from matplotlib.patches import Rectangle
fig, ax = plt.subplots(1, figsize=(12, 9))
ax.imshow(image)
# COCO class names (simplified)
if class_names is None:
class_names = [
'__background__', 'person', 'bicycle', 'car', 'motorcycle',
'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light',
'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird',
'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear',
'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie',
'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball',
'kite', 'baseball bat', 'baseball glove', 'skateboard',
'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup',
'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich',
'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut',
'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table',
'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard',
'cell phone', 'microwave', 'oven', 'toaster', 'sink',
'refrigerator', 'book', 'clock', 'vase', 'scissors',
'teddy bear', 'hair drier', 'toothbrush'
]
# Generate colors for each instance
np.random.seed(42)
colors = np.random.rand(len(results['boxes']), 3)
# Overlay each instance
for idx, (box, label, score, mask) in enumerate(zip(
results['boxes'], results['labels'],
results['scores'], results['masks']
)):
color = colors[idx]
# Draw mask with transparency
mask_colored = np.zeros_like(image, dtype=np.float32)
for c in range(3):
mask_colored[:, :, c] = mask[0] * color[c]
ax.imshow(mask_colored, alpha=0.5)
# Draw bounding box
x1, y1, x2, y2 = box
rect = Rectangle((x1, y1), x2-x1, y2-y1,
linewidth=2, edgecolor=color,
facecolor='none')
ax.add_patch(rect)
# Add label
class_name = class_names[label] if label < len(class_names) else 'unknown'
label_text = f'{class_name}: {score:.2f}'
ax.text(x1, y1 - 5, label_text,
bbox=dict(facecolor=color, alpha=0.5),
fontsize=10, color='white')
ax.axis('off')
plt.tight_layout()
plt.show()
Use these functions together:
# Load model
model, device = load_maskrcnn_model()
# Segment image
image, results = segment_image('test_image.jpg', model, device)
# Visualize results
print(f"Found {len(results['boxes'])} instances")
visualize_segmentation(image, results)
This implementation provides complete instance segmentation with mask overlay visualization. The model runs efficiently on GPUs, processing images in under 100 milliseconds. CPU inference takes several seconds but remains viable for non-real- time applications.
PART FOUR: DEPLOYING ON IOT EDGE DEVICES
Understanding Edge Computing Constraints
Edge devices like NVIDIA Jetson Nano and Raspberry Pi occupy the middle ground between powerful desktop systems and resource-constrained microcontrollers. These platforms provide dedicated hardware acceleration for neural networks while maintaining compact form factors and power efficiency suitable for embedded deployment.
The Jetson Nano features a Maxwell GPU with 128 CUDA cores providing 472 GFLOPS of compute power. It includes 4GB RAM and supports TensorRT for optimized inference. The Raspberry Pi 4 offers a quad-core ARM processor with optional neural accelerators like the Coral Edge TPU or Intel Neural Compute Stick. These devices enable real-time computer vision in applications including robotics, surveillance, and industrial automation.
Model optimization becomes critical for edge deployment. Techniques include quantization reducing precision from 32-bit floating point to 16-bit or 8-bit integers, pruning removing redundant network parameters, and knowledge distillation training smaller models to mimic larger ones. TensorRT applies these optimizations automatically, often achieving 2-5x speedup on Jetson devices.
Setting Up YOLO on Jetson Nano
Install JetPack SDK following NVIDIA instructions, then set up the Python environment:
# Install prerequisites
sudo apt-get update
sudo apt-get install -y python3-pip python3-dev
# Install PyTorch for Jetson (ARM64)
wget https://nvidia.box.com/shared/static/[latest-jetson-pytorch-wheel].whl
pip3 install [pytorch-wheel-file]
# Install Ultralytics YOLO
pip3 install ultralytics
# Install additional dependencies
pip3 install opencv-python numpy
Deploy YOLOv11 optimized for Jetson:
from ultralytics import YOLO
import cv2
import time
def deploy_yolo_jetson(camera_id=0, model_size='n'):
"""
Deploy YOLO on Jetson Nano with CSI camera.
Arguments:
camera_id: Camera device index
model_size: Model size ('n', 's', 'm')
"""
# Load model (smaller models for edge devices)
model_name = f'yolo11{model_size}.pt'
print(f"Loading {model_name}...")
model = YOLO(model_name)
# Open camera with GStreamer pipeline for CSI camera
gstreamer_pipeline = (
"nvarguscamerasrc sensor-id=%d ! "
"video/x-raw(memory:NVMM), width=1280, height=720, "
"format=NV12, framerate=30/1 ! "
"nvvidconv ! video/x-raw, format=BGRx ! "
"videoconvert ! video/x-raw, format=BGR ! "
"appsink"
) % camera_id
# Try CSI camera first, fall back to USB
cap = cv2.VideoCapture(gstreamer_pipeline, cv2.CAP_GSTREAMER)
if not cap.isOpened():
print("CSI camera not available, using USB camera")
cap = cv2.VideoCapture(camera_id)
if not cap.isOpened():
raise RuntimeError("Cannot open camera")
print("Camera opened successfully")
print("Press 'q' to quit")
frame_times = []
while True:
ret, frame = cap.read()
if not ret:
break
# Time inference
start_time = time.time()
results = model(frame, verbose=False)
inference_time = time.time() - start_time
frame_times.append(inference_time)
if len(frame_times) > 30:
frame_times.pop(0)
# Calculate metrics
avg_inference = np.mean(frame_times) * 1000 # Convert to ms
fps = 1.0 / np.mean(frame_times)
# Get annotated frame
annotated = results[0].plot()
# Add performance overlay
perf_text = f"Inference: {avg_inference:.1f}ms | FPS: {fps:.1f}"
cv2.putText(annotated, perf_text, (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.7,
(0, 255, 0), 2)
cv2.imshow('Jetson YOLO Detection', annotated)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
print(f"Average inference time: {np.mean(frame_times)*1000:.1f}ms")
print(f"Average FPS: {1.0/np.mean(frame_times):.1f}")
Optimizing with TensorRT
TensorRT dramatically improves inference speed on Jetson. Export and optimize YOLO:
def export_tensorrt_model(model_path='yolo11n.pt', workspace=4):
"""
Export YOLO model to TensorRT format.
Arguments:
model_path: Path to YOLO model weights
workspace: TensorRT workspace size in GB
"""
from ultralytics import YOLO
print(f"Exporting {model_path} to TensorRT...")
model = YOLO(model_path)
# Export to TensorRT with FP16 precision
model.export(format='engine',
workspace=workspace,
half=True) # Use FP16 precision
print("Export complete!")
print(f"TensorRT engine saved")
```
Run inference with TensorRT engine:
```
def run_tensorrt_inference(engine_path, camera_id=0):
"""
Run optimized TensorRT inference.
Arguments:
engine_path: Path to .engine file
camera_id: Camera device index
"""
model = YOLO(engine_path)
cap = cv2.VideoCapture(camera_id)
print("Running TensorRT optimized inference")
while True:
ret, frame = cap.read()
if not ret:
break
# TensorRT automatically accelerates inference
results = model(frame, verbose=False)
annotated = results[0].plot()
cv2.imshow('TensorRT Inference', annotated)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
TensorRT optimization typically doubles inference speed on Jetson Nano. The nano model achieves 15-20 FPS, while the small model runs at 8-12 FPS with improved accuracy.
Raspberry Pi Deployment
Raspberry Pi requires different optimization strategies due to its ARM CPU architecture:
# Install dependencies on Raspberry Pi
sudo apt-get update
sudo apt-get install -y python3-pip python3-opencv
pip3 install ultralytics numpy
Deploy lightweight model on Raspberry Pi:
def deploy_raspberry_pi(camera_id=0):
"""
Deploy YOLO on Raspberry Pi with optimizations.
"""
from ultralytics import YOLO
import cv2
# Use smallest model for Raspberry Pi
model = YOLO('yolo11n.pt')
# Reduce input resolution for faster processing
model.overrides['imgsz'] = 320 # Reduced from default 640
cap = cv2.VideoCapture(camera_id)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
print("Raspberry Pi detection starting...")
while True:
ret, frame = cap.read()
if not ret:
break
# Process every other frame to improve framerate
results = model(frame, verbose=False)
annotated = results[0].plot()
cv2.imshow('Pi Detection', annotated)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
On Raspberry Pi 4, expect 2-4 FPS with the nano model at reduced resolution. The Raspberry Pi 5 performs a bit better. Adding a Coral Edge TPU accelerator increases performance to 10-15 FPS with compatible models.
PART FIVE: MICROCONTROLLER DEPLOYMENT WITH TENSORFLOW LITE MICRO
Understanding Microcontroller Constraints
Microcontrollers like the ESP32 present extreme resource constraints. With typical specifications of 520KB SRAM, 4MB flash storage, and a 240MHz dual-core processor, running neural networks requires aggressive optimization. TensorFlow Lite Micro enables this deployment by providing a minimal runtime optimized for embedded systems.
The key to successful microcontroller deployment lies in model quantization and architecture selection. Full 32-bit floating point models are impractical. Instead, we employ 8-bit integer quantization reducing model size by 75 percent while typically sacrificing less than 1 percent accuracy. Model architectures must be carefully designed with limited layers and parameters.
Common microcontroller computer vision applications include person detection, gesture recognition, and simple object classification. Real-time video processing remains challenging, with typical inference times measured in hundreds of milliseconds to seconds per frame. However, for many IoT applications where continuous monitoring suffices, this performance proves adequate.
Setting Up TensorFlow Lite Micro on ESP32
Install the Arduino IDE and ESP32 board support:
# In Arduino IDE:
# File -> Preferences -> Additional Board Manager URLs
# Add: https://dl.espressif.com/dl/package_esp32_index.json
# Tools -> Board -> Boards Manager
# Search and install "ESP32"
Install TensorFlow Lite Micro library:
# In Arduino IDE:
# Sketch -> Include Library -> Manage Libraries
# Search and install "TensorFlowLite_ESP32"
Simple Person Detection Example
Let‘s create a complete person detection system for ESP32-CAM:
#include <esp_camera.h>
#include <TensorFlowLite_ESP32.h>
#include "person_detect_model_data.h"
// Include TensorFlow Lite Micro headers
#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
// Model input dimensions
const int kImageWidth = 96;
const int kImageHeight = 96;
const int kImageChannels = 1; // Grayscale
// TensorFlow Lite objects
tflite::MicroErrorReporter micro_error_reporter;
tflite::ErrorReporter* error_reporter = µ_error_reporter;
const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;
// Memory allocation for model
// Increase if model is more complex
constexpr int kTensorArenaSize = 70 * 1024;
alignas(16) uint8_t tensor_arena[kTensorArenaSize];
// Camera pin configuration for ESP32-CAM (AI-Thinker)
camera_config_t camera_config = {
.pin_pwdn = 32,
.pin_reset = -1,
.pin_xclk = 0,
.pin_sscb_sda = 26,
.pin_sscb_scl = 27,
.pin_d7 = 35,
.pin_d6 = 34,
.pin_d5 = 39,
.pin_d4 = 36,
.pin_d3 = 21,
.pin_d2 = 19,
.pin_d1 = 18,
.pin_d0 = 5,
.pin_vsync = 25,
.pin_href = 23,
.pin_pclk = 22,
.xclk_freq_hz = 20000000,
.ledc_timer = LEDC_TIMER_0,
.ledc_channel = LEDC_CHANNEL_0,
.pixel_format = PIXFORMAT_GRAYSCALE,
.frame_size = FRAMESIZE_96X96,
.jpeg_quality = 10,
.fb_count = 1
};
void setup() {
Serial.begin(115200);
delay(1000);
Serial.println("TensorFlow Lite Micro Person Detection");
// Initialize camera
esp_err_t err = esp_camera_init(&camera_config);
if (err != ESP_OK) {
Serial.printf("Camera init failed with error 0x%x\n", err);
return;
}
Serial.println("Camera initialized");
// Load the TensorFlow Lite model
model = tflite::GetModel(g_person_detect_model_data);
if (model->version() != TFLITE_SCHEMA_VERSION) {
error_reporter->Report(
"Model schema version %d not supported, version %d",
model->version(), TFLITE_SCHEMA_VERSION);
return;
}
Serial.println("Model loaded");
// Create operations resolver
static tflite::AllOpsResolver resolver;
// Build interpreter
static tflite::MicroInterpreter static_interpreter(
model, resolver, tensor_arena, kTensorArenaSize,
error_reporter);
interpreter = &static_interpreter;
// Allocate tensors
TfLiteStatus allocate_status = interpreter->AllocateTensors();
if (allocate_status != kTfLiteOk) {
error_reporter->Report("AllocateTensors() failed");
return;
}
Serial.println("Tensors allocated");
// Get input tensor pointer
input = interpreter->input(0);
Serial.println("Setup complete. Starting inference...");
}
void loop() {
// Capture image from camera
camera_fb_t* fb = esp_camera_fb_get();
if (!fb) {
Serial.println("Camera capture failed");
delay(1000);
return;
}
// Verify image dimensions
if (fb->width != kImageWidth || fb->height != kImageHeight) {
Serial.printf("Image size mismatch: %dx%d\n",
fb->width, fb->height);
esp_camera_fb_return(fb);
delay(1000);
return;
}
// Copy image data to model input
// Model uses quantized input (-128 to 127)
for (int i = 0; i < kImageWidth * kImageHeight; i++) {
// Convert unsigned byte to signed byte
input->data.int8[i] = fb->buf[i] - 128;
}
// Return frame buffer to camera driver
esp_camera_fb_return(fb);
// Run inference
unsigned long start_time = millis();
TfLiteStatus invoke_status = interpreter->Invoke();
unsigned long inference_time = millis() - start_time;
if (invoke_status != kTfLiteOk) {
error_reporter->Report("Invoke failed");
delay(1000);
return;
}
// Get output tensor (2 values: person score, no-person score)
TfLiteTensor* output = interpreter->output(0);
int8_t person_score = output->data.int8[1];
int8_t no_person_score = output->data.int8[0];
// Convert quantized scores to probabilities
// Quantization zero point and scale depends on training
float person_probability = (person_score + 128) / 256.0;
float no_person_probability = (no_person_score + 128) / 256.0;
// Print results
Serial.printf("Inference time: %lu ms\n", inference_time);
Serial.printf("Person: %.2f | No person: %.2f\n",
person_probability, no_person_probability);
// Determine detection
if (person_probability > 0.6) {
Serial.println("PERSON DETECTED!");
} else {
Serial.println("No person");
}
Serial.println("---");
// Wait before next inference
delay(2000);
}
This implementation performs person detection at approximately 700-1000 milliseconds per inference on ESP32. The model input requires 96x96 grayscale images, dramatically reducing computational requirements compared to standard RGB inputs.
Optimizing Inference Performance
Several optimization strategies improve microcontroller performance:
Reduce input resolution. Smaller images require fewer computations. Consider 48x48 or even 32x32 for simple detection tasks.
Apply aggressive quantization. Post-training quantization converts trained models to 8-bit integer precision with minimal accuracy loss. Some models support 4-bit or even binary quantization for extreme optimization.
Simplify model architecture. Use depthwise separable convolutions instead of standard convolutions, reducing parameters by factors of 8-10. MobileNet-style architectures work well on microcontrollers.
Optimize tensor arena size. The tensor arena holds intermediate activation tensors. Finding the minimum sufficient size reduces memory pressure, allowing larger models or additional application code.
Here is an optimized configuration example:
// Optimized tensor arena sizing
// Start large, then reduce until allocation fails
constexpr int kTensorArenaSize = 45 * 1024; // Reduced from 70KB
// Use quantized operations resolver for smaller code size
// Instead of AllOpsResolver, register only needed operations
static tflite::MicroMutableOpResolver<5> micro_op_resolver;
micro_op_resolver.AddConv2D();
micro_op_resolver.AddMaxPool2D();
micro_op_resolver.AddFullyConnected();
micro_op_resolver.AddReshape();
micro_op_resolver.AddSoftmax();
Custom Model Training for Microcontrollers
Training custom models for microcontrollers requires TensorFlow with quantization- aware training. Here is a simplified training pipeline:
import tensorflow as tf
import tensorflow_model_optimization as tfmot
def create_quantized_model(input_shape=(96, 96, 1), num_classes=2):
"""
Create quantization-aware training model.
Arguments:
input_shape: Input image dimensions
num_classes: Number of output classes
Returns:
model: Quantization-aware model
"""
# Build simple CNN model
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(16, (3, 3), activation='relu',
input_shape=input_shape),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(32, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D((2, 2)),
tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(num_classes, activation='softmax')
])
# Apply quantization-aware training
quantize_model = tfmot.quantization.keras.quantize_model
q_aware_model = quantize_model(model)
# Compile with appropriate optimizer
q_aware_model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
return q_aware_model
def convert_to_tflite_micro(model, representative_data):
"""
Convert trained model to TensorFlow Lite Micro format.
Arguments:
model: Trained Keras model
representative_data: Dataset for calibration
Returns:
tflite_model: Quantized TFLite model
"""
# Create converter
converter = tf.lite.TFLiteConverter.from_keras_model(model)
# Enable optimizations
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# Provide representative dataset for full integer quantization
def representative_dataset_gen():
for data in representative_data.take(100):
yield [data]
converter.representative_dataset = representative_dataset_gen
# Enforce full integer quantization
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS_INT8
]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
# Convert model
tflite_model = converter.convert()
return tflite_model
# Training example
# model = create_quantized_model()
# model.fit(train_data, epochs=20, validation_data=val_data)
# tflite_model = convert_to_tflite_micro(model, representative_data)
# Save model
# with open('model.tflite', 'wb') as f:
# f.write(tflite_model)
# Convert to C array for embedding
# xxd -i model.tflite > model_data.cc
This training pipeline produces fully quantized models optimized for microcontroller deployment. The representative dataset calibrates quantization ranges, ensuring minimal accuracy degradation.
CONCLUSION AND FUTURE DIRECTIONS
This comprehensive guide has explored object detection, tracking, and segmentation across the full spectrum of computing hardware. We began with high-performance desktop systems running state-of-the-art models like YOLOv11 and Mask R-CNN, progressed through edge devices like Jetson Nano enabling efficient embedded deployment, and concluded with microcontrollers demonstrating that even severe resource constraints cannot prevent sophisticated computer vision capabilities.
The key takeaway is that modern AI democratizes computer vision. The same conceptual approaches scale from server-class GPUs processing 4K video at 60 FPS to microcontrollers analyzing single frames every few seconds. Success requires understanding the engineering trade-offs: model complexity versus inference speed, accuracy versus power consumption, generality versus application-specific optimization.
Future developments promise even greater capabilities. New model architectures continue improving the accuracy-efficiency frontier. Hardware accelerators specifically designed for neural networks proliferate across all tiers. Automated neural architecture search discovers optimal models for specific constraints. Federated learning enables privacy-preserving training on edge devices.
The practical implementations presented here provide solid foundations for real-world applications. Whether building autonomous systems, industrial automation, healthcare diagnostics, or consumer electronics, these techniques enable sophisticated visual understanding at every scale. The code examples demonstrate not just how to run inference but how to build complete systems handling data capture, processing, visualization, and result delivery.
Computer vision continues its rapid evolution. Staying current requires continuous learning, experimentation with new architectures, and practical deployment experience. The tools and techniques presented here serve as starting points for exploration and innovation. The future of AI at the edge has never been brighter, and the barriers to entry have never been lower.
Start with the examples provided, adapt them to your specific requirements, and build the next generation of intelligent systems. The capability to perceive and understand visual information now fits in the palm of your hand and costs less than a meal. Use it wisely, use it well, and use it to build a better future.
