SAM (Segment Anything Model)
Meta's Segment Anything Model (SAM) introduced the "promptable segmentation" paradigm. The latest SAM 2 (2024) unifies image and video segmentation, enabling real-time object tracking and zero-shot generalization across frames.
From Images to Video (SAM vs SAM 2)
SAM (2023) - The Foundation
Separated the Image Encoder (Heavy ViT, runs once) from the Prompt Encoder and Mask Decoder (Lightweight, runs in real-time). This allowed instant mask generation from clicks, boxes, or text on static images.
SAM 2 (July 2024) - Unified Video
SAM 2 treats video as a stream of images. It introduces a Streaming Memory architecture. When you click an object in one frame, the model stores its features in memory and "propagates" the mask to future frames, handling occlusion and reappearance automatically.
SAM 2 Architecture (Streaming)
Capabilities & Performance
Zero-Shot Generalization
Trained on the massive SA-V dataset (50k+ videos, 35M+ masks). It can segment objects it has never seen before (e.g., underwater species, microscopic cells) without finetuning.
Real-Time Video
Achieves ~44 FPS processing speed. This unlocks real-time video editing effects (like removing a moving person) and live AR applications.
Ambiguity Handling
If a user clicks strictly on a wheel, SAM outputs 3 valid masks: the wheel, the car, and the entire scene depth. This "ambiguity awareness" is key for UX.
Using SAM 2 (Python)
import torch
from sam2.build_sam import build_sam2_video_predictor
checkpoint = "sam2_hiera_large.pt"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)
# Initialize with video
inference_state = predictor.init_state(video_path="video.mp4")
# Add a click on the first frame (frame_idx=0) to track an object
# points are (x, y), labels are 1 (positive click)
predictor.add_new_points(
inference_state=inference_state,
frame_idx=0,
obj_id=1,
points=[[500, 300]],
labels=[1],
)
# Propagate forward to track through the video
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
show_mask(out_mask_logits) # visualize result