Recipe — Python depth

For LYNX SDK 1.0. Complete, end-to-end. Install: install/python.md. API: api/python.md. Conventions: api/conventions.md.

Goal: run detection and depth in one pass, read the dense depth map plus a per-object depth, and turn depth into a picture you can display. Model: lynx-basic (keyless — it ships a depth head alongside detection; see models/catalog.md).

The whole thing

Depth is a frame-global head, so you ask for it with tasks=. Combine the box head with the depth head: tasks=lynx.Task.BOX | lynx.Task.DEPTH. The result carries a dense frame.depth_map (an HxW numpy array, or None if no depth was realized) and each detection gets a scalar d.depth sampled at the object. frame.depth_is_metric tells you whether those numbers are meters or an unitless relative scale. To show depth to a human, lynx.depth_to_u8(frame) renders it as a displayable (H, W, 3) uint8 RGB image (closer = brighter) at the original input resolution.

1import lynx
2from lynx.errors import LynxError, ModelNotFound
3 
4def depth(image_path, min_confidence=0.4):
5    """Load lynx-basic (keyless), run detection + depth, print per-object depth."""
6    # Open once. First call downloads + verifies + caches the model.
7    with lynx.open("lynx-basic") as model:
8        # Ask for BOTH heads: the box head (detections) and the depth head.
9        frame = model.predict(
10            image_path,                              # path, or an (H, W, 3) uint8 RGB array
11            tasks=lynx.Task.BOX | lynx.Task.DEPTH,
12            conf=min_confidence,
13        )
14 
15        # Dense frame-global depth grid — an (H, W) ndarray, or None if the model
16        # realized no depth for this frame.
17        dmap = frame.depth_map
18        if dmap is None:
19            print("No depth map for this frame.")
20            return
21 
22        unit = "m" if frame.depth_is_metric else "rel"   # meters vs. relative scale
23        print(f"depth map {dmap.shape[1]}x{dmap.shape[0]}, "
24              f"{'metric (meters)' if frame.depth_is_metric else 'relative (unitless)'}")
25 
26        # Per-object depth: a scalar sampled at each detection's root.
27        print(f"{len(frame.detections)} detection(s) in {image_path!r}:")
28        for d in frame.detections:
29            name = model.classes(d.class_id).name        # e.g. "PERSON"
30            z = d.depth                                  # float; NaN if no depth here
31            z_str = "n/a" if z != z else f"{z:.2f}{unit}"  # z != z tests for NaN
32            print(f"  {name:16} {d.confidence:.2f}  depth={z_str}")
33 
34        # Render depth as a displayable image (closer objects brighter than far).
35        depth_rgb = lynx.depth_to_u8(frame)              # (H, W, 3) uint8 RGB, or None
36        if depth_rgb is not None:
37            from PIL import Image                         # pip install pillow
38            Image.fromarray(depth_rgb).save("depth.png")
39            print("wrote depth.png")
40 
41 
42if __name__ == "__main__":
43    import sys
44    image_path = sys.argv[1] if len(sys.argv) > 1 else "photo.jpg"
45    try:
46        depth(image_path)
47    except ModelNotFound as e:
48        print("model not found (bad slug/version):", e.message)
49    except LynxError as e:
50        print(f"lynx error [{e.code}]: {e.message}")

The dense depth map

frame.depth_map is a plain (H, W) numpy array at the model's depth resolution — index it, threshold it, or feed it to your own code:

1dmap = frame.depth_map                       # (H, W) float ndarray, or None
2if dmap is not None:
3    print("nearest:", float(dmap.min()), "farthest:", float(dmap.max()))
4 
5    # Depth under a detection's box (input-pixel coords), averaged:
6    d = frame.detections[0]
7    x1, y1, x2, y2 = (int(v) for v in d.box)
8    patch = dmap[y1:y2, x1:x2]
9    if patch.size:
10        print("mean depth over box:", float(patch.mean()))

Rendering depth for display

lynx.depth_to_u8(frame) (equivalently frame.depth_u8()) normalizes and resizes depth to a viewable RGB image in the C core — no matplotlib needed. It returns None when the frame has no depth map:

1depth_rgb = lynx.depth_to_u8(frame)          # (H, W, 3) uint8 RGB at input resolution
2if depth_rgb is not None:
3    from PIL import Image
4    Image.fromarray(depth_rgb).show()

Confirm the model has a depth head

Not every model emits depth. Check model.capabilities (a Task bitmask) before asking for it:

1if lynx.Task.DEPTH in model.capabilities:
2    frame = model.predict(image_path, tasks=lynx.Task.BOX | lynx.Task.DEPTH)
3else:
4    frame = model.predict(image_path, tasks=lynx.Task.BOX)   # detection only

Notes

Metric vs. relative. frame.depth_is_metric is True when frame.depth_map and d.depth are in meters; False when they're a unitless relative scale (near/far only, no absolute distance). Always branch on it before treating a number as meters.
None / NaN when there's no depth. frame.depth_map and lynx.depth_to_u8(frame) return None if depth wasn't realized (e.g. you didn't pass Task.DEPTH, or the model has no depth head). Per-object d.depth returns NaN when no depth map was realized or the sampled point has no valid depth — test with z != z (NaN is never equal to itself).
Ask for depth explicitly. Pass tasks=lynx.Task.BOX | lynx.Task.DEPTH to predict (opening with tasks=0, the default, opens every head the model declares — but being explicit keeps the pass lean). d.depth needs a box to sample at, so keep Task.BOX in the mix.
Depth resolution. frame.depth_map is at the model's depth grid resolution; lynx.depth_to_u8(frame) resizes back to the original input resolution for display. d.box coordinates are in input pixels — scale if you index depth_map directly with them.
3D from depth (beta). With camera intrinsics you can unproject an object to a camera-frame 3D point: d.position_3d(lynx.Intrinsics(fx, fy, cx, cy)) → Point3D (or None), and d.box_3d(...) → Box3D. Units follow the depth map (meters when depth_is_metric).
Reuse one loaded Model across calls; don't reopen per image. Run the first open off the UI thread (it downloads).