Teaching the agent to remember objects

Same model, 47 semantic-object VQA cases over a YOLO+odom object memory from the go2_bigoffice replay. The only change is ImageDetections3DPC.agent_encode() — one method, no extra skills, no wrapper stores (gate-enforced abstraction budget).

Result

0.355

before · rich-table str()

0.9997

after · exp_0013

0.06

blind control (gate ≤ 0.35)

Stability replicates: 0.9997 / 0.953 / 0.9998 — mean 0.984, every replicate above 0.95. Residual is per-row sampling variance (one flaky MCQ), not encoding capability.

Semantic representation — before / after

Before: 293 raw detections. After: 19 canonical labeled objects. — FIG. 01The object memory, drawn in world coordinates. Left: every raw YOLO detection odom-grounded along the robot's 162 m path (grey = noise classes: surfboard, train, pizza…). Right: what the winning encoding presents — the greedy 1.5 m instance scan's canonical objects with ids (chair#0…#5), noise deleted, one position per physical object. — robot path.

What the model reads

Before — 161 rich tables, ~156k chars

        ImageDetections3DPC [1 detections @ 6.670]        
┏━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ # ┃ Name      ┃ Class ┃ Track ┃ Conf ┃ Dist   ┃ Points ┃
┡━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ 0 │ surfboard │ -1    │ -1    │ 0.51 │ 17.44m │ 1      │
└───┴───────────┴───────┴───────┴──────┴────────┴────────┘

"How many distinct chairs?" → 3 (truth 6)

"What is within 3 m of the bed?" → "surfboard"

nearest-object: answered refrigerator on 3 of 5 rows regardless of pose.

After — name#k entries + P1–P9 procedure legend, ~17k chars

bottle#0 conf=0.57 pos=(-4.0, -2.8, 0.3)
chair#2 conf=0.61 pos=(-1.4, 4.6, 0.3)   ← instance ids pre-assigned
   by the encoder's greedy scan

SEMANTIC OBJECT MAP -- reading guide and reasoning procedures.
Each observation below is one camera frame; each entry reads
"name#k conf=C pos=(x, y, z)": one detection of class name at world-frame
position (x, y, z) in meters (+x east, +y north, z up). The same physical
object is re-detected on many frames, so it appears as many entries with
slightly different positions; #k is its instance id — entries sharing a
tag (class AND id) are sightings of the SAME physical object (ids come
from an exact greedy scan: a detection joins the first same-class instance
whose running-mean position lies within 1.5 m, else it founds the next id).

Use these procedures exactly; do every computation numerically, step by step.

P1 BUILD THE OBJECT LIST FIRST (before answering anything):
  a. Group entries by their full tag (class AND id). Each tag is ONE
     physical object: position …

Sample QA (ground truth from the frozen teacher)

recall	“List every distinct object class recorded.”	10 classes (set-F1)
count	“How many distinct chair objects are in the mapped area?”	6 ±1
within	“List every class within 3 m of the bed.”	[laptop]
nextto	“Which class is nearest to the bed (other than bed)?”	laptop
between	“Which single class lies between the laptop and the teddy bear?”	chair
objdist	“How far apart horizontally are the bed and the refrigerator?”	15.07 m ±3.77
robotdist	“You are the robot (pose = last odom). How far are you from the book?”	5.52 m ±1.38
egoside	“You face your odom heading. Is the refrigerator ahead, behind, left or right?”	right
compass	“Which compass direction is the bed from the refrigerator?”	northwest

Per-family scores

presence7 rows

1.00 → 1.00

robotdist3

0.59 → 1.00

within / nextto / between2+2+2

0.50 → 1.00

recall3

0.43 → 1.00

zonecount3

0.33 → 1.00

objdist8

0.20 → 1.00

egoside6

0.17 → 1.00

compass / nearest / count5+5+1

0.00 → 1.00

How it was found

exp_0004	0.355	baseline — raw per-frame rich tables (all information, no structure)
exp_0005	0.831	closed-form recipe legend + compact name/conf/pos rows
exp_0007	0.945	procedures made exactly mechanical (greedy time-order scan)
exp_0009	0.957	clustering moved into the encoder — pre-assigned instance ids
exp_0011	0.978	strictly-between admission test; dominant-axis compass caution
exp_0013	0.9997	anti-salience rule forcing the per-class distance table — all 47 pass
structure lane	0.827	exp_0006→0010: compact canonical encoding lane (converged; recipes lane won)

Findings that generalize

Precompute what models won't execute. The model cannot run a greedy clustering scan over 290 rows in-context; moving instance ids into the encoder fixed what three legend rewrites couldn't.

Salience beats arithmetic — unless forced. The model answered "refrigerator" to nearest-object regardless of pose until the legend mandated writing out the per-class distance table before choosing.

Ego frame = signed comparisons. fwd/left dot-products with |fwd| ≥ |left| → ahead/behind, else left/right. Zero trig in the prompt; 0.17 → 1.0.

Blind-gate your benchmark. Question sets drift guessable (office-prior presence, binary left/right). Two calibration rounds pushed blind from 0.39 to 0.06 — before any optimization was trusted.