DVIDIA Observation Lab pilot

Engineering integration passed on six public excerpts; annotation accuracy remains unmeasured. On 10 October 2026, we ran a local observation pipeline that decodes video, proposes sampled object and palm boxes, associates nearby detections and opens an editable review timeline. Review changes persist as revisions and export with their source identity. This establishes a working research tool, not reliable automatic task labeling, a cup-placement benchmark or a trained robot skill. The published implementation is anchored to DVIDIA Training commit d355a4a.

Install the Observation Lab release · GitHub release and downloads · Try the synthetic review demo · Frozen engineering receipts

The pilot produces reviewable 2D evidence

The local adapter uses the pinned OpenCV Zoo YOLOX COCO object detector and MediaPipe palm detector. It samples at two frames per second, up to 120 frames and a 640-pixel maximum image side per clip. Proposals retain decoded presentation timestamps, frame and source hashes, raw labels and scores. Same-class image overlap associates detections; palm/object overlap proposes 2D proximity. Neither association establishes physical identity nor overlap proves touching. Palm rectangles are not full-hand segmentation. No optical flow, depth reconstruction, natural-language action captioner or language model participates in this pilot. (Observation contract, frozen model and recipe)

The review interface shares one clock between the original video and timeline. It displays a box only within 0.25 seconds of its sampled timestamp, leaving gaps visible. Reviewers can correct labels and boundaries, approve or reject proposals and add missed events. Raw model labels remain separate from the accepted draft. Outcomes default to unknown; visible_completion and incomplete describe the operator's visual assessment. They are not robot-success measurements. (Review interface contract)

Saving appends a revision with conflict detection against the current base revision. A stale tab cannot silently replace newer edits. Every event needs a decision before review completion; unsaved changes trigger navigation protection, and export is disabled until saving. Downloads contain the last saved review, episode ID and original source hash. These checks make corrections inspectable. They do not authenticate the operator or turn agreement into independent ground truth. The loopback server performs no uploads and checks the local origin for writes. (Local workflow)

Six excerpts establish integration, not accuracy

The fixed input order was shirt, stir, pack, onion, melon and jeans, selected before examining model output. Five excerpts come from EgoAnnotate, credited to Taher Panbiharwala and Zainab Barwaniwala; the stir excerpt comes from CoMind, with its full author credit retained in the protocol. The records identify CC BY 4.0 terms and upstream source links. We hash-verified the exact excerpts; full upstream originals were not downloaded or hash-verified for this run. This publication distributes sanitized receipts rather than footage or sampled frames. (Input protocol and credits)

The six excerpts total 19 seconds, 2,753,287 bytes and 38 sampled frames. They are neither independent complete attempts nor the intended cup-placement cohort. Unknown session provenance remains one conservative shared group. There was no training split, learning run or physical evaluation. Publisher titles were display metadata and were not fed to the detector. (Episode summary, run and grouping)

These are two single runs on the same inputs and a warm macOS ARM64 machine. Timing includes intake, copy/hash, decoding, detection, association and per-episode writes. It excludes model provisioning/loading, final receipt writing, UI startup and human review. The difference is not a controlled speedup. Peak RAM, energy, cost, p95 latency, minimum hardware and production throughput were not measured. No paid inference API was called. The runtime used Python 3.12.9, OpenCV 5.0.0.93 and NumPy 2.5.3. Execution requested four CPU threads and one inference worker, while frozen OpenCV metadata records ten threads; an enforced thread cap was not independently verified. No GPU path was requested or measured. (Initial receipt, hardened receipt, full frozen protocols, original benchmark description)

Workflow checks preserve failures and revisions

Review after the initial run identified a possible browser-clock mismatch for nonzero-start videos. Recipe v1.1 rejects nonzero container, video or decoded starts rather than silently moving boxes against an unchanged original. All six inputs have zero starts and were accepted in both runs. Initial artifacts remain alongside the hardened attempt. Same-size media replacement, malformed review IDs and oversized metadata also received workflow checks. These changes do not establish improved detector quality. (Original and revised recipes)

The implementation records 307 passing repository tests followed by 24 focused observation tests, including four additional detector-admission/suppression checks. Fifteen browser checks exercised playback, sampled overlays, correction save/reload, manual events, export, unsaved edits and completion validation on an isolated copy. The 390px and 320px layouts had no horizontal overflow; no browser console or page errors were recorded. These are interface and software checks, not physical-phone, Safari or annotation-accuracy qualification. Test edits do not count as independent human references. (Test scope, browser receipt)

Qualitative inspection found a saucepan labeled cup, storage jars labeled cups and missed palms in the preliminary stir check. These observations are not a scored accuracy estimate. COCO categories omit task-specific objects; overlap association can fragment or switch identities; two samples per second can miss brief actions. Model scores remain uncalibrated and all automatic outcomes remain unknown. Without independent references, inspection reports accuracy: null and accuracy_status: not_assessed. (Recorded limitations)

The release keeps ownership and missing evidence explicit

DVIDIA owns the orchestration, intake, timing, proposal association, review contract and workflow. Original DVIDIA Training code remains MIT. Detector preprocessing/postprocessing adapted from OpenCV Zoo retains Apache-2.0 attribution, and the pinned detector artifacts retain their separate upstream licenses and credits. The two ONNX files total about 40 MB and are provisioned explicitly with size/hash verification; inference does not download weights. Model terms do not grant rights to input footage. No new foundation model, SpatialLM, MANO or depth-model weights are claimed. (Exact artifacts and notices)

This lab is upstream of robot learning. Image rectangles supply no calibrated XYZ, contact forces, joint actions, gripper commands or robot policy. No automatic physical action is produced, and the existing visual predictor and movement head gain no new skill from this release. The next proposed study collects 30–50 complete permission-cleared cup-placement attempts with explicit session/person grouping, failures, occlusion and recovery. Independent blinded references should precede comparisons of detector-only, temporal-caption, combined and manual annotation at matched quality, correction time and full cost. Depth, a supervised action bridge and closed-loop qualification require separate experiments. (Uncollected cohort and gates)

The public Hugging Face Observation Lab Space reuses our review interface with two authored geometric videos and authored image boxes. It is a synthetic workflow demonstration: no detector, VLM or robot policy runs in the hosted page. Revisions stay in each visitor's browser, with a visible temporary-memory fallback if browser storage or cross-tab locking is unavailable. Visitors can edit, save and export their own review without server writes. The actual detector pipeline is installed and run locally on permitted footage. All sixteen hosted files were read anonymously at Space commit 8547b78 and compared with the staged hashes. The public runtime was observed running with its synthetic scope labels visible.

Publication preparation reran 311 repository tests and separately checked 22 publisher tests, six synthetic preparation/privacy/media tests and browser-local revision/conflict behavior. Both pinned ONNX detectors loaded and ran on a neutral image in the local CPU runtime; that smoke test measures neither detection accuracy nor task understanding. Release source preserves the original pilot receipts and clarifies the requested-versus-reported thread discrepancy. Original code uses MIT and the detector adapter retains its Apache-2.0 notices. Production uploads, accounts, storage and application analysis remain separate from this research release.

Image-motion source alpha estimates speed and future paths

On 10 October 2026, the next source alpha adds causal image motion and passive forecasting to our local research tools. Candidate pull request 7 and exact source commit ade2b54 identify this addition. The published observation-lab-v0.1.0 release and Hugging Face review demo remain separate; neither includes these motion capabilities. The complete Observation Lab v0.2 remains planned. (Installation and motion contract, next-version plan)

The user supplies one first-frame image region and label. Sparse CPU optical flow translates its initial center using robust feature displacement; two contiguous observations estimate velocity, and constant-velocity forecasts project that image anchor 100, 250 and 500 ms ahead. Integer input presentation timestamps multiplied by the declared rational time base preserve the source clock. Initial labels and regions are human-authored, not detector outputs; no new weights are trained or downloaded. Position and speed are analysis-image pixels and pixels/second. The anchor is not a physical centroid, and this fixed-camera baseline has no calibrated metric speed, contact model or robot command authority. A lost track remains lost without interpolation or reacquisition. (Exact implementation contract)

The frozen authored benchmark includes stationary motion, constant velocity, stop/start, direction change, brief occlusion and low texture. Each generated clip is four seconds at 640 × 360 and 30 fps; the batch contains six cases, 720 attempted frames and 24 source seconds. Future authored reference trajectories enter only the scorer, after analysis; they do not enter earlier tracking or forecasts. These six cases share one generator and texture rather than independent real-world task attempts. (Frozen protocol, aggregate scores and denominators)

The batch ran once on macOS ARM64 with Python 3.12.9, OpenCV 5.0.0 and NumPy 2.5.3. Processing covers intake, probing, streaming decode, tracking/forecast, writes and scoring; the offline processing/source-duration ratio is 0.0606. One OpenCV thread was requested while its runtime reported ten; actual concurrency and an enforced thread limit remain unmeasured. Fast replay does not establish live capture-to-command delay or a reliable 30 Hz robot loop. The 250 ms forecast falls between source frames and uses a disclosed nearest-frame reference, so it is not an exact-time accuracy result. Heuristic uncertainty envelopes remain uncalibrated. (Measured run and limitations)

Local validation recorded 378 repository tests and six browser workflow checks for this candidate. These are software/interface checks; no CI completion count, independent real-motion accuracy, live sensing or physical control result follows from them. Original DVIDIA motion, fixture and scoring code uses MIT, while existing upstream detector notices retain their separate terms. Next evidence is independently reviewed permitted motion footage, measured observation-only live streaming and calibrated geometry, followed by synchronized robot actions/state for interaction-conditioned prediction. (Reviewable candidate and validation, motion and interaction gates)

Compact CPU pilot learns the next actuator boundary

On 11 October 2026, a separate laptop-first source pilot learns seven next-20-ms actuator deltas from native episode tapes. Its 20 inputs are current joint positions, joint velocities and measured jaw gap, plus issued joint and jaw targets. Weighted ridge regression fits on training groups; development groups choose regularization. The selected model is saved before test-tape parsing and transition-target extraction. Authored phases, intervention names, configuration, future state, object/contact truth and final applied servo controls do not enter its feature vector. The measurements are privileged simulator state in sensor-shaped fields, not calibrated physical encoders or observations recovered from human footage. (Exact source commit 0d3f05f, feature contract and runnable commands, episode evidence)

The corpus records 36 attempts across 12 seeded native scene groups, each with nominal, open-jaw and release-after-lift commands. All sibling interventions stay together: eight groups train, two select on development and two form the test partition. The authored rigid-box controller generates every recording, with predeclared scene jitter and bounded mass/friction variation. All recordings complete; 24 terminal partial boundaries remain explicit exclusions. The test scores 4,553 retained transitions from six recordings, not 4,553 independent robot trials. The recording controller's 24 simulated successes and 12 horizon failures are not learned-model success rates. Manifest outcomes are available during preflight; the freeze declaration concerns unread test transition targets specifically. (Corpus and denominators, frozen model selection)

The action model improves joint and jaw RMSE over these baselines, but its jaw MAE and worst error regress against persistence. Constant joint velocity and training-mean delta are also retained in the full results; constant velocity predicts zero jaw delta because jaw velocity is not an input. On the nearly stationary open-jaw test case, action-model jaw RMSE is 0.057758 mm against 0.002761 mm for persistence. No mixed-unit accuracy percentage or universally better actuator behavior follows from the aggregate scores. (Exact final report and case/slice results)

An identical-state nominal/release pair around 5.8 simulation seconds exposes a larger transient problem. Both have the same before-state and joint request, while the jaw target changes from 33.538 mm to 80 mm. The action model predicts a 3.430 mm next-boundary opening, but the native observation opens only 0.0104 mm. The state-only model gives the same answer for both branches. This establishes command sensitivity while revealing a serious opening overprediction; it does not validate contact dynamics or qualify an action-conditioned world model. (Matched intervention receipt)

The action artifact contains 6,438 bytes. On the recorded macOS ARM64 host, 100 warmup calls followed by 1,000 measured calls give 0.207 ms p95 model computation, with 0.198 ms median and 0.232 ms worst. The timed call includes model validation and arithmetic over pre-extracted test features. It excludes camera capture, perception, feature extraction, transport, actuator response and cold saved-model loading; it is not robot reaction time or an operating-system deadline guarantee. Fitting uses NumPy, while saved-model inference uses Python's standard library. Corpus generation requires the optional native arm dependency; no GPU or model API participates. The process-lifetime memory high-water mark is 176.062 MiB, including earlier work, rather than an isolated learner requirement or minimum laptop specification. Local software validation records 424 passing tests, with no physical accuracy claim. (Measured runtime boundaries, open the offline interactive report)

This pilot adds a learned command-to-motion component to the Skillspace path. Reviewed footage can supply task structure and visual evidence; synchronized teleoperation must supply actual robot actions and measured state before a policy can learn corresponding movements. A future compact policy needs independent task benchmarks and a compatible robot/gripper adapter. ACT's action-chunking approach is one research candidate, and LeRobot documents conditional CPU checkpoint deployment with a measured control-rate requirement. Neither establishes the speed or shoe competence of a DVIDIA policy. (ACT research, LeRobot inference documentation)

The accompanying shoe collection plan begins with one shoe and one marked slot, a fixed calibrated camera/table, a verified grasp region and tucked laces. Its proposed 120-attempt, 12-pair workload and per-condition group quotas are study estimates, not known training sufficiency thresholds. Proposed 15-degree orientation, 20-mm position and one-second stability criteria need calibrated observable measurement procedures before validation. Real shoe evidence, physical sensing, a hardware adapter and validated shoe physics remain absent; the shoe benchmark stays blocked. The current predictor issues no commands and learns no shoe policy. Original DVIDIA coefficients and evidence are published under MIT, with no raw tapes, private footage, credentials or third-party pretrained weights in the bundle. The frozen Observation Lab v0.1 release and synthetic Hugging Face review demo remain separate from this source pilot. (Proposed capture plan, pipeline and remaining gates, artifact license)