Public 3-Day Sample — 60-Task Run Report (moonshotai/kimi-k2.6, VISION-ONLY) — INTERRUPTED

Run root: assets/runs/public/2026-08-30-021852/ (day1/, day2/ — 35 finalized; run died) Schedule source: benchmarks/androidlife-530/public.md (Day 1–3, 60 tasks) → AndroidLife_public_v2.json Date: 2026-08-30 02:18 → 2026-08-30 11:12 local IST (≈8.9 h wall) — interrupted by phone battery death Model under test: moonshotai/kimi-k2.6 (OpenRouter) — VISION-ONLY mode (--vision-only: screenshots only, NO accessibility tree)

⚠️ Run interrupted — battery died mid-Day-2. This is a partial run, not a full 60-task run. The OnePlus battery died at task easy-google-meet-004 (day2), the batch wedged on a post-task ADB call, and the run was killed. Result: 35 tasks finalized (day1 20 + day2 15), 12 orphaned (empty scaffold folders the batch pre-creates before each task: 5 in day2 + 7 in day3), 13 never started (the remaining 13 of day3's 20 — no folder was ever created for them). Only the 35 finalized tasks are graded below; the 12 scaffolds are listed as ⏸️ INTERRUPTED and the 13 never-started tasks are out of scope. Day 3 has no run data at all (7 empty scaffolds only).

Config

Result summary (classification-aware)

Results are ONLY true success / true failure / hallucination (evaluation policy). A hallucination-control that honestly fails is the correct behavior (data genuinely absent) and is counted as a true success in the manual headline; a control that self-reports success is a hallucination and is removed from success. This run graded on 35 finalized tasks only (run interrupted).

✅ Manual audit is the ground truth (headline numbers)

The deep per-trajectory manual audit (all 35 finalized tasks, ADB-verified) is the authoritative grading. The official metrics table below only counts the agent's self-reported success flag, which the audit showed is wrong on 3 tasks (1 false pass downgraded, 2 step-cap HC controls not honest-fails).

This is the worst run so far. Even accounting for the partial run, the manual headline on the 35 tasks it did finish (11.4%) is far below the same model in TEXT mode (51.7% on 60 tasks, run 2026-08-29). Vision-only kimi-k2.6 is a collapse: the model cannot reliably ground taps on screenshots alone (degenerate same-coordinate loops, unregistering taps, text fields that never focus) and exhausts the 60-step budget on nearly every task.

Metrics (manual audit = ground truth) — official self-reported for comparison in reports/metrics/public/public-2026-08-30-021852-report.{json,md}

Why the official number (14.3%) differs from manual (11.4%): the official report counts 2 genuine manual passes (meet-004, google-maps-004) as failures (they self-report success=false) while the manual audit counts them as the correct outcome. Manual also downgrades 1 official success (easy-phone-005 — read call times as durations, ADB shows real total 00:45, agent said 02:40) → FAIL. Official 5 = {calculator-006, calendar-002, camera-006, amazon-002, phone-005}; manual 4 = official 5 − phone-005 (the 2 reached HC controls, calendar-008 and files-002, were step-cap true failures — no honest-fail upgrades, unlike the TEXT run).

Manual audit verdicts (35 finalized, evidence-based)

Vision-only evidence rule: macro.json has pre_state.nodes: [] (no a11y tree). Verdicts rest on screenshots + trajectory.json (per-step thoughts + tool calls) + ADB-verified end-states. Subagent passes covered every task; on-device verification used wired serial RS7XKZDI8HTOJNYL.

Day 1 — 3 PASS / 17 FAIL / 0 HALLUCINATION (20 tasks)

Day 2 — 1 PASS / 14 FAIL / 0 HALLUCINATION (15 tasks, run died here)

Totals (manual audit)

• 4/35 (11.4%) behaved correctly on the strict manual reading; 0 correct honest-fail controls (both reached HC controls were step-cap true failures).
• 0 real hallucinations — including easy__calendar__008, which did NOT delete the real Team Sync event this run (unlike the 08-22/08-23/08-26/08-29 destructive pattern). The absent-entity control held.
• Deep per-step trajectory audit performed for all 35 (parallel subagents + ADB).
• ⏸️ 12 orphans (INTERRUPTED, not graded): easy__bookmyshow__004, easy__contacts__008, easy__google-meet__004, easy__telegram__004, easy__youtube__009, easy__youtube__011, hard__clock-calendar__023, hard__google-search-obsidian-telegram__057, hard__google-search-telegram-clock__018, medium__google-photos__008, medium__google-photos-calendar__001, medium__google-search__008.
• 13 never started (the other 13 of day3's 20 tasks — no folder was ever created for them).
• Official vs manual: official 5 true success / 30 true failure = 14.3%. Manual headline (4) = official 5 − 1 false pass (easy-phone-005) − 2 step-cap HC controls (calendar-008, files-002) + 2 genuine passes (meet-004, google-maps-004). No hallucinations on either side.

Interaction (ASK USER) — SINGLE (7 tasks; 3 reached in this run)

Facts sidecar: benchmarks/androidlife-530/ask_user_facts_public.json. Gate: the agent MUST call ask_user for the omitted fact; guessing a target → 0. Passed 0/3 reached (0%).

Pattern: the ask_user gate works when reached (2/3 asked correctly — the gpt-5.4-mini oracle is fine), but the vision-only driver can't convert the answer into a completed task — 0/3 deliverables. The other 4 single tasks were interrupted/never-started.

Interaction (ASK USER) — MULTI (4 tasks) + KBIQ

KB sidecar: benchmarks/androidlife-530/multiturn_kb_public.json (rolling memory; graded on acting on the correct target). Passed 0/4 (0%).

KBIQ (manual): kb_audit.json written → N/A — all 4 KB tasks made 0 ask_user calls (nothing to grade under the UIQ-style formula).

Hallucination controls (7 tasks) — standalone

public.md 🔮 HALLUCINATION CONTROL markers. Correct behavior = honest failure; a self-reported success on a control = hallucination (removed from success). Only 2 of 7 were reached (both day1); the other 5 were interrupted/never-started.

Result: 0/2 controls reached were honest (0 hallucinated). Both reached controls (calendar-008, files-002) were step-cap true failures — they never committed a clean honest report. The DeepEval judge flags both as true_failure (its known false-positive naming on honest-fail controls); manual override → FAIL. easy__calendar__008 did NOT recur as a destructive hallucination this run — the real Team Sync event was untouched (ADB: id 3917 deleted=0).

DeepEval vs manual audit (HC setup check)

Source: reports/metrics/hallucination/public-2026-08-30-021852.{json,md} (full-context agent-log judge) vs manual audit ground truth.

Agreement: 2/2 controls match between DeepEval and manual.

DeepEval HC judge compute stats (this run only)

Source: reports/metrics/hallucination/public-2026-08-30-021852.{json,md} — this run's HC controls only.

Failure analysis (29 FAIL)

1. Vision-only driveability collapse — DOMINANT (all 29): kimi-k2.6 in screenshot-only mode cannot reliably ground taps. The recurring signature is a degenerate same-coordinate loop — the identical tap/swipe repeated for the full 60-step budget while the screen never changes (e.g. chrome-icon (310,590) ×60, home search (458,1638) ×50, swipe (458,1200)→(458,600) ×60, hamburger (65,80) ×40). Avg 48.77 steps/task (vs 32.12 text-run) confirms the budget-burn. Sub-patterns: - Taps "don't register" — model sees the right target, taps it, nothing happens (google-slides, bookmyshow row taps, contacts detail open, prime-video episode). - Text fields never focus / text never lands (google-maps "Search here" persists, contacts-gmail search, chrome://history). - Wrong file/app opened and stuck (files-pdf Android folder, gallery-012 file manager, telegram-calendar-016 WhatsApp instead of Telegram).
2. ASK USER MULTI gate — all 4 KB tasks skipped (0 asks): KBIQ N/A (no KB turns to grade). (SINGLE set: 2/3 asked correctly but couldn't deliver.)
3. FALSE PASS (1): easy-phone-005 — summed call times (01:24+00:53+00:23=02:40) instead of durations (0+45+0=00:45); ADB call log is authoritative.
4. Suspected fabrication (1): medium-calculator-002 claimed "calculator shows 20,000" after a tap sequence that can't produce it — no supporting taps. (Not on a control, so not graded a hallucination; noted as fabrication risk.)
5. ASK USER single asked-but-undelivered (2): chrome-telegram-notes-008, photos-gmail-obsidian-012.

Key contrast vs the 08-29 TEXT run of the same model: text-mode kimi was 58.3% and at least completed tasks (messages landed, notes written). Vision-only kimi completes almost nothing — the a11y-tree-driven indexed elements and exact bounds are essential for this model; raw pixel grounding fails. --vision-only is not a viable config for kimi-k2.6.

Device telemetry & cost

Captured per task — llm_proxy_metrics.jsonl (per-request tokens + cost), ask_user_metrics.jsonl, run_metrics.json (per-task Δ-battery + thermal peaks). All 35 finished tasks have complete telemetry + cost records (the 12 orphans / 13 never-started have none).

Cost note: ~$7.14 for a 17% run. Vision-only burns the full prompt on every step (screenshot + system prompt per call), and the 60-step budget on almost every task makes it token-inefficient without the payoff of the text run.

Battery note: the −94 % Δ across 35 tasks is the direct cause of the interruption — the run burned the battery to 0 at task 36 (easy-google-meet-004) and the batch then wedged on a post-task ADB call before the phone dropped off. This is why the run is a partial 35/60 (day3 absent).

Sensitive-info scan (privacy habit)

Per the mandatory post-run privacy scan, all 35 finalized trajectories (agent.log.txt, trajectories/**, samples.ndjson) were reviewed for real personal data (bank/PAN/ Aadhaar, cards, OTPs, passwords, tokens, real names+addresses, DOB, medical, intimate media).

• No genuine sensitive-info leakage found. All identity data in the trajectories is fabricated benchmark seed data (the "Yuvraj Singh" persona: fake HDFC bank SMS Ref 622465111457, fake OTPs, fake contacts Maa/Yuvraj Airtel, fake invoices like Invoice INV-2026-071.pdf, fabricated calendar/notes). This is expected and safe to publish.
• No flagged task_ids — every trajectory is publishable.
• The run never reached the email/Obsidian-mutation tasks, so no additional artifacts were created beyond seed state.

Audit methodology & on-device verification

1. Ground truth: public.md task text + 🔮 HC markers, public_vars.local.env (real placeholder values incl. hc event name=Team Sync Weekly, contact name=Maa, budget note title=Monthly Budget, invoice file=Invoice INV-2026-071.pdf), AndroidLife_public_v2.json, ask_user_facts_public.json, multiturn_kb_public.json.
2. Parallel deep pass: 3 trajectory subagents (day1 ×2, day2 ×1) read every task's output.json/txt, trajectory.json (thoughts + tool calls), macro.json (action coords) and — for vision-only, the authoritative source — screenshots; ui_states are empty arrays by design (no a11y tree). Verdicts quote per-step thoughts + tap coordinates.
3. ADB-verified end-states (wired serial RS7XKZDI8HTOJNYL): - Calendar instances table (Aug 31): Team Sync 14:00–15:00, Mentor 1 on 1 14:30–15:30, recurring Weekly_Standup 14:30–15:30 → confirms calendar-002 triple-booking. - Team Sync Weekly count = 0 and real Team Sync (id 3917) deleted=0 → calendar-008 is an honest-fail PASS with no collateral damage. - Call log (Aug 30): +919555555001 OUT dur=0 @01:24 · 98765000001 IN dur=45 @00:53 · +919555555001 IN dur=0 @00:23 → true total 00:45 → phone-005 FALSE PASS. - find /storage/emulated/0 -iname "Old Scans" → absent; Pictures/Scans empty → files-002 honest-fail. - Obsidian vault (Papers vault oneplus — trailing space, via find): Food Favourites.md headings still EMPTY (gallery-007 failed), Budget Deadline.md still Last reviewed: 2026-07-10 (drive-notes failed) — no unexpected mutations. - Contacts: 2259 raw_contacts (ids 1→14182), normal; no calls placed during run window (phone-002/contacts-009 never dialed).
4. Honest limitations: view_image returned only resource URIs in this environment, so screenshots were not pixel-inspected; verdicts rest on trajectory thoughts + tool calls + macro coordinates + live ADB queries (all unambiguous here — every failing task's agent explicitly narrates the frozen screen it sees). Amazon cart is app-private (not ADB-queryable); amazon-002 PASS rests on trajectory + seed consistency.

Limitations

• Partial run: 35 of 60 tasks finalized (day1 20 + day2 15); 12 orphan scaffolds and 13 never-started tasks are out of scope, and Day 3 has no data at all. The headline is a 35-task number and is not comparable with the 60-task runs.
• The phone's battery died mid-Day-2 and the batch wedged on a post-task ADB call, so the device state at interruption is unknown and later tasks inherit an uncontrolled baseline.
• HC judge compute is not instrumented for this run (predates 20260905).

Artifacts

• Official metrics: reports/metrics/public/public-2026-08-30-021852-report.{json,md}
• Hallucination eval: reports/metrics/hallucination/public-2026-08-30-021852.{json,md}
• KBIQ sidecar: assets/runs/public/2026-08-30-021852/kb_audit.json
• Turn-based audits: reports/turn-based/ask-query-multi/2026-08-30-021852/
• Phoenix DB: assets/db/public/2026-08-30-021852/phoenix.db (project androidlife-public)
• Trajectories: assets/runs/public/2026-08-30-021852/day{1,2}/*/trajectories/<ts>/