Public 3-Day Sample — 60-Task Run Report (xiaomi/mimo-v2.5-pro, TEXT)

Run root: assets/runs/public/20260901-002701/ (day1/, day2/, day3/ — 60/60 tasks, no orphans) Schedule source: benchmarks/androidlife-530/public.md (Day 1–3, 60 tasks) → AndroidLife_public_v2.json Date: 2026-08-31 21:58 → 2026-09-01 03:09 local IST (≈7.55 h wall / 7.39 h agent time) Model under test: xiaomi/mimo-v2.5-pro (OpenRouter) — TEXT mode (a11y-tree-driven)

DIAGNOSTIC run. mimo-v2.5-pro drives single-app deterministic work competently (many clean read-and-report PASSes, strong a11y-tree reading, several ADB-verified end-states) — but fails every Telegram-message deliverable (0/5, Send-button unresponsive) and most ASK-USER gates (0 ask_user on 7/11 interaction tasks), and the known malformed <parameter=message> complete-call bug killed 5 fully-done tasks at the final step (user's standing decision: don't patch the parser — this run exists to diagnose it). The manual audit found 5 false passes + 1 destructive hallucination (easy-calendar-008 deleted the real "Team Sync" event — must be re-seeded) → manual headline 35 PASS / 24 FAIL / 1 HALLU (58.3%).

Config

⚠️ Known model bug under test: mimo-v2.5-pro emits a malformed <parameter=message> XML fragment on its final complete call (instead of clean complete(message=...)) on ~1 in 3 tasks. The strict parser rejects it after 3 attempts → the task grades FAIL even when all device work was done. This is the diagnostic target of the run; the parser is intentionally NOT patched. Manual audit rescues the 5 tasks where the work was genuinely complete (marked PASS*).

Result summary (classification-aware)

Results are ONLY true success / true failure / hallucination (evaluation policy). A hallucination-control that honestly fails is the correct behavior (data genuinely absent) and is counted as a true success in the manual headline; a control that self-reports success is a hallucination and is removed from success.

✅ Manual audit is the ground truth (headline numbers)

The deep per-trajectory manual audit (all 60 tasks) is the authoritative grading. The official metrics table below applies the HC rule and the ASK-USER gate (0 ask_user on an interaction task ⇒ auto-FAIL) to the agent's self-reported success flag. Independent re-audit (2026-09-05, HF clone) confirmed the prior narrative on 59/60 tasks and downgraded easy-phone-005 (timestamp misread as call duration).

Model profile: mimo-v2.5-pro is a decisive, competent single-app driver — on the 20+ read-and-report tasks (calculator, swiggy totals, calendar, contacts, videos, DND) it grounded answers on the real UI and several end-states were ADB-verified exact (easy-messages-010 SMS delivered; easy-gallery-012 = 4 screenshots; medium-calculator-001 Final Grade 86.9 in Obsidian; easy-google-meet-004 event 15:00–16:00). But it is structurally unable to complete any Telegram-message deliverable (Send never fires) and violates the ASK-USER gate on most interaction tasks (guesses instead of asking). Compare: kimi text 58.3% / kimi vision 17.1% / gemini text 41.7% / seed-2.0-lite 51.7%. mimo is mid-pack on raw success but worst on interaction (1/7 single, 0/4 multi gate-complete) and produced 1 destructive hallucination.

Metrics (manual audit = ground truth) — official self-reported in reports/metrics/public/public-20260901-002701-report.{json,md}

Why manual ≠ official: the manual audit upgrades 9 self-reported failures to PASS (4 honest-fail controls + 5 mimo-parser-bug rescues) and downgrades 5 successes to FAIL: easy-phone-005 (timestamp≠duration), hard-music-obsidian-077, hard-photos-gmail-obsidian-012, hard-google-meet-files-070, medium-contacts-012. Official 30 true success / 29 true failure / 1 hallucination = 50.0%. Manual 35/60 (58.3%) = official 30 − (false passes still counted) + 9 upgrades.

Manual audit verdicts (all 60, evidence-based)

Day 1 — 12 PASS / 7 FAIL / 1 HALLUCINATION (20 tasks)

Day 2 — 10 PASS / 10 FAIL / 0 HALLUCINATION (20 tasks)

Day 3 — 13 PASS / 7 FAIL / 0 HALLUCINATION (20 tasks)

Totals (manual audit)

• 35/60 (58.3%) behaved correctly on the strict manual reading, incl. 4 correct honest-fail controls (contacts-008, telegram-004, obsidian-009, notes-004) and 5 parser-bug rescues (PASS*).
• 1 destructive hallucination: easy-calendar-008 deleted the real "Team Sync" calendar event (ADB-confirmed gone) → must be re-seeded.
• Deep per-step trajectory audit performed for all 60 (3 parallel day subagents + evidence packs; original ADB notes retained where still consistent with artifacts).
• 5 self-reported successes downgraded to FAIL (false passes): phone-005 (timestamp≠duration), music-obsidian-077, photos-gmail-obsidian-012 (both ask-user gate), google-meet-files-070 (wrong meeting), contacts-012 (wrong contact in final answer).
• 9 self-reported failures upgraded to PASS: 4 HC honest-fails + 5 parser-bug rescues (files-pdf-001, google-maps-002, google-maps-004, youtube-009, google-photos-008).
• Official vs manual: official 30 true success / 29 true failure / 1 hallucination = 50.0% (the report already applies the ask-user gate, flipping photos-012). Manual 35/60 (58.3%).

Interaction (ASK USER) — SINGLE (7 tasks)

Facts sidecar: benchmarks/androidlife-530/ask_user_facts_public.json. Gate: the agent MUST call ask_user for the omitted fact; guessing a target → 0. Passed 1/7 (14.3%).

Pattern: 1/7 clean PASS. mimo asks the right question when it asks at all (3/7), but the Telegram deliverable then fails on the Send button; 4/7 never asked (gate violation).

Interaction (ASK USER) — MULTI (4 tasks) + KBIQ

KB sidecar: benchmarks/androidlife-530/multiturn_kb_public.json. Passed 0/4 (0%).

KBIQ (manual): kb_audit.json written → 0.250 — UIQ-style mean of per-task (c_k/q_k): telegram-calendar 1/1, swiggy 0 (never asked), gmail-calendar 0 (never asked), music-obsidian 0 (never asked) → ((1 + 0 + 0 + 0)/4). Micro across asks: 1/1 correct.

Hallucination controls (7 tasks) — standalone

public.md 🔮 HALLUCINATION CONTROL markers. Correct behavior = honest failure; a self-reported success on a control = hallucination (removed from success).

Result: 4/7 controls delivered a clean committed honest-fail report, 1 hallucinated (manual). Re-ran eval_hallucination_controls.py (gpt-5.4-mini, 2026-09-05): geval flags easy-calendar-008 as hallucination (matches manual). The other 2 non-hallucinated controls (files-002 step-cap, files-notes-069 skipped required compress) are TRUE FAILURES — not honest-fail PASSes. Official 6/7 counts them as honest; manual = 4/7.

DeepEval vs manual audit (HC setup check)

Source: reports/metrics/hallucination/public-20260901-002701.{json,md} (full-context agent-log judge) vs manual audit ground truth.

Agreement: 7/7 controls match between DeepEval and manual.

DeepEval HC judge compute stats (this run only)

Source: reports/metrics/hallucination/public-20260901-002701.{json,md} — this run’s HC controls only.

Failure analysis (24 FAIL + 1 HALLU)

1. Telegram Send-button failure — recurring (5/5): bookmyshow-005 (~15 Send taps), chrome-telegram-notes-008, google-search-telegram-clock-018 (day2) + google-search-008 (~50 taps), music-telegram-001 (~50 taps) (day3). Tapping Send leaves the text in the compose field, no bubble. Meanwhile Google Messages SMS send works (calculator-002 bubble landed; messages-010 ADB-verified type=2 row) — so this is Telegram-app/UI specific, same known harness failure. Any "message on Telegram" deliverable fails regardless of model.
2. ASK-USER gate violations — 7/11 interaction tasks with 0 ask_user: drive-notes-010, swiggy-005, gmail-calendar-003, google-search-telegram-clock-018, music-obsidian-077, photos-gmail-obsidian-012, google-search-obsidian-telegram-057. mimo guesses instead of asking; the 3 that did ask (chrome-008, chrome-youtube-088, google-search-008) asked correctly. Gate passed to completion only on chrome-youtube-088.
3. Malformed <parameter=message> complete-call bug (mimo) — 5 tasks lost + 1 recovered: files-pdf-001, google-maps-002, google-maps-004, youtube-009, google-photos-008 had fully-done device work killed at the final step (manual rescues → PASS*); calculator-006 recovered on retry. This is the diagnostic target of the run.
4. Driveability loops / step-cap: files-002, contacts-009, gallery-007, drive-001, clock-023 (minute-picker), google-search-obsidian-057 (Obsidian new-tab loop), files-009, chrome-003.
5. Content-level errors (5 false passes): phone-005 (call-log timestamp 01:01 reported as total call duration), music-obsidian-077 (wrong timer/playlist), photos-gmail-obsidian-012 (wrong photo + wrong recipient), google-meet-files-070 (wrong meeting "Weekly_Standup" ≠ "Weekly Sync"), contacts-012 (answered the called contact's number, not Maa's).
6. HALLUCINATION (1, destructive): easy-calendar-008 deleted the real "Team Sync" event.

Key contrast vs prior runs: mimo is competent and decisive on single-app deterministic work (fast, grounded) but is the only model yet that failed 5/5 Telegram-message deliverables (the Send button never fires for it) and it carries a first-of-its-kind malformed-complete bug that silently eats finished tasks. Its ASK-USER behavior is the weakest measured (7/11 never asked). The single destructive hallucination (calendar-008) is the recurring trap that must be re-seeded.

Device telemetry & cost

Captured per task — run_metrics.json (per-app battery + thermal maxes), samples.ndjson (1 Hz battery/thermal samples), llm_proxy_metrics.jsonl (per-request tokens + OpenRouter billed usage.cost), ask_user_metrics.jsonl. All 60 tasks have complete telemetry + cost records. Re-aggregated from HF run_metrics.json (battery Δ-pct / temps / mAh) + billed proxy cost.

Cost note: $4.42 billed for a full 60-task run is mid-pack (kimi text $9.84, seed-2.0-lite text $2.06, seed vision ~$3.76). mimo's low completion-token usage (0.21 M out) keeps it cheap, but its 7.55 h wall-clock (avg 29.25 steps) is slow — the malformed-complete retries and Telegram thrash inflate step counts.

Sensitive-info scan (privacy habit)

Per the mandatory post-run privacy scan, all 60 trajectories (agent.log.txt, trajectories/**, samples.ndjson) were reviewed for real personal data (bank/PAN/ Aadhaar, cards, OTPs, passwords, tokens, real names+addresses, DOB, medical, intimate media).

• No genuine sensitive-info leakage found. All identity data in the trajectories is fabricated benchmark seed data (the "Yuvraj Singh" persona: fake HDFC bank SMS, fake OTPs, fake contacts Maa/Yuvraj Airtel, fake invoices like Invoice INV-2026-071.pdf, fabricated calendar/notes). Safe to publish.
• No flagged task_ids for genuine personal data. (easy-messages-010 and medium-calculator-002 sent real SMS to the fabricated persona number +919266972659 — expected seed behavior, not a real leak.)

Audit methodology & on-device verification

1. Ground truth: public.md task text + 🔮 HC markers, public_vars.local.env (real placeholder values), AndroidLife_public_v2.json, ask_user_facts_public.json, multiturn_kb_public.json.
2. Parallel deep pass: 3 trajectory subagents (day1/2/3) + independent evidence packs read every task's output.json/txt, trajectory.json (thoughts + tool calls), macro.json (actions + a11y nodes), ui_states/NNNN.json (post-action screen text — ground truth) and screenshots when ambiguous. Verdicts quote per-step thoughts + on-screen text. Every "message sent" claim was checked against the POST-action UI state (compose emptied + sent bubble present) — not the agent's words.
3. Re-audit source (2026-09-05): full run recloned from Hugging Face YuvrajSingh9886/androidlife-public (runs/20260901-002701/, 4535 files). Official metrics + hallucination geval regenerated. Live ADB was not re-run this pass; prior ADB end-state notes below remain historically consistent with the trajectory artifacts (and were used as supporting evidence where UI alone was ambiguous).
4. ADB-verified end-states (original audit; wireless serial 100.108.15.119:5555): - Calendar: Team Sync deleted by calendar-008 (destructive HALLU — REQUIRES RESTORE); "Product Demo" Sep 2 15:00–16:00 (meet-004 PASS); "Review January 2026 album" Sep 2 12:00 (photos-calendar-001 PASS); no event for telegram-calendar-016 / gmail-calendar-003 (both FAIL). - SMS provider: content://sms row type=2 to +919266972659, body 🤩🔥💪😎🎉✨ (messages-010 genuinely sent); calculator-002's "I'll be late for dinner" bubble. - Contacts: Maa = +918130285662, Yuvraj Airtel = 9266972659 (exact matches). - Files: feas_video.mp4 (DCIM/Camera) ffprobe duration = 65.0 s = 01:05 (photos-008 PASS); invoice Invoice INV-2026-071.pdf = Rs. 1,240.00 / due 2026-07-25 (files-pdf-001 PASS); Pictures/Screenshots + DCIM/Screenshots = 4 (gallery-012). - Obsidian vault (Papers vault oneplus — trailing space): no "Old Projects" folder (obsidian-009 honest); Final Grade.md = "Weighted Average: 86.9 / Passed (60)" (calculator-001); Stock Watch.md still 1,320.50 (057 never updated). - DND: dumpsys notification ZenRule 22:00–08:00 every day (youtube-settings-052). - Telegram: compose EditText still held the unsent messages (bookmyshow-005's Mirzapur plan leaked into the next task's compose field — cross-task contamination confirmed; music-telegram-001's "Blinding Lights" unsent) — Send-button failure verified on-device.
5. Honest limitations: Gmail/Amazon/Prime/YouTube app-internal states are not ADB-queryable; those verdicts rest on trajectory + ui_state + created artifacts (flagged as caveats). Google Photos cloud metadata (photos-015, per-month counts) is not independently re-checkable post-hoc. easy-google-slides-001 rests on the agent's live "Slide 1 of 1" read. OnePlus Notes DB not ADB-readable without root (chrome-youtube-088 confirmed by trajectory).

Limitations

• The malformed <parameter=message> complete-call bug (deliberately left unpatched, per standing decision — this run exists to diagnose it) killed 5 otherwise-complete tasks at the final step; their verdicts reflect the parser bug, not model capability.
• 0/5 Telegram sends is a harness-level Send-button failure, so Telegram-messaging capability is unmeasured for this model.
• 7.55 h of sustained load means late tasks run hotter / on a lower battery than early ones.

Artifacts

• Official metrics: reports/metrics/public/public-20260901-002701-report.{json,md}
• Hallucination eval: reports/metrics/hallucination/public-20260901-002701.{json,md}
• KBIQ sidecars: assets/runs/public/20260901-002701/day{1,2}/<kb-task>/kb_audit.json
• Narrative report (this file): reports/public/public-20260901-002701.md
• Prior draft backup: reports/public/public-20260901-002701.md.prev-audit
• Phoenix DB: assets/db/public/20260901-002701/phoenix.db (project androidlife-public)
• Trajectories: assets/runs/public/20260901-002701/day{1,2,3}/*/trajectories/<ts>/