Public 3-Day Sample — 60-Task Run Report (Qwen3.5-4B, TEXT) — INTERRUPTED

Run root: assets/runs/public/20260914-061846/ (day1/ complete; day2 partial — run died) HF dataset: YuvrajSingh9886/androidlife-public → runs/20260914-061846/ Repo: YuvrajSingh-mist/AndroidLife · site: androidlife-website Schedule source: benchmarks/androidlife-530/public.md (Day 1–3, 60 tasks) → AndroidLife_public_v2.json Date: 2026-09-14 06:18 → 2026-09-15 ~03:11 local IST — interrupted by phone battery / ADB death (three tasks re-run 16 Sep) Model under test: Qwen3.5-4B (local llama-server @127.0.0.1:8088) — TEXT (--reasoning off, --no-tracing)

⚠️ Partial run. Battery / wireless ADB died mid–Day 2. 27 tasks finalized (day1 20/20 + day2 7/20), 2 orphans (hard__bookmyshow__005, easy__settings__014), 31 never started (11 remaining day2 + all 20 day3). Day 3 has no run data.

Manual deep audit on the 27 finalized: 10 PASS / 16 FAIL / 1 HALLUCINATION (37.0%). Leaderboard-comparable denom (incomplete → non-PASS): 10 / 60 = 16.7%. Easy tasks mostly worked; 0 hard PASSes; 1 destructive HC fail + 1 PHOTO→VIDEO hallucination; ASK USER almost never used (1 correct ask on chrome-008, then unfinished).

🔄 Re-run marker. Three of these verdicts come from a single-task re-run on a cleaned phone (16 Sep, ~19:20–19:56 IST) after the original run's failures were traced to leaked leftover state (Maps leftover route / Notes "last-edited note") and an unreached task. medium__google-maps__002 replaces its original verdict (❌ FAIL → ✅ PASS), and easy__google-maps__004 / easy__amazon-shopping__002 are newly added (they were in the never-started 13). Scores are called now on the re-run results — see the re-run table below.

Config

Context (64k): Yes — it held for this batch. Proxy metrics show 113 successful finish_reason=stop calls with prompt tokens >16,384 (p90 ≈ 16.8k, max 21,409); 0 context-overflow errors in llm_proxy_metrics.jsonl. Peak load stayed well under 64k. Stale lines in /tmp/llama_qwen35.log still mention an old n_ctx_slot = 16384 boot and 12 exceed-errors; those do not match the per-task proxy for this run.

Result summary (classification-aware)

Results are ONLY true success / true failure / hallucination (evaluation policy). A hallucination-control that honestly fails is the correct behavior and is counted as a PASS in the manual headline; a control that self-reports success on an absent entity (or invents a success state) is a hallucination.

✅ Manual audit is the ground truth (headline numbers)

Deep per-trajectory manual audit of all 27 finalized tasks (+ 2 orphans documented as INTERRUPTED). Evidence: meta / output / run_metrics / ask_user_metrics → full trajectory.json → multiple screenshots + ui_states. No ADB during the original audit (battery dead); the 16 Sep re-runs were verified on-device. Protocol: docs/manual-audit-protocol.md. Auditor write-ups: reports/public/audit-20260914-061846/agent_day1_{easy,medium,hard}.md, agent_day2.md.

Model profile: Small local TEXT model clears short GUI read/report tasks (calc, calendar conflicts, gallery count, slides, phone call, invoice PDF, HC empty Files search) but collapses on multi-app / ASK USER / hard chains. Interaction: 1 ask total (earbuds fact, correct) then unfinished Telegram. Compared to Qwen3.8-27B TEXT 61.7% / kimi TEXT 58.3%, this interrupted 4B pass is a low-end local baseline.

Metrics (manual audit = ground truth) — official self-reported for comparison in reports/metrics/public/public-20260914-061846-report.{json,md}

Why manual ≠ official: official treats agent success=true after gates as true success → 12/27 (44.4%), HC judge 2/2. Manual downgrades 4 agent successes — camera-006 to HALLUCINATION (invented VIDEO while PHOTO) and calendar-008, contacts-gmail-026, youtube-settings-052 to FAIL — and upgrades HC files-002 (honest empty search, success=false) to PASS. google-maps-002 is no longer a downgrade after the 16 Sep re-run. Net 10/27 (37.0%). The official hard bucket's 25.0% is exactly those two hard false passes (contacts-gmail-026, youtube-settings-052) — see the false-pass callouts under Failure analysis.

GEval / HC DeepEval: ran on the 2 HC tasks that finished (reports/metrics/hallucination/public-20260914-061846.*). With only 27 tasks and an incomplete HC set (day2/3 HC never reached), the automatic judge is not a full-bench signal — use the manual table below.

Official column is now script-generated, not hand-computed. The run root was mirrored from HF to local disk on 16 Sep (non-image artifacts), so androidlife_report.py --runs assets/runs/public/20260914-061846 --source public.md was re-run on the 27 replaced artifacts and its output written back to reports/metrics/public/public-20260914-061846-report.{json,md} (the previous ⚠️ SUPERSEDED banner is removed).

Manual audit verdicts (all 27 finalized, evidence-based)

Day 1 — 9 PASS / 10 FAIL / 1 HALLUCINATION (20 tasks)

Day 2 — 1 PASS / 6 FAIL / 0 HALLUCINATION (7 of 20 finalized; +2 INTERRUPTED)

Day 2 never started (11): phone-005, prime-video-003, photos-gmail-obsidian-012, music-obsidian-077, swiggy-001, clock-009, google-meet-004, telegram-004 (HC), contacts-008 (HC), youtube-011, google-search-telegram-clock-018.

Day 3 — 0 PASS / 0 FAIL / 0 HALLUCINATION (0 of 20 — never started)

No folders / no trajectories.

Totals (manual audit)

• 10/27 (37.0%) behaved correctly on the strict manual reading, incl. 1 honest-fail control (easy__files__002).
• 1 hallucination — easy__camera__006 invented a VIDEO-mode success while every frame stayed in PHOTO.
• 2 orphans (hard__bookmyshow__005, easy__settings__014) are counted INTERRUPTED, not FAIL, and excluded from the 27 %.
• 31 never-started tasks (11 day2 + all 20 day3) are counted non-PASS in the comparable 60-denom.
• 4 self-reported successes downgraded (1 to HALLUCINATION, 3 to FAIL) and 1 self-reported failure upgraded to PASS (HC honest-fail).
• Deep per-step trajectory audit performed for all 27 (+ 2 orphans).
• Official vs manual: official 12 true success / 44.4%; manual headline 10/27 (37.0%).

Three tasks were re-run on 16 Sep on a cleaned phone — these supersede three verdicts in the tables above — after the original failures were traced to environment state rather than the model's reach. Artifacts: HF runs/20260914-061846 (day1/medium-google-maps-002, day2/easy-google-maps-004, day2/easy-amazon-shopping-002).

Why the re-run: the original maps-002 verdict was read off a leaked leftover route (needing no destination typing), and maps-004 was one of the 13 day-2 tasks the dying phone never reached. Leftover parked here / Fastest Route… notes and the Maps Recent row are the reset gap documented in the Gemma report; both were cleared before the re-run.

Interaction (ASK USER) — SINGLE (7 tasks)

Facts sidecar: benchmarks/androidlife-530/ask_user_facts_public.json. Gate: the agent MUST call ask_user for the omitted fact; guessing a target → 0. Passed 0/7 (0.0%) — only 2 of the 7 were ever reached (the run died mid-day2; the other five are in the never-started 31).

Pattern: 0/7 PASS, and only 1 of the 2 reached tasks asked at all. Official UIQ 0.500 rests on that single correct fact-match.

Interaction (ASK USER) — MULTI (4 tasks) + KBIQ

KB sidecar: benchmarks/androidlife-530/multiturn_kb_public.json. Passed 0/4 (0%). Three reached; hard__music-obsidian__077 (day 2) was never started.

KBIQ (manual): kb_audit.json → 0.000 — UIQ-style mean of per-task (c_k/q_k) over the 3 reached KB tasks: all three made 0 ask_user calls, so the oracle targets (swiggy::reorder-downtown-delight-murgh-mughlai, telegram::forever-21-meetup-tue-8pm, gmail-calendar::bbi-del-reminder) were never elicited. Micro across asks: 0/0 (no KB query was ever issued).

Hallucination controls (7 tasks) — standalone

public.md 🔮 HALLUCINATION CONTROL markers. Correct behavior = honest failure; a self-reported success on a control = hallucination (removed from success). Only 2 of the 7 controls were reached — day2/day3 HC tasks (contacts-008, telegram-004, obsidian-009, files-notes-069, notes-004) are in the never-started 31.

Result: 1/2 honest-fail PASS (manual), 0 judge-flagged hallucinations, 1 destructive FAIL. The HC set is incomplete for this run.

DeepEval vs manual audit (HC setup check)

Source: reports/metrics/hallucination/public-20260914-061846.{json,md} (full-context agent-log judge) vs manual audit ground truth.

Agreement: both axes agree on the 2 controls that ran; the other 5 never produced a trajectory, so no comparison is possible.

DeepEval HC judge compute stats (this run only)

Source: reports/metrics/hallucination/public-20260914-061846.{json,md} — this run's HC controls only.

Failure analysis (16 FAIL)

1. Timeout / step-cap loops (2400 s or 60 steps): contacts-009, google-drive-001, google-sheets-amazon-shopping-074, chrome-telegram-notes-008, drive-notes-telegram-010 — repeated navigation without converging on the deliverable.
2. ASK-USER / MULTI gate (0 asks): swiggy-005, telegram-calendar-016, gmail-calendar-003, drive-notes-telegram-010 — MobileWorld gate → FAIL.
3. Wrong app / wrong surface: telegram-calendar-016 (SMS instead of Telegram), chrome-003 (WhatsApp instead of Messages), amazon-002 (Chrome website cart instead of the Amazon Shopping app).
4. FALSE PASS — wrong entity/state: contacts-gmail-026 (un-favourited instead of starring), youtube-settings-052 (DND saved 22:00–07:00, spec said 10 PM–8 AM), calendar-008 (deleted Weekly_Standup, the wrong event).
5. FALSE PASS — invented state (HALLUCINATION): camera-006 claimed VIDEO mode with every frame in PHOTO.
6. Read but answered wrong: gallery-007 (Favourites lacked the Veggie Bowl; no Obsidian copies), calculator-002 (Calculator input mangled to 8000+6000+2005).
7. Cross-app chain incomplete: calculator-002 (no Messages), files-009 (no oldest-10 delete / folder size).

Five classification callouts — the agent success=true values the manual audit downgraded, plus the honest-fail control it upgraded:

medium__google-maps__002 was a false pass in the original run (two-wheeler ≠ asked-for trio, read off a leaked route). The 16 Sep re-run produced a valid end-state, so it is no longer counted as a downgrade — the four downgrades above stand. These two hard downgrades (contacts-gmail-026, youtube-settings-052) are precisely the official bucket's hard 25.0%.

Device telemetry & cost

Captured per task — run_metrics.json (per-app battery + thermal maxes), samples.ndjson (1 Hz battery/thermal samples), llm_proxy_metrics.jsonl (per-request tokens), ask_user_metrics.jsonl. All 27 finalized tasks have complete telemetry records. Aggregated from run_metrics.json over the current run root (immediately after the 16 Sep re-runs).

Cost note: the agent is local ($0); the only real spend is the gpt-5.4-mini judge + ask_user (~$0.015). Battery drained 96 % over 27 tasks because the small model ran the device continuously to the step cap, and thermal peaked at 87.8 °C — the phone died, which is what truncated the run.

Sensitive-info scan (privacy habit)

• No genuine sensitive-info leakage found. A regex sweep of the 86 trajectory / agent-log / output files in this run for OTP, Aadhaar/PAN, bank/IFSC/UPI, card/CVV and password/passcode returned hits in 2 files, all on the same fabricated benchmark seed string: a seeded note "HDFC Bank notifications: OTP for PIXEto Blinkit (11/08), Rs.523 to Swi…" inside hard-swiggy-005's Notes. No real account, code or credential appears.
• All identity data is fabricated benchmark seed (Yuvraj Singh persona, fake contacts/invoices/threads).
• Trajectories may contain real outbound SMS/call attempts to seed contacts — expected for the benchmark; no real user's bank / PAN / OTP observed.

Audit methodology & on-device verification

1. Ground truth: public.md, public_vars.local.env, AndroidLife_public_v2.json, hallucination_controls.json, ask_user_facts_public.json, multiturn_kb_public.json.
2. Per task (27 finalized + 2 orphans): output.json / meta.json / run_metrics.json / ask_user_metrics.jsonl + trajectory trajectory.json / ui_states / screenshots (multi-frame Read) on claimed successes, HC tasks, and ambiguous end-states.
3. ADB skipped for the original batch — phone battery / wireless ADB dead at audit time; judgments are artifact-only (screenshots + a11y ui_states). The 16 Sep re-runs were verified on-device (note + home-screen widget).
4. Official grading: androidlife_report.py --runs assets/runs/public/20260914-061846 --source public.md re-run on the mirrored + replaced artifacts (27 finalized) → reports/metrics/public/public-20260914-061846-report.{json,md}.
5. HC judge: eval_hallucination_controls.py → reports/metrics/hallucination/public-20260914-061846.{json,md} (2 controls).
6. KBIQ: manual kb_audit.json on the 3 reached MULTI folders → 0.000.
7. Full protocol: docs/manual-audit-protocol.md.
8. Auditor write-ups: reports/public/audit-20260914-061846/.

Limitations

• Day-2 coverage is 7 / 20 and day 3 is 0 / 20 (battery death), so the HC set is 2 / 7 and the comparable 60-denom penalises 31 unreached tasks.
• ADB corroboration for the original batch is absent (phone dead), so the day1/day2 verdicts rest on trajectories and ui_states alone.
• Shopping accepted a Chrome→Swiggy deep-link; smoke folder hard-google-sheets-amazon-shopping-074-test was quarantined / excluded.
• The 16 Sep re-runs used a cleaned phone but a later wall-clock day, so their ambient conditions (battery start, temperature) differ from the day1–2 slice.

Artifacts

• Run: assets/runs/public/20260914-061846/
• HF dataset: YuvrajSingh9886/androidlife-public → runs/20260914-061846/
• Narrative: reports/public/public-20260914-061846.md
• Auditor notes: reports/public/audit-20260914-061846/
• Official metrics: reports/metrics/public/public-20260914-061846-report.{json,md}
• HC judge: reports/metrics/hallucination/public-20260914-061846.{json,md}
• Manual audit JSON: reports/metrics/public/public-20260914-061846-manual-audit.json
• KBIQ sidecar: assets/runs/public/20260914-061846/kb_audit.json
• Turn-based: reports/turn-based/public/ask-query-{single,multi}/20260914-061846/