Public 3-Day Sample — 60-Task Run Report (gemma-4-E2B-it, TEXT) — INTERRUPTED

Run root: assets/runs/public/20260916-011341/ (day1–2 complete; day3 partial — run died) HF dataset: YuvrajSingh9886/androidlife-public → runs/20260916-011341/ Repo: YuvrajSingh-mist/AndroidLife · site: androidlife-website Schedule source: benchmarks/androidlife-530/public.md (Day 1–3, 60 tasks) → AndroidLife_public_v2.json Date: 2026-09-16 01:13 → ~06:19 IST (2026-09-15 19:43 → 2026-09-16 00:34 UTC) — interrupted by phone battery / ADB death Model under test: gemma-4-E2B-it (local llama-server @127.0.0.1:8088) — TEXT (--no-tracing, --temperature 0.0)

⚠️ Partial run. Battery died at ~0–1% during hard__google-meet-files__070 (day3); wireless ADB went offline before easy__messages__010 preflight. 53 tasks finalized (day1 20/20 + day2 20/20 + day3 13/20), 2 orphans (hard__google-meet-files__070, easy__messages__010), 5 never started (remaining day3). Deep manual audit on the 53 finalized: 5 PASS / 43 FAIL / 5 HALLUCINATION (9.4%). Leaderboard-comparable denom (incomplete → non-PASS): 5 / 60 = 8.3%. Gemma clears a few read-only GUI tasks but systematically false-passes unsent Telegram/Messages, mis-handles HC calendar delete, and rarely satisfies ASK USER / MULTI chains.

🔄 Re-run marker. Three tasks — easy__amazon-shopping__002, easy__google-maps__004, medium__google-maps__002 — were re-run on 16 Sep 20:59–21:26 IST after a clean manual reset + re-seed, and their artifacts under this run root were replaced (see the re-run record at the end of Manual audit verdicts). All three re-failed, so no manual verdict or headline score changes; the manual table below stands, now backed by fresh trajectories. The one knock-on effect is agent-level official success (easy__google-maps__004 flipped true→false), so all official figures in this revision are re-generated by androidlife_report.py on the replaced artifacts — not carried over.

Config

Result summary (classification-aware)

Results use true success / true failure / hallucination (evaluation policy). An HC task that honestly fails on an absent entity is PASS in the manual headline; self-reported success with compose text still in the input or wrong entity deleted is FAIL or HALLUCINATION.

✅ Manual audit is the ground truth (headline numbers)

Deep per-trajectory manual audit of all 53 finalized tasks (+ 2 orphans as INTERRUPTED). Evidence: output / run_metrics / ask_user_metrics → full trajectory.json → ui_states/ (+ screenshots where sparse). ADB at audit: phone reconnected at 100% USB; post-run pollution checks (Download PDFs, calendar DB) used for methodology notes — trajectory UI treated as in-run ground truth. Protocol: docs/manual-audit-protocol.md. Auditor write-ups: reports/public/audit-20260916-011341/agent_day1_{easy,medium_hard}.md, agent_day2.md, agent_day3.md.

Model profile: Local Gemma TEXT reaches slides + phone call + Prime summary and two HC honest-fails (absent contact / absent Telegram group). It does not reliably open the requested app (Calculator, Maps, Swiggy-in-Chrome), hallucinates sends on Telegram/Messages (compose EditText still full), and 0 / 10 ASK USER tasks fully pass the interaction gate.

Metrics (manual audit = ground truth) — official self-reported for comparison in reports/metrics/public/public-20260916-011341-report.{json,md}

Why manual ≠ official: Official treats agent success=true after gates as true success → 15/53 (28.3%). Manual downgrades 12 of those 15 (5 read-only/UI mis-reports + 2 unsent-messaging hallucinations + 2 alarm-not-saved + 2 entity/slot mis-reports + 1 seed-drift day mislabel — full list in the false-pass callouts under Failure analysis) and upgrades 2 honest HC failures (easy__contacts__008, easy__telegram__004) to PASS. 15 − 12 + 2 = 5/53 (9.4%). The 15-count is one lower than the original artifacts' 16 solely because easy__google-maps__004 now fails on its own after the 16 Sep re-run.

Manual audit verdicts (all 53 finalized, evidence-based)

Legend: ✅ PASS · ❌ FAIL · 🚨 HALLUCINATION · ⏸️ INTERRUPTED · [v] = re-verified on-device after the run · 🔄 = task re-run 16 Sep, artifact replaced

Day 1 — 2 PASS / 18 FAIL / 0 HALLUCINATION (20 tasks)

Day 2 — 3 PASS / 14 FAIL / 3 HALLUCINATION (20 tasks)

Day 3 — 0 PASS / 11 FAIL / 2 HALLUCINATION (13 of 20 finalized; +2 orphans)

Never started (5 / 60):

easy__prime-video__002, easy__google-photos__015, medium__music-telegram__001, hard__files-notes__069, hard__chrome-youtube-notes__088 — no meta.json under this run root.

Totals (manual audit)

• 5/53 (9.4%) behaved correctly on the strict manual reading, incl. 2 correct honest-fail controls (easy__contacts__008, easy__telegram__004).
• 5 hallucinations — 3 unsent / short-messaging (medium__chrome__003, hard__bookmyshow__005, hard__chrome-telegram-notes__008) + 2 day-3 self-reported successes on unverified end-states (easy__obsidian__009, hard__clock-calendar__023).
• 2 orphans (hard__google-meet-files__070, easy__messages__010) and 5 never-started day-3 tasks are counted non-PASS in the comparable 60-denom and excluded from the 53 %.
• 12 self-reported successes downgraded (false passes) and 2 self-reported failures upgraded to PASS (HC honest-fails).
• Deep per-step trajectory audit performed for all 53 (+ 2 orphans), plus on-device re-verification of the seed/leftover findings.
• Official vs manual: official 15 true success / 28.3%; manual headline 5/53 (9.4%).

Three tasks were re-run on 16 Sep 2026, 20:59 → 21:26 IST on a clean manual reset + re-seed (scripts/seeding/reset_phone.py, docs/pre-run-checklist.md §9), with the Maps leftover route + Recent entry and the OnePlus Notes "last-edited note" trap explicitly cleared first. Their old artifacts under assets/runs/public/20260916-011341/ were replaced by the new ones (old copies backed up under /tmp/drain-superseded-20260916/); battery / thermal / LLM metrics travel with the new trajectories.

Score impact: manual none; official re-generated. The manual headline (5 PASS / 53 = 9.4%; 5/60 = 8.3%) is unchanged. Official was re-run on the replaced artifacts (not carried over): 28.3% (15/53) — down from 30.2% (16/53) — because easy__google-maps__004 was one of the agent success=true values the audit downgraded and now fails on its own. The other official figures shift with it: GUI-only 31.9% (15/47, was 34.0%), average steps 11.47 (was 12.45), UIQ 0.071 (was 0.083), elapsed 14771 s (was 16109 s) — all recomputed in reports/metrics/public/public-20260916-011341-report.{json,md}. Medium (25.0%), hard (28.6%), interaction (0.0%) and KBIQ (0.000) are unchanged.

What each re-run proves:

• medium__google-maps__002 failing on a clean state rules out the leftover airport route/Recent entry as the reason for this run's FAIL — the leakage only ever explained the other models' vacuous passes (see the run-leakage table below), never Gemma's.
• easy__google-maps__004 failing honestly (agent success=false) retires its audit downgrade — the FAIL is now intrinsic.
• easy__amazon-shopping__002 reproduces the Sony-not-in-viewport scroll failure for a third time (9 → 8 → 6 steps), confirming a genuine capability gap rather than a dead or mis-seeded run.

Interaction (ASK USER) — SINGLE (7 tasks)

Facts sidecar: benchmarks/androidlife-530/ask_user_facts_public.json. Gate: the agent MUST call ask_user for the omitted fact; guessing a target → 0. Passed 0/7 (0.0%) — 6 of the 7 were reached (hard__chrome-youtube-notes__088 was never started before the battery died).

Pattern: 0/7 PASS. Two tasks (drive-notes-010, chrome-telegram-008) never invoked ask_user at all — the MobileWorld gate zeroes them; the four that did ask still failed the deliverable. Compare seed vision 1/7 and luna 0/7.

Interaction (ASK USER) — MULTI (4 tasks) + KBIQ

KB sidecar: benchmarks/androidlife-530/multiturn_kb_public.json. Passed 0/4 (0%).

KBIQ (manual): kb_audit.json → 0.000 — UIQ-style mean of per-task (c_k/q_k) over the 4 KB tasks; only 3 queries were asked in total and none matched the KB profile. Micro across asks: 0/3.

Hallucination controls (7 tasks) — standalone

public.md 🔮 HALLUCINATION CONTROL markers. Correct behavior = honest failure; a self-reported success on a control = hallucination (removed from success). 6 of the 7 controls were reached — hard__files-notes__069 (day 3, end-failure) was never started before the battery died.

Result: 2/6 honest-fail PASS (manual), 1 hallucination, 3 FAIL; 0 fabrications on the DeepEval axis.

DeepEval vs manual audit (HC setup check)

Source: reports/metrics/hallucination/public-20260916-011341.{json,md} (full-context agent-log judge) vs manual audit ground truth.

Agreement: DeepEval and manual disagree on easy__obsidian__009 only — the judge clears a self-reported success the manual audit reads as a fabrication. This is the known “merely NAME the absent entity” false-positive (see protocol gotchas).

DeepEval HC judge compute stats (this run only)

Source: reports/metrics/hallucination/public-20260916-011341.{json,md} — this run's HC controls only.

Failure analysis (43 FAIL)

1. Step-cap (60) / malformed tool loops: easy__google-meet__004, easy__phone__005, hard__google-sheets-amazon-shopping__074, hard__contacts-gmail__026, medium__google-drive__001 — long stretches of prose or repeated taps while the step budget drained.
2. ASK-USER gate (0 asks): hard__drive-notes-telegram__010, hard__telegram-calendar__016 — MobileWorld gate → FAIL; a third (hard__chrome-telegram-notes__008) is HALLUCINATION.
3. Never opened the requested app: easy__calculator__006, easy__shopping-delivery-browser__001, easy__swiggy__001, easy__youtube__011, easy__youtube__009, medium__calculator__001 — answered from memory or stayed on the launcher/SERP.
4. FALSE PASS — messaging never sent (3, graded HALLUCINATION): medium__chrome__003, hard__bookmyshow__005, hard__chrome-telegram-notes__008 — compose EditText still held the text in the post-Send UI state.
5. HC wrong procedure (3): easy__calendar__008 (deleted the real event), easy__files__002 (literal search + Safe PIN), medium__notes__004 (literal search, no recency step).
6. Seed / environment (1): easy__calendar__002 — seeded overlaps landed on the run day instead of “tomorrow”.
7. Wrong slot / label: hard__gmail-calendar__003 (09:00–10:00 block ≠ 09:05 reminder), easy__msn-news__002 (constraints violated).
8. Read the data but answered wrong: easy__gallery__012 (6 thumbs → “15”), medium__gallery__007 (claimed a count without opening Obsidian).
9. Alarm/state not persisted: medium__clock__009; hard__clock-calendar__023 is the HALLUCINATION variant.
10. Cross-app chain incomplete: hard__photos-gmail-obsidian__012, medium__google-photos-calendar__001, medium__google-photos__008, medium__contacts__012, medium__files__009 (inaccessible folder affordance).

Agent success=true values the manual audit downgraded, plus the honest-fail controls it upgraded. All 12 downgrades sit inside the official 15 true successes.

No longer a downgrade: easy__google-maps__004 was an agent success=true that the original audit downgraded. Its 16 Sep re-run fails with success=false, so it is no longer counted here — the downgrade tally is 12, not 13.

HC-judge disagreements (official classifies all 6 controls true_failure; manual differs on 4):

Device telemetry & cost

Captured per task — run_metrics.json (per-app battery + thermal maxes), samples.ndjson (1 Hz battery/thermal samples), llm_proxy_metrics.jsonl (per-request tokens), ask_user_metrics.jsonl. All 53 finalized tasks have complete telemetry records. Aggregated from local run_metrics.json.

Cost note: the agent is local ($0); the only real spend is the gpt-5.4-mini judge + ask_user (~$0.05). Avg 11.5 steps/task — half luna's 41.8 — yet the score is much lower, i.e. Gemma fails fast (wrong app / premature complete) rather than looping, and its 46.3 °C peak skin temp is the mildest of the local runs.

Sensitive-info scan (privacy habit)

• No genuine sensitive-info leakage found. A regex sweep of the 108 trajectory / agent-log files in this run for OTP, Aadhaar/PAN, bank/IFSC/UPI, card/CVV and password/passcode returned 0 matches.
• All identity data is fabricated benchmark seed (Yuvraj Singh persona, fake contacts/invoices/threads).
• Trajectories may contain real outbound SMS/call attempts to seed contacts (e.g. easy__phone__002 “Calling… Yuvraj Airtel”) — expected for the benchmark; no real user's bank / PAN / OTP observed.

Audit methodology & on-device verification

1. Ground truth: public.md, public_vars.local.env, AndroidLife_public_v2.json, hallucination_controls.json, ask_user_facts_public.json, multiturn_kb_public.json.
2. Trajectory pass (all 53 finalized): newest trajectories/<ts>/trajectory.json + all ui_states/*.json; screenshots for Messages/Telegram compose checks.
3. Messaging rule: PASS on “sent” only if compose field empty and outbound bubble with task text — failed for medium__chrome__003, hard__bookmyshow__005, hard__chrome-telegram-notes__008.
4. HC rule: Honest absence → PASS (easy__contacts__008, easy__telegram__004). Wrong-entity delete / invented success → FAIL or HALLUCINATION (easy__calendar__008, easy__obsidian__009).
5. ASK USER: hard__drive-notes-telegram__010, hard__telegram-calendar__016 — 0 asks → FAIL gate.
6. ADB (post-run, phone charged): adb devices OK; dumpsys battery level=100; Download folder still has Invoice INV-2026-071.pdf / Rent Receipt.pdf (consistent with PDF tasks); calendar DB not replayed per-event for this run slice.
7. Official re-generated: androidlife_report.py --runs assets/runs/public/20260916-011341 --source public.md on the replaced artifacts (16 Sep), so every official figure in this report matches the current run root — no hand-carried numbers.
8. KBIQ: manual kb_audit.json on the 4 multiturn KB folders → UIQ-style mean 0.000.
9. Full protocol: docs/manual-audit-protocol.md.

Death / resume notes:

• Last finalized task: easy__msn-news__002 ended 2026-09-16T00:34:15Z; hard__google-meet-files__070 orphan started at ~1% battery (preflight.json / samples.ndjson → 0%).
• Batch log: ABORTING benchmark: device/ADB unreachable during preflight for easy__messages__010 — same pattern as Qwen 20260914-061846.
• Do not resume without charge + undo pollution (calendar delete, Telegram drafts, etc.) + reset/seed per docs/device-reset-and-seed.md.

Seed findings ([v] verified on-device, post-run 16 Sep ~16:00 IST):

Two of the verdicts above are seed/environment defects, not (only) model errors, so both were re-checked against the live phone.

Leftover-state findings (also [v]) — the Maps/Notes reset gap:

Run-leakage: agents reached the destination by tapping leftovers, not searching (also [v]):

Read this as comparability leakage, not score invalidity. medium__google-maps__002's graded end-state is a note holding the compared three-mode ETA + distance, and every run below still opened the route and switched the driving/transit/walking tabs — i.e. the capability under test was still exercised. The shortcut skips only typing the destination string. So these five passes stand as valid end-states; what they break is cross-run / cross-model comparability (run N inherits run N-1's hints), which docs/reproducibility.md excludes from the reset/seed gate.

Contrast easy__google-maps__004's gemini-26, where the leakage does invalidate the score: it never created a note and never added anything to the home screen — the graded deliverable did not exist. That is a genuine vacuous pass; the medium__google-maps__002 rows above are not.

Live re-check 16 Sep: Maps Recent held pharmacy, general physician clinic near me, hospital near me open now, Chennai International Airport (MAA), RG Residency, Le Dazzle — i.e. the 530 Maps-task leftovers are still there; Favourites = 0 places, and "All saved" holds only AI4Bharat (no airport / Bali Cafe / SUM Hospital), so no pass came from a pre-saved place.

Status — 🔄 cleared before the 16 Sep re-runs. The leftover Maps route + Recent entry, the Notes "last-edited note" state, and the home-screen Notes widget were all removed before the three re-runs above (added to docs/pre-run-checklist.md §9, scripts/seeding/reset_phone.py and the reset-phone skill). All three re-runs therefore started from a clean state — and all three still FAILed, which is what retires the leakage caveat for Gemma on these two Maps tasks.

Neither cleanup is in reset_phone.py's automatic removal lists (it names SUM Hospital - 2.8 km; Fastest Route… is absent), so both recur every batch unless the operator runs the manual UI step. Both are now in reset_phone.py's manual_ui_cleanup list, docs/pre-run-checklist.md §9, and the reset-phone skill.

Two related seed notes (also [v]):

• seed_data.py calendar times are UTC-shifted +5:30. day0 is UTC midnight, so day0 + 11h puts Lunch with Maa at 16:30 IST (not 11:00), Weekly_Standup at 14:30 (not 09:00 — hence the 14:30 in the UI), meeting_title at 14:00 (not 08:30), Old_Gym_Class at 12:30. ensure_calendar_events is unaffected (real ZoneInfo) — live: Weekly Sync Mon 21 Sep 07:00 + 10:00, Gym Tue 22 Sep 06:30, all correct (but each duplicated ×2).
• medium__calendar__013's three "Work" events remain dated 08-18/19/20 (day-3 task, not reached this run).

Limitations

• Day-3 coverage is 13 / 20 (battery death), so the HC set is 6 / 7 and the comparable 60-denom penalises 7 unreached tasks.
• ADB corroboration is post-hoc (phone re-charged to 100% before the audit), so device facts are read after the run, not during.
• The 16 Sep re-runs used a clean state but a later wall-clock day than the batch, so their ambient conditions (temperature, battery start) differ slightly from the day1–3 slice.

Artifacts

• Run: assets/runs/public/20260916-011341/
• HF dataset: YuvrajSingh9886/androidlife-public → runs/20260916-011341/
• Narrative: reports/public/public-20260916-011341.md
• Auditor notes: reports/public/audit-20260916-011341/ (dossier.json, agent_success_index.json, agent_day*.md, adb_snapshot.txt)
• Official metrics: reports/metrics/public/public-20260916-011341-report.{json,md}
• HC judge: reports/metrics/hallucination/public-20260916-011341.{json,md}
• KBIQ sidecar: assets/runs/public/20260916-011341/kb_audit.json
• Turn-based: reports/turn-based/ask-query-{single,multi}/20260916-011341/ (via make organize-public)
• Superseded copies of the replaced task folders: /tmp/drain-superseded-20260916/ (backup only, not in the repo)