AndroidLife

Most agent benchmarks grade a model in a sandbox or on screenshots (AITW, SeeClick, AndroidWorld, MobileWorld). Phone agents live where people do - on devices with finite batteries, heat limits, and messy real apps. AndroidLife asks what those benchmarks skip: not only can the agent do the task on a live phone? but what does doing it cost a real phone?

We are not only targeting models deployed locally on the handset. The public leaderboard already includes open-weight frontier and mid-size models served over APIs (and smaller language models in the same spirit). The long-term focus is heavy on SLMs that run on the phone - same tasks, same MobileRun loop, so on-device and cloud open-weight agents stay comparable.

A task finished in 40 steps with 3% battery drain and a cool SoC beats the same task in 180 steps that thermal-throttles the device. That phone cost is first-class, not an afterthought.

Emulator leaderboards answer a narrow question: can a model click the right widget in a clean sandbox? That is not how people use phones. Real instructions are vaguely complicated, span several apps, and collide with notifications, incoming calls, and half-finished context. A useful phone agent has to survive a day of real use and finish the job and leave the device in a state a human would accept - without burning the battery or overheating the SoC along the way.

AndroidLife exists because that combination was missing: a real-device protocol for everyday tasks, with phone cost (dollars, battery, heat, latency, steps) as a first-class score and not a footnote left for future work. It evaluates open-weight models on tasks that require ambiguity resolution through human input one or many times, or requires to know when the data is not available to complete the task with query worded in a way that mimics a human's query. This is a broad set of tasks that is not limited to a single app or a single task.

The dataset + benchmark is shaped like ordinary Day In The Life of a user, while each run stays a short, bounded session on live hardware. Fabricated persona seeds keep privacy intact; every published trajectory is a real on-device execution you can replay.

The agent loop is MobileRun by Droidrun framework over wireless ADB. The LLM endpoint is swappable - OpenRouter, a local OpenAI-compatible server, or an on-phone runtime - so the same tasks and phone metrics apply whether the model is remote or on-device.

Tasks are grounded in how people actually use phones (~9-10 apps/day; see How it was formed and app-usage grounding - data.ai State of Mobile and related sources). Seeded persona data is fabricated for privacy; trajectories are real on-device runs.

AndroidLife ships as two tiers:

• 60-task public benchmark - a representative, holistic evaluation set of 60 tasks (3 days × 20 tasks), sharing the same MobileRun harness and grading philosophy as the 530 corpus. The reset / seed / verify protocol is documented so others can re-run the same evaluation ^*. Every one of the 60 tasks has been executed on a real phone (OnePlus 10R 5G) by real models (open-weight and API models).
• 530-task dataset^* - the complete dataset: 530 tasks on a fixed 28-day schedule across 31 apps (easy/medium/hard, ASK USER + non-ASK USER + hallucination controls). The 28 days model one month of real daily use (close replication of any usual day of a user projected to a month) (a person touches ~10 apps a day, not all 31), so the corpus spreads coverage across a simulated month. Trajectories and artifacts are rolling out day by day - 5 of 28 days have been fully run so far.

Metrics

Literature measures model performance on a number of metrics. However, we have kept it simple with just the metrics that matter:

• Success: did the goal actually get achieved, verified against on-device state (not just the agent's claim).
  • SR - success rate: verified true successes / tasks run (overall and per bucket).
  • UIQ - user interaction quality: fact-match on ask_user answers for ASK USER SINGLE (guessing without asking → 0).
  • KBIQ - KB interaction quality: multi-turn ASK USER MULTI dialogue correctness against the KB correct_target.
  • Hallucination rate - self-reported success on known-absent controls that failed verification (honest fail = correct).
• Efficiency: how many steps and LLM calls did it take.
• Latency: mean agent time per finished task. Elapsed (full run) is the whole-batch E2E wall-clock (all tasks + cooldowns/resets); its subline is total agent time with cooldowns removed.
• Cost: real USD per run, tracked through an LLM proxy per call.
• Phone cost: battery drain (mAh, % delta), CPU/GPU/NPU/skin thermals, and charge-counter deltas captured per second on-device.

The 530 corpus spans three difficulty buckets (1 / 3 / 5 points; design steps 1 / 3 / 5). See How it was formed for the full taxonomy and single- vs cross-app design.

Evaluation

Same MobileRun loop for both tiers. Resolve the task prompt (and placeholders), then run on-device through telemetry, trajectory capture, and graded audit.

See Evaluation for ASK USER / hallucination-control details and Metrics for score definitions.

The published evaluation (leaderboard / trajectories) is the 60-task public benchmark - a fixed 3-day sample drawn from the corpus, same harness and grading, reused identically for every model. Browse the 30 apps in the public sample (Weather and MakeMyTrip stay in the full corpus only). Placeholders: 50 uses / 37 keys, pinned in public_vars.local.env.

The 530-task dataset is the full corpus the public sample is drawn from - a one-time-shuffled 28-day schedule (≈15-22 tasks / day). Task definitions ship for all 28 days; on-device execution of every day is still rolling out ^*. Browse apps (public sample lists 30; full corpus adds Weather + MakeMyTrip). Placeholders: 303 uses / 119 keys, resolved from the 530 vars pin files.

Every task captures, per second, the true phone cost independent of the model under test:

• Battery: dumpsys battery level delta %, charge-counter delta (µAh), per-app mAh.
• Thermal: dumpsys thermalservice CPU, GPU, NPU, skin, and vendor phone temps, plus thermal status.
• Cost: real USD per run through the LLM proxy (OpenRouter pricing or local host estimates).
• Steps / tokens: step budget (50 default), LLM call count, prompt/completion tokens.

Reference device

• Phone: OnePlus CPH2423 (OnePlus 10R 5G), non-rooted.
• OS: Android 15 (API 35) · OxygenOS / ColorOS base V15.0.0 · build CPH2423_15.0.0.1901(EX01) · OTA CPH2423_11.H.21_3210_202607011222 · security patch 2026-07-01.
• Panel: 1080×2412 @ 480 dpi.
• ADB: wireless (e.g. Tailscale) preferred for measured runs; USB serial works to re-arm adb tcpip 5555 when needed. Pass your device with --serial.
• Model host: any OpenAI-compatible endpoint - OpenRouter, Requesty, or a local server.
• Agent harness: MobileRun by Droidrun — pinned to mobilerun==0.6.15 (v0.6.15, commit dac3e36).

Controlled run environment

• Both public 60-task leaderboard and full dataset runs use the same fixed phone state (live phone — best-effort) so battery, thermal, and success stay comparable across models.
• Amazon Music plays in the background ^† (volume ~20%). Intentional: a real user’s phone is rarely idle — media often runs while they switch apps.
• Screen brightness fixed at 50%.
• Wi-Fi only.
• Auto-rotate / location / dark mode on.
• Public caps: --steps 60, --task-timeout 2400 (40 min)
• Live-phone shortcuts are part of the setting (and not hidden from the model). The phone is a real, used device, so residual UI state can hand an agent a shortcut a pristine VM would not have — a destination still in Maps' Recent list, a notes app that reopens the last-edited note, an item already in a cart, a place already saved. We do not sanitise that away, because acting on the state a real phone is actually in is the capability under test; the alternative — a scrupulously clean device — would be the unrealistic condition. We do reset run-created state between runs so scores stay comparable (see footnote 5). ⁵

Reset → seed → verify

Full command sequences live in the repo (not duplicated on this page): docs/device-reset-and-seed.md · README · docs/getting-started.md · GUI/cloud seeds: docs/pre-run-checklist.md.

• Public benchmark (60 tasks) — Before each scored public batch, reset the phone to the public baseline (reset_phone.py --profile public_v2), re-seed days 1–3 (plus enrich notes + public PDFs), run the verify gate, then finish UI/cloud seeds. Profile public_v2 is for the public benchmark only. Launch the batch from the README (nohup / OpenRouter / local llama examples, --vision, resume).
• Full dataset (530 tasks) — Before each day (or campaign slice) on the 28-day schedule, undo agent side-effects with a reset, push that day’s fabricated seeds (seed_data.py --day N), and verify day seeds before scoring. Do not use public_v2 enrich/PDF steps as a stand-in for the full corpus — those are public-sample specific. Day launch examples: README · 530-day schedule.

Browse public trajectories

Per-task screen replays and traces live under the Tasks tab (public 60 + full 530), not on this homepage.

If you use AndroidLife — the benchmark, leaderboard, tasks, or results — please credit this work and cite it as:

Code: Apache-2.0  ·  Dataset / tasks: CC BY 4.0 (attribution required)