How it was formed

AndroidLife measures what it actually costs a real Android phone to have an LLM-driven agent use it - not just whether the agent finishes each task, but the dollars, battery percentage, and heat it costs to do so, across a realistic month of everyday phone use. Most mobile-agent benchmarks report task success rate and stop there.

Who we evaluate. AndroidLife targets open-weight models across sizes - API-hosted open LLMs now, and on-device SLMs as the heavy future focus. Agents run on MobileRun (Droidrun); the LLM can be remote or local. Task density is grounded in real usage (~9-10 apps/day; data.ai State of Mobile and related sources below).

Two load-bearing axes make it different:

• What success costs on real hardware: live battery drain (mAh, per-app), device thermal load (CPU/GPU/skin/battery temperature sampled through the run), and real dollar cost per model - reported alongside success rate, never blended into it.
• Honesty under pressure: a subset of tasks reference data verified absent on the device, so a model's self-reported "success" can be checked against real device state and classified as a true success, an honest failure, or a hallucination.

It runs on one real, non-rooted Android phone (OnePlus 10R 5G / CPH2423, Android 15, OxygenOS CPH2423_15.0.0.1901(EX01)) through the mobilerun (Droidrun) SDK - no emulator, no synthetic sandbox. The agent drives the same accessibility-tree UI automation a human uses, and the model never runs on the phone being benchmarked, so its own inference cost and heat never contaminate the device-cost measurement.

Every benchmark day is meant to look like an ordinary person's real phone use. The schedule is therefore anchored to how many apps a real person actually touches in a day, not to an arbitrary task list.

The average smartphone user has ~80 apps installed but uses only ~9-10 per day and ~30 per month - data.ai (App Annie), State of Mobile.

The design drives the per-day app density down to reality:

The 530-task corpus spans 28 days (15-22 tasks/day, ~18.9/day) across 31 apps. Per-day distinct apps run 10-12 (mean ~10.6); cross-app tasks are spread 4-10 per day (mean ~6.4) so no day is all-single-app or all-cross-app. (The full, cited breakdown lives in docs/app-usage-grounding.md.)

Source of truth: benchmarks/androidlife-530/tasks_530.md → AndroidLife_530_v1.json. The corpus is pinned at exactly 530 tasks.

Difficulty taxonomy

• Easy - 1 app, 1 step, 1 point - single-app by design.
• Medium - 1-2 apps, 3 steps, 3 points (~47% cross-app).
• Hard - 2-3 apps, 5 steps, 5 points, split into non-ASK USER / ASK USER (SINGLE / MULTI).

Single-app vs cross-app

A task is cross-app when its apps array has more than one app - the agent must switch apps mid-task, not camp on one screen. Easy tasks are single-app by design (216/216). Medium is ~47% cross-app; hard is mostly cross-app (~85%). Cross-app load is spread across the 28-day schedule so no day is all-single-app or all-cross-app. See also benchmark-spec.md.

Placeholders - endless variations

Task text is templated with placeholders such as [contact], [presentation name], and [video name]. Swapping the pin file changes the concrete instance without editing the task list - so the corpus supports many variations of the same graded behaviors. The 530 uses 303 placeholder slots across 119 distinct keys. For the public 60, template: public_vars.example.env; device pin: public_vars.local.env.

Beyond easy / medium / hard, two task modes stress whether the agent knows what it does not know. Both appear on the homepage as ASK USER and hallucination controls.

ASK USER

A load-bearing fact is deliberately withheld from the prompt (who to message, which file, which place, …). The agent must call the ask_user tool and get the answer from a simulated user - guessing scores zero even if the later UI steps look right (MobileWorld-style SR gate).

Simulated user model: default gpt-5.4-mini, selected with --ask-user-model (see DEFAULT_ASK_USER_MODEL in src/androidlife/custom_tools.py). The judge answers from the ground-truth fact sidecar or multi-turn KB profile - not from the phone screen.

• ASK USER SINGLE (36 in the 530) - one or two withheld facts; each question is answered independently from the ground-truth fact sidecar (ask_user_facts_*.json). Success requires: ask → correct answer → completed on-device end state.
• ASK USER MULTI (13 in the 530) - a multi-turn KB-oracle dialogue (multiturn_kb_*.json). The agent must drive the conversation to the profile's correct_target (KBIQ) and finish the graded end state.

Hallucination controls

The target data is verified absent on the device (no such email, file, note, …). The correct behavior is an honest failure: stop and report that the thing is not there. Claiming success anyway is scored as a hallucination. On the public 60, 7 tasks are honesty-graded this way; the 530 has 60 such controls across days 2-28.

Post-run GEval / DeepEval pipeline: after the batch, scripts/eval/eval_hallucination_controls.py runs a DeepEval DAGMetric judge over the full agent.log.txt (src/androidlife/hallucination_judge.py). Default judge model is again gpt-5.4-mini (--hallucination-judge-model / --model). Outputs land in reports/metrics/hallucination/public-<RUN_TS>.{json,md}. Manual audit remains ground truth for the published leaderboard; the DAGMetric pass is the automated GEval-style check for HC honesty.

See the site evaluation flowchart (Fig. 3). Manual leaderboard ground truth: manual-audit protocol.

The public benchmark (benchmarks/androidlife-530/public.md → AndroidLife_public_v2.json) is a standalone publishable evaluation set (3 days × 20 tasks). It shares MobileRun, seeds, and grading with the 530 corpus and is shaped to be representative of that mix - not a throwaway subset. See reproducibility for reset / seed / verify and why live-phone runs are never bitwise-identical.

Scoring is summarized on the site evaluation flowchart (Fig. 3). Published leaderboard rows use the manual-audit protocol as ground truth after automated report / HC DAGMetric passes. Metric definitions live under Metrics on the homepage.

To make the deterministic tasks genuinely solvable, a controlled, fabricated test persona ("Yuvraj Singh" and fictional family, friends, and vendors) is set up on the device. No real personal details are exposed. See docs/fabricated-test-data.md for the full disclosure.

Seeds are documented per task in seed manifests (scripts/seeding/build_day_seed_manifest.py); days 1-6 use hand-authored specs, days 7-28 are auto-generated from the dataset. Per-day placeholder vars are written by scripts/seeding/generate_day_vars.py and passed with --vars-file.

The corpus favours Google-ecosystem apps (Docs, Sheets, Slides, Meet, Drive, Photos, Maps, Search) because their terms permit normal human-equivalent UI use and their end-states are verifiable. Telegram stands in for the messaging category (permissive enough, open bot API); Chrome/Search exercise social-adjacent browsing without touching the social apps themselves. Food, OTT, travel, tickets, news, and shopping gaps were closed by adding real native apps (Swiggy, Prime Video, MakeMyTrip, BookMyShow, MSN News, Amazon Shopping).

Categories still absent are excluded for a mix of automation ToS and verifiability, not oversight: WhatsApp/Instagram/TikTok/X/Snapchat/Reddit ban automated access in their terms and actively block scripted drivers; gaming and passive OTT have no crisp verifiable end-state; finance/UPI/banking is excluded on safety grounds (credentials and real money).

The homepage Benchmark Summary is computed directly from the public dataset (AndroidLife_public_v2.json) plus the hallucination-control and multi-turn-KB sidecars in website/tools/build_site_data.mjs, so it always matches the 60 tasks you can browse. The Leaderboard is hand-copied from the run reports in reports/public/ (the run folders are private / gitignored). The full spec is in docs/benchmark-spec.md and docs/benchmark-spec-public.md.