Quick Start

Setup, Smoke Tests, Batch Runs, and Metrics

This page mirrors the benchmark workflow in the repository: create the environment, run a smoke test, run a day of tasks, and aggregate MobileWorld-style metrics.

Setup Back to Overview

Environment Setup

Clone the repository and install dependencies. This repo is uv-managed end to end.

git clone https://github.com/YuvrajSingh-mist/AndroidLife.git
cd AndroidLife
uv sync --extra dev --extra tracing --extra hf
cp .env.example .env
OPENROUTER_API_KEY=... # needed if using OpenRouter (default agent host)
OPENAI_API_KEY=... # only needed for the ask_user tool (Hard/ASK USER tasks)
HF_TOKEN=hf_... # only needed for pushing dataset exports to Hugging Face

Prerequisites are system tools (not uv-managed): adb, scrcpy, and Python 3.11–3.13. Every run writes into assets/runs/.

Smoke Test

Confirm the phone and model server are reachable, then run the pre-flight check before any real benchmark run (or after changing phones/model hosts). It checks prerequisites, the LLM server (GET /models + a real chat completion), wired and wireless ADB with a device health check, and one real one-step agent run: 

adb devices -l
curl -s "$LLM_UPSTREAM/models"

./scripts/run/smoke_test.sh --help # full flag reference
./scripts/run/smoke_test.sh # defaults: first USB device + first model
./scripts/run/smoke_test.sh --llm-url http://192.168.1.50:8080/v1 --model my-model
./scripts/run/smoke_test.sh --skip-llm --skip-agent-run --wireless-serial 192.168.1.23:5555

Running a Benchmark

Point LLM_UPSTREAM/MODEL at OpenRouter for a hosted model, or at a local host. List a task slice first, then dry-run it to see the exact commands that would execute, then run it for real: 

export ANDROIDLIFE_SERIAL=RS7XKZDI8HTOJNYL # adb serial of your phone
export LLM_UPSTREAM=https://openrouter.ai/api
export MODEL='qwen/qwen3.7-flash'

uv run androidlife_tasks.py --bucket easy --app gmail --list
uv run androidlife_tasks.py \
 --bucket easy --app gmail --skip-unresolved \
 --serial "$ANDROIDLIFE_SERIAL" \
 --llm-upstream-base "$LLM_UPSTREAM" --model "$MODEL" \
 --dry-run

Run a whole day from the 28-day schedule (or all 530) with one command: 

uv run python scripts/run/run_day.py --day 1
uv run python scripts/run/run_day.py --day 1 --dry-run # preview the exact per-task commands
uv run python scripts/run/run_day.py --all # the whole 530-task set

# resume an interrupted batch in place:
uv run python scripts/run/run_day.py --day 1 \
 --run-root assets/runs/full-bench/2026-08-09-153930 \
 --resume-from medium__gallery__001

Single one-off task with full harness artifacts:

uv run androidlife_runner.py \
 --serial "$ANDROIDLIFE_SERIAL" \
 --label gmail-unread-count \
 --llm-upstream-base "$LLM_UPSTREAM" \
 --model "$MODEL" --temperature 0 --steps 50 \
 --goal "Check how many unread emails are in the inbox"

Metrics

After a batch, aggregate the run folders into MobileWorld-style metrics (Success Rate with the ASK USER gate, Average Completion Steps, Average User Queries, and User Interaction Quality): 

uv run scripts/eval/androidlife_report.py \
 --runs assets/runs/full-bench/2026-08-09-153930/day1
uv run scripts/eval/androidlife_report.py \
 --runs 'assets/runs/full-bench/2026-08-09-153930/day1' --cooldown-seconds 10

Preview This Site

The website is static and GitHub Pages-ready. Rebuild the task data and preview locally:

node website/tools/build_site_data.mjs # regenerate site_data.json from the datasets
cd website
python3 -m http.server 8000
# then open http://localhost:8000