Testing in CI
Run a plain-English test on a fresh cloud phone from GitHub Actions using curl and the REST API, fail the build on a failed task, and keep the video.
This guide runs an end-to-end check on every pull request: boot a device, install the APK your workflow just built, ask ARTEMIS to walk through a flow, fail the job if the task fails, upload the video and screenshots as build artifacts, and always stop the device. It uses only curl and jq, which are preinstalled on GitHub-hosted Ubuntu runners.
Secrets#
Add two repository secrets (Settings › Secrets and variables › Actions):
| Secret | Value |
|---|---|
AKAPHONE_API_KEY |
A ak_live_… key. Create a dedicated one named github-actions so you can revoke it alone. |
MODEL_API_KEY |
Your Gemini key (or Anthropic/OpenAI; then set MODEL_PROVIDER below). |
The workflow#
name: Phone test
on: [pull_request]
jobs:
phone-test:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
API: https://akaphone.abhgnx-417.workers.dev
AUTH: "Authorization: Bearer ${{ secrets.AKAPHONE_API_KEY }}"
MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
MODEL_PROVIDER: gemini
steps:
- uses: actions/checkout@v4
# Build your app here; this example expects app/build/outputs/apk/debug/app-debug.apk
- name: Build APK
run: ./gradlew assembleDebug
- name: Start a device
run: |
DEV=$(curl -sf -X POST "$API/v1/devices" -H "$AUTH" -H "Content-Type: application/json" \
-d "{\"name\": \"ci-${GITHUB_RUN_ID}\", \"wait\": true}")
DEVICE_ID=$(echo "$DEV" | jq -r .id)
echo "DEVICE_ID=$DEVICE_ID" >> "$GITHUB_ENV"
# wait=true holds up to 90 s; keep polling if the device is still starting.
for i in $(seq 1 60); do
STATUS=$(curl -sf "$API/v1/devices/$DEVICE_ID" -H "$AUTH" | jq -r .status)
[ "$STATUS" = running ] && break
case "$STATUS" in error|lost|terminated) echo "device $STATUS"; exit 1;; esac
sleep 5
done
[ "$STATUS" = running ] || { echo "device did not start"; exit 1; }
- name: Install the APK
run: |
FILE=app/build/outputs/apk/debug/app-debug.apk
SIZE=$(wc -c < "$FILE" | tr -d ' ')
UP=$(curl -sf -X POST "$API/v1/uploads" -H "$AUTH" -H "Content-Type: application/json" \
-d "{\"filename\": \"app-debug.apk\", \"size_bytes\": $SIZE}")
UPLOAD_ID=$(echo "$UP" | jq -r .upload.id)
curl -sf -X PUT "$(echo "$UP" | jq -r .put_url)" \
-H "Content-Type: $(echo "$UP" | jq -r '.headers["content-type"]')" \
--data-binary @"$FILE"
curl -sf -X POST "$API/v1/uploads/$UPLOAD_ID/complete" -H "$AUTH" > /dev/null
INSTALL_ID=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/apps" -H "$AUTH" -H "Content-Type: application/json" \
-d "{\"upload_id\": \"$UPLOAD_ID\"}" | jq -r .install_id)
for i in $(seq 1 100); do
INSTALL=$(curl -sf "$API/v1/devices/$DEVICE_ID/apps/$INSTALL_ID" -H "$AUTH")
STATUS=$(echo "$INSTALL" | jq -r .status)
[ "$STATUS" != pending ] && break
sleep 3
done
echo "$INSTALL"
[ "$STATUS" = installed ] || { echo "install $STATUS"; exit 1; }
- name: Run the test
run: |
INSTRUCTION='Open the Example Shop app. Add the first product to the cart, open the cart and check it shows 1 item. Report PASS if it does, otherwise FAIL and what you saw.'
TASK=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/tasks" -H "$AUTH" \
-H "Content-Type: application/json" \
-H "X-Model-Key: $MODEL_API_KEY" -H "X-Model-Provider: $MODEL_PROVIDER" \
-d "$(jq -n --arg i "$INSTRUCTION" '{instruction: $i, profile: "pro"}')")
TASK_ID=$(echo "$TASK" | jq -r .id)
echo "TASK_ID=$TASK_ID" >> "$GITHUB_ENV"
while :; do
TASK=$(curl -sf "$API/v1/tasks/$TASK_ID" -H "$AUTH")
STATUS=$(echo "$TASK" | jq -r .status)
case "$STATUS" in succeeded|failed|cancelled) break;; esac
sleep 5
done
echo "$TASK" | jq '{status, result, error, steps: [.steps[].label]}'
[ "$STATUS" = succeeded ] || exit 1
echo "$TASK" | jq -r .result | grep -q PASS
- name: Download artifacts
if: always() && env.TASK_ID != ''
run: |
mkdir -p phone-artifacts
sleep 5 # artifacts are uploaded right after the task ends
curl -sf "$API/v1/tasks/$TASK_ID/artifacts" -H "$AUTH" \
| jq -r '.files[] | "\(.name) \(.url)"' \
| while read -r NAME URL; do curl -sf -o "phone-artifacts/$NAME" "$URL"; done
- uses: actions/upload-artifact@v4
if: always()
with:
name: phone-test
path: phone-artifacts/
if-no-files-found: ignore
- name: Stop the device
if: always() && env.DEVICE_ID != ''
run: curl -sf -X DELETE "$API/v1/devices/$DEVICE_ID" -H "$AUTH" > /dev/nullHow it decides pass or fail#
status tells you whether ARTEMIS finished: succeeded means it completed the instruction, failed means it could not (or the model call failed, or it timed out). That alone does not mean your app behaved correctly, so the instruction asks the agent to report PASS or FAIL and the step greps result for it. Write instructions with an explicit check and an explicit verdict.
Cost#
Each run uses one device for the build-independent part of the job: typically 3 to 8 minutes, so 3 to 8 credits. The if: always() stop step matters: without it, a cancelled or failed job leaves the device running until the 30-minute idle stop.
Faster runs with a snapshot#
If every run needs the same setup (a signed-in test account, onboarding skipped), do it once on a device, take a snapshot, and start CI devices from it:
curl -sf -X POST "$API/v1/snapshots/$SNAPSHOT_ID/restore" -H "$AUTH" \
-H "Content-Type: application/json" -d '{"name": "ci", "wait": true}'Remember that snapshots are pinned to one host: if it is busy the device queues, and if the host is lost so is the snapshot. Keep the setup scripted so you can recreate it.
Parallel runs#
Start one device per flow and run the steps in a matrix. Your plan's concurrency limit applies across everything on your account (3 devices on Dev, 5 on Team), and device creation is limited to 10 per minute. A 409 concurrency_limit in CI means other jobs or people are using your devices.