AkaPhoneDocs
Guides

Testing in CI

Run a plain-English test on a fresh cloud phone from GitHub Actions using curl and the REST API, fail the build on a failed task, and keep the video.

This guide runs an end-to-end check on every pull request: boot a device, install the APK your workflow just built, ask ARTEMIS to walk through a flow, fail the job if the task fails, upload the video and screenshots as build artifacts, and always stop the device. It uses only curl and jq, which are preinstalled on GitHub-hosted Ubuntu runners.

Secrets#

Add two repository secrets (Settings › Secrets and variables › Actions):

Secret Value
AKAPHONE_API_KEY A ak_live_… key. Create a dedicated one named github-actions so you can revoke it alone.
MODEL_API_KEY Your Gemini key (or Anthropic/OpenAI; then set MODEL_PROVIDER below).

The workflow#

.github/workflows/phone-test.yml
name: Phone test
on: [pull_request]

jobs:
  phone-test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      API: https://akaphone.abhgnx-417.workers.dev
      AUTH: "Authorization: Bearer ${{ secrets.AKAPHONE_API_KEY }}"
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
      MODEL_PROVIDER: gemini
    steps:
      - uses: actions/checkout@v4

      # Build your app here; this example expects app/build/outputs/apk/debug/app-debug.apk
      - name: Build APK
        run: ./gradlew assembleDebug

      - name: Start a device
        run: |
          DEV=$(curl -sf -X POST "$API/v1/devices" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"name\": \"ci-${GITHUB_RUN_ID}\", \"wait\": true}")
          DEVICE_ID=$(echo "$DEV" | jq -r .id)
          echo "DEVICE_ID=$DEVICE_ID" >> "$GITHUB_ENV"
          # wait=true holds up to 90 s; keep polling if the device is still starting.
          for i in $(seq 1 60); do
            STATUS=$(curl -sf "$API/v1/devices/$DEVICE_ID" -H "$AUTH" | jq -r .status)
            [ "$STATUS" = running ] && break
            case "$STATUS" in error|lost|terminated) echo "device $STATUS"; exit 1;; esac
            sleep 5
          done
          [ "$STATUS" = running ] || { echo "device did not start"; exit 1; }

      - name: Install the APK
        run: |
          FILE=app/build/outputs/apk/debug/app-debug.apk
          SIZE=$(wc -c < "$FILE" | tr -d ' ')
          UP=$(curl -sf -X POST "$API/v1/uploads" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"filename\": \"app-debug.apk\", \"size_bytes\": $SIZE}")
          UPLOAD_ID=$(echo "$UP" | jq -r .upload.id)
          curl -sf -X PUT "$(echo "$UP" | jq -r .put_url)" \
            -H "Content-Type: $(echo "$UP" | jq -r '.headers["content-type"]')" \
            --data-binary @"$FILE"
          curl -sf -X POST "$API/v1/uploads/$UPLOAD_ID/complete" -H "$AUTH" > /dev/null
          INSTALL_ID=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/apps" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"upload_id\": \"$UPLOAD_ID\"}" | jq -r .install_id)
          for i in $(seq 1 100); do
            INSTALL=$(curl -sf "$API/v1/devices/$DEVICE_ID/apps/$INSTALL_ID" -H "$AUTH")
            STATUS=$(echo "$INSTALL" | jq -r .status)
            [ "$STATUS" != pending ] && break
            sleep 3
          done
          echo "$INSTALL"
          [ "$STATUS" = installed ] || { echo "install $STATUS"; exit 1; }

      - name: Run the test
        run: |
          INSTRUCTION='Open the Example Shop app. Add the first product to the cart, open the cart and check it shows 1 item. Report PASS if it does, otherwise FAIL and what you saw.'
          TASK=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/tasks" -H "$AUTH" \
            -H "Content-Type: application/json" \
            -H "X-Model-Key: $MODEL_API_KEY" -H "X-Model-Provider: $MODEL_PROVIDER" \
            -d "$(jq -n --arg i "$INSTRUCTION" '{instruction: $i, profile: "pro"}')")
          TASK_ID=$(echo "$TASK" | jq -r .id)
          echo "TASK_ID=$TASK_ID" >> "$GITHUB_ENV"
          while :; do
            TASK=$(curl -sf "$API/v1/tasks/$TASK_ID" -H "$AUTH")
            STATUS=$(echo "$TASK" | jq -r .status)
            case "$STATUS" in succeeded|failed|cancelled) break;; esac
            sleep 5
          done
          echo "$TASK" | jq '{status, result, error, steps: [.steps[].label]}'
          [ "$STATUS" = succeeded ] || exit 1
          echo "$TASK" | jq -r .result | grep -q PASS

      - name: Download artifacts
        if: always() && env.TASK_ID != ''
        run: |
          mkdir -p phone-artifacts
          sleep 5   # artifacts are uploaded right after the task ends
          curl -sf "$API/v1/tasks/$TASK_ID/artifacts" -H "$AUTH" \
            | jq -r '.files[] | "\(.name) \(.url)"' \
            | while read -r NAME URL; do curl -sf -o "phone-artifacts/$NAME" "$URL"; done

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: phone-test
          path: phone-artifacts/
          if-no-files-found: ignore

      - name: Stop the device
        if: always() && env.DEVICE_ID != ''
        run: curl -sf -X DELETE "$API/v1/devices/$DEVICE_ID" -H "$AUTH" > /dev/null

How it decides pass or fail#

status tells you whether ARTEMIS finished: succeeded means it completed the instruction, failed means it could not (or the model call failed, or it timed out). That alone does not mean your app behaved correctly, so the instruction asks the agent to report PASS or FAIL and the step greps result for it. Write instructions with an explicit check and an explicit verdict.

Cost#

Each run uses one device for the build-independent part of the job: typically 3 to 8 minutes, so 3 to 8 credits. The if: always() stop step matters: without it, a cancelled or failed job leaves the device running until the 30-minute idle stop.

Faster runs with a snapshot#

If every run needs the same setup (a signed-in test account, onboarding skipped), do it once on a device, take a snapshot, and start CI devices from it:

Shell
curl -sf -X POST "$API/v1/snapshots/$SNAPSHOT_ID/restore" -H "$AUTH" \
  -H "Content-Type: application/json" -d '{"name": "ci", "wait": true}'

Remember that snapshots are pinned to one host: if it is busy the device queues, and if the host is lost so is the snapshot. Keep the setup scripted so you can recreate it.

Parallel runs#

Start one device per flow and run the steps in a matrix. Your plan's concurrency limit applies across everything on your account (3 devices on Dev, 5 on Team), and device creation is limited to 10 per minute. A 409 concurrency_limit in CI means other jobs or people are using your devices.