# Testing in CI

> Run a plain-English test on a fresh cloud phone from GitHub Actions using curl and the REST API, fail the build on a failed task, and keep the video.

Source: https://akaphone-docs.pages.dev/guides/testing-in-ci/

This guide runs an end-to-end check on every pull request: boot a device, install the APK your workflow just built, ask ARTEMIS to walk through a flow, fail the job if the task fails, upload the video and screenshots as build artifacts, and always stop the device. It uses only `curl` and `jq`, which are preinstalled on GitHub-hosted Ubuntu runners.

## Secrets

Add two repository secrets (**Settings › Secrets and variables › Actions**):

| Secret | Value |
|---|---|
| `AKAPHONE_API_KEY` | A `ak_live_…` key. Create a dedicated one named `github-actions` so you can revoke it alone. |
| `MODEL_API_KEY` | Your Gemini key (or Anthropic/OpenAI; then set `MODEL_PROVIDER` below). |

## The workflow

```yaml title=".github/workflows/phone-test.yml"
name: Phone test
on: [pull_request]

jobs:
  phone-test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      API: https://akaphone.abhgnx-417.workers.dev
      AUTH: "Authorization: Bearer ${{ secrets.AKAPHONE_API_KEY }}"
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
      MODEL_PROVIDER: gemini
    steps:
      - uses: actions/checkout@v4

      # Build your app here; this example expects app/build/outputs/apk/debug/app-debug.apk
      - name: Build APK
        run: ./gradlew assembleDebug

      - name: Start a device
        run: |
          DEV=$(curl -sf -X POST "$API/v1/devices" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"name\": \"ci-${GITHUB_RUN_ID}\", \"wait\": true}")
          DEVICE_ID=$(echo "$DEV" | jq -r .id)
          echo "DEVICE_ID=$DEVICE_ID" >> "$GITHUB_ENV"
          # wait=true holds up to 90 s; keep polling if the device is still starting.
          for i in $(seq 1 60); do
            STATUS=$(curl -sf "$API/v1/devices/$DEVICE_ID" -H "$AUTH" | jq -r .status)
            [ "$STATUS" = running ] && break
            case "$STATUS" in error|lost|terminated) echo "device $STATUS"; exit 1;; esac
            sleep 5
          done
          [ "$STATUS" = running ] || { echo "device did not start"; exit 1; }

      - name: Install the APK
        run: |
          FILE=app/build/outputs/apk/debug/app-debug.apk
          SIZE=$(wc -c < "$FILE" | tr -d ' ')
          UP=$(curl -sf -X POST "$API/v1/uploads" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"filename\": \"app-debug.apk\", \"size_bytes\": $SIZE}")
          UPLOAD_ID=$(echo "$UP" | jq -r .upload.id)
          curl -sf -X PUT "$(echo "$UP" | jq -r .put_url)" \
            -H "Content-Type: $(echo "$UP" | jq -r '.headers["content-type"]')" \
            --data-binary @"$FILE"
          curl -sf -X POST "$API/v1/uploads/$UPLOAD_ID/complete" -H "$AUTH" > /dev/null
          INSTALL_ID=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/apps" -H "$AUTH" -H "Content-Type: application/json" \
            -d "{\"upload_id\": \"$UPLOAD_ID\"}" | jq -r .install_id)
          for i in $(seq 1 100); do
            INSTALL=$(curl -sf "$API/v1/devices/$DEVICE_ID/apps/$INSTALL_ID" -H "$AUTH")
            STATUS=$(echo "$INSTALL" | jq -r .status)
            [ "$STATUS" != pending ] && break
            sleep 3
          done
          echo "$INSTALL"
          [ "$STATUS" = installed ] || { echo "install $STATUS"; exit 1; }

      - name: Run the test
        run: |
          INSTRUCTION='Open the Example Shop app. Add the first product to the cart, open the cart and check it shows 1 item. Report PASS if it does, otherwise FAIL and what you saw.'
          TASK=$(curl -sf -X POST "$API/v1/devices/$DEVICE_ID/tasks" -H "$AUTH" \
            -H "Content-Type: application/json" \
            -H "X-Model-Key: $MODEL_API_KEY" -H "X-Model-Provider: $MODEL_PROVIDER" \
            -d "$(jq -n --arg i "$INSTRUCTION" '{instruction: $i, profile: "pro"}')")
          TASK_ID=$(echo "$TASK" | jq -r .id)
          echo "TASK_ID=$TASK_ID" >> "$GITHUB_ENV"
          while :; do
            TASK=$(curl -sf "$API/v1/tasks/$TASK_ID" -H "$AUTH")
            STATUS=$(echo "$TASK" | jq -r .status)
            case "$STATUS" in succeeded|failed|cancelled) break;; esac
            sleep 5
          done
          echo "$TASK" | jq '{status, result, error, steps: [.steps[].label]}'
          [ "$STATUS" = succeeded ] || exit 1
          echo "$TASK" | jq -r .result | grep -q PASS

      - name: Download artifacts
        if: always() && env.TASK_ID != ''
        run: |
          mkdir -p phone-artifacts
          sleep 5   # artifacts are uploaded right after the task ends
          curl -sf "$API/v1/tasks/$TASK_ID/artifacts" -H "$AUTH" \
            | jq -r '.files[] | "\(.name) \(.url)"' \
            | while read -r NAME URL; do curl -sf -o "phone-artifacts/$NAME" "$URL"; done

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: phone-test
          path: phone-artifacts/
          if-no-files-found: ignore

      - name: Stop the device
        if: always() && env.DEVICE_ID != ''
        run: curl -sf -X DELETE "$API/v1/devices/$DEVICE_ID" -H "$AUTH" > /dev/null
```

## How it decides pass or fail

`status` tells you whether ARTEMIS finished: `succeeded` means it completed the instruction, `failed` means it could not (or the model call failed, or it timed out). That alone does not mean your app behaved correctly, so the instruction asks the agent to **report PASS or FAIL** and the step greps `result` for it. Write instructions with an explicit check and an explicit verdict.

## Cost

Each run uses one device for the build-independent part of the job: typically 3 to 8 minutes, so 3 to 8 credits. The `if: always()` stop step matters: without it, a cancelled or failed job leaves the device running until the 30-minute idle stop.

## Faster runs with a snapshot

If every run needs the same setup (a signed-in test account, onboarding skipped), do it once on a device, [take a snapshot](https://akaphone-docs.pages.dev/concepts/snapshots/), and start CI devices from it:

```bash
curl -sf -X POST "$API/v1/snapshots/$SNAPSHOT_ID/restore" -H "$AUTH" \
  -H "Content-Type: application/json" -d '{"name": "ci", "wait": true}'
```

Remember that snapshots are pinned to one host: if it is busy the device queues, and if the host is lost so is the snapshot. Keep the setup scripted so you can recreate it.

## Parallel runs

Start one device per flow and run the steps in a matrix. Your plan's concurrency limit applies across everything on your account (3 devices on Dev, 5 on Team), and device creation is limited to 10 per minute. A `409 concurrency_limit` in CI means other jobs or people are using your devices.