write-e2e-test

Follow this guide when writing e2e tests.

E2E Environment Setup

CRITICAL: All make commands must be run from e2e/ directory, not project root.

Standard Workflow

When starting fresh or after code changes:

cd e2e                          # ← MUST be in e2e directory
make teardown && make setup     # ← Tears down old containers, rebuilds binaries, applies migrations

This command:

• Stops and removes Docker containers (Redis, PostgreSQL, Deno hook server)
• Rebuilds authgear and authgear-portal binaries with latest code
• Applies all database migrations
• Starts services fresh

Running Tests

After setup completes, run tests from the e2e directory:

cd e2e  # ← Must be in e2e directory

# Run a specific test
go test ./pkg/testrunner/ -count 1 -v -timeout 10m -run "TestAuthflow/<folder>/<filename_without_extension>"

# Run all tests
go test ./pkg/testrunner/ -count 1 -v -timeout 10m

When Setup Fails

If make teardown && make setup fails:

1. Try again - transient Docker issues are common:

   cd e2e && make teardown && make setup
   

2. Check if containers are stuck:

   docker ps  # Run from any directory to list containers
   

3. If still failing - ask the user to manually intervene. Do NOT attempt low-level Docker operations (docker-compose down -v, etc.) as these can affect other projects and data.

Always use the Makefile (make teardown && make setup). Do not skip to ./run.sh or other workarounds.

When to Re-setup

Always run make teardown && make setup from e2e/ directory when:

• You modified server code (rebuilds binaries)
• Database schema/migrations changed
• Tests fail with DB schema errors
• Starting fresh after a long break

Overview

E2e tests are YAML files placed under e2e/tests/<feature>/. The test runner auto-discovers all *.test.yaml files. Each file defines one test case.

Test File Structure

name: Human-readable test name
authgear.yaml:           # Optional: config overrides
  extend: path/to/base.yaml   # Optional base config
  override: |
    fraud_protection:
      enabled: true
before:                  # Optional: setup hooks (run before steps)
  - type: custom_sql
    custom_sql:
      path: fixtures.sql
steps:
  - name: step name      # Optional but recommended
    action: create
    input: |
      {
        "type": "signup",
        "name": "default"
      }
    output:
      result: |
        {
          "action": {
            "type": "identify"
          }
        }

Step Actions

create — Start a new auth flow

- action: create
  input: |
    {
      "type": "signup",
      "name": "default"
    }
  output:
    result: |
      {
        "action": {
          "type": "identify"
        }
      }

input — Provide input to the current flow step

- action: input
  input: |
    {
      "identification": "phone",
      "login_id": "+6591230001"
    }
  output:
    result: |
      {
        "action": {
          "type": "verify"
        }
      }

To expect an error response:

- action: input
  input: |
    {
      "channel": "sms"
    }
  output:
    error: |
      {
        "name": "TooManyRequest",
        "reason": "BlockedByFraudProtection",
        "code": 429
      }

query — SELECT from the main app database

- action: query
  query: |
    SELECT id, email FROM _auth_user
    WHERE app_id = '{{ .AppID }}'
    ORDER BY created_at
  query_output:
    rows: |
      [
        {
          "id": "[[string]]",
          "email": "user@example.com"
        }
      ]

audit_query — SELECT from the audit database

- action: audit_query
  audit_query: |
    SELECT activity_type, data->'payload'->'record'->>'decision' AS decision
    FROM _audit_log
    WHERE app_id = '{{ .AppID }}'
      AND activity_type = 'fraud_protection.decision_recorded'
    ORDER BY decision, created_at
  audit_query_output:
    rows: |
      [
        {
          "activity_type": "fraud_protection.decision_recorded",
          "decision": "allowed"
        }
      ]

Important: The _audit_log.data column stores the full serialized event.Event. JSON paths must include the payload level: data->'payload'->'field'.

Ordering tip: When multiple rows can land in the same second, ORDER BY created_at is non-deterministic. Add a secondary sort on a stable column (e.g., ORDER BY decision, created_at).

http_request — Raw HTTP request

- action: http_request
  http_request_method: POST
  http_request_url: http://{{ .AppID }}.authgeare2e.localhost:4000/oauth2/token
  http_request_form_urlencoded_body:
    grant_type: client_credentials
    client_id: myclient
    client_secret: mysecret
  http_output:
    http_status: 200
    json_body: |
      {
        "access_token": "[[string]]",
        "token_type": "Bearer"
      }

admin_api_graphql — Admin API GraphQL

- action: admin_api_graphql
  admin_api_request:
    query: |
      mutation { deleteUser(input: {userID: $userID}) { deletedUserID } }
    variables: |
      {"userID": "{{ nodeID "User" .steps.get_user.result.rows 0 .id }}"}
  admin_api_output:
    result: |
      {
        "data": {
          "deleteUser": {
            "deletedUserID": "[[string]]"
          }
        }
      }

sleep — Wait for async operations

- action: sleep
  sleep_for: 2s

Template Variables

Available in all string fields (inputs, queries, outputs, SQL files):

Template functions available: all Sprig functions plus:

• {{ linkOTPCode "phone" "+6591230001" }} — Get OTP code (test mode sends 111111)
• {{ generateTOTPCode "secret" }} — Generate TOTP code
• {{ nodeID "User" "some-uuid" }} — Encode a relay global ID
• {{ printf "%s-suffix" .AppID }} — String formatting

Matcher Patterns

Use these in output.result, output.error, query_output.rows, and audit_query_output.rows:

Extra fields in objects are allowed by default (partial matching).

Before Hooks

before:
  # Run SQL fixtures against the main database
  - type: custom_sql
    custom_sql:
      path: fixtures.sql   # relative to the test file

  # Run SQL fixtures against the audit database
  - type: custom_audit_sql
    custom_audit_sql:
      path: audit_fixtures.sql

  # Import users from JSON
  - type: user_import
    user_import: users.json

SQL fixture files also support template variables ({{ .AppID }}, {{ uuidv4 }}, etc.).

authgear.yaml Overrides

The override snippet is merged into the default config. Use it to enable features, add identity providers, or change policies:

authgear.yaml:
  override: |
    fraud_protection:
      enabled: true
      decision:
        action: deny_if_any_warning

To base the test on a different config file:

authgear.yaml:
  extend: ../base/authgear.yaml
  override: |
    authentication:
      primary_authenticators:
        - password

After Writing Tests

After creating the test file(s), always run them to verify they pass.

If the e2e environment may be stale (e.g. first run in this session, or migrations/server code changed), set it up first:

make teardown && make setup

Then run the new test(s):

cd e2e && go test ./pkg/testrunner/ -count 1 -v -timeout 10m -run "TestAuthflow/<folder>/<filename_without_extension>"

If the authgear/e2e daemons are started by ./run.sh setup and do not survive shell exit, combine setup and test in one shell:

./run.sh teardown
./run.sh setup
go test ./pkg/testrunner -count 1 -v -timeout 10m -run "TestAuthflow/<folder>/<filename_without_extension>"

If a test fails, read the error output, fix the test file, and re-run. Do not report the tests as done until they pass.

Common Mistakes

❌ Testing Only Endpoint Existence, Not Functionality

Wrong approach — just verify the endpoint works:

steps:
  - name: query_endpoint
    action: admin_api_graphql
    admin_api_request:
      query: |
        query { user { accountLockout { isLocked } } }
    admin_api_output:
      result: |
        {
          "data": {
            "node": {
              "accountLockout": {
                "isLocked": "[[boolean]]"
              }
            }
          }
        }

This test only verifies the endpoint exists and returns data. It doesn't test whether the feature actually works.

Right approach — trigger real behavior, then verify state changed:

steps:
  # 1. Trigger actual failed authentication attempts
  - name: failed_login_1
    action: input
    input: |
      {
        "authentication": "primary_password",
        "password": "wrong"
      }
    output:
      error: |
        {
          "name": "InvalidCredentials"
        }

  - name: failed_login_2
    action: input
    input: |
      {
        "authentication": "primary_password",
        "password": "wrong"
      }
    output:
      error: |
        {
          "reason": "TooManyRequest"
        }

  # 2. Now verify the feature actually changed state
  - name: verify_locked
    action: admin_api_graphql
    admin_api_request:
      query: |
        query { user { accountLockout { isLocked } } }
    admin_api_output:
      result: |
        {
          "data": {
            "node": {
              "accountLockout": {
                "isLocked": true
              }
            }
          }
        }

The right approach:

1. Triggers actual functionality via auth flow create/input actions
2. Verifies state changed via queries and mutations
3. Tests the feature end-to-end, not just the endpoint

❌ Using Go Tests Instead of YAML Tests

Wrong approach — writing e2e tests in Go code:

// ❌ Don't write e2e tests in Go
func TestAccountLockout(t *testing.T) {
    // Direct API calls, manual JSON marshaling, etc.
}

Right approach — use YAML-driven e2e tests:

# ✅ Always use YAML for e2e tests
name: Account lockout prevents further attempts
steps:
  - action: input
    input: |
      {"authentication": "primary_password", "password": "wrong"}
    output:
      error: |
        {"name": "InvalidCredentials"}

Why YAML?

• Declarative, human-readable format
• Consistent with existing test suite in e2e/tests/
• Automatic test discovery and execution
• Built-in support for auth flows, queries, mutations, and fixtures
• Clear separation of test data from test logic

Tips

1. Give steps descriptive name fields — they make failures readable and enable {{ .steps.<name> }} references.
2. Use comment lines to group related steps (e.g., # Flow 1 — signup).
3. OTP codes in test mode are always 111111 — no need to fetch them dynamically for phone/email OTP.
4. Each test case gets a fully isolated app (unique AppID) — no cleanup needed.
5. When testing audit events, always check the JSON path includes payload: data->'payload'->'...'.
6. Prefer ORDER BY <stable_column>, created_at over ORDER BY created_at alone to avoid flaky ordering.
7. To focus on one test during development, pass -run "TestAuthflow/path/to/test" to the test command.
8. After environment changes, run make teardown && make setup to apply latest migrations.
9. Always write JSON values in input, output, query_output, and audit_query_output as multi-line for readability. Prefer:

   input: |
     {
       "identification": "phone",
       "login_id": "+6591230001"
     }
   

   over:

   input: |
     {"identification": "phone", "login_id": "+6591230001"}
   

10. For messaging tests in e2e, check e2e/var/authgear.features.yaml first. SMS and WhatsApp may be suppressed in test mode, so the right audit signals can be sms.suppressed / whatsapp.suppressed instead of sms.sent / whatsapp.sent.
11. When asserting message delivery behavior, prefer querying all relevant send/suppress audit rows for the app, normalize the target in SQL, and assert the full result set so unexpected-target rows fail the test.
12. Do not hard-code zsh or another shell in examples unless the task specifically depends on it.
13. Do not assume a specific interactive shell. If setup and test must run in one shell session, execute the commands sequentially in the developer's current shell.

Common Patterns

Testing Feature Variants with Different Configurations

When a feature has multiple modes/types (e.g., per_user vs per_user_per_ip lockout), create separate test files for each variant with different authgear.yaml overrides:

# File 1: feature_variant_a.test.yaml
authgear.yaml:
  override: |
    feature:
      mode: variant_a

# File 2: feature_variant_b.test.yaml
authgear.yaml:
  override: |
    feature:
      mode: variant_b

This ensures each variant is tested independently with its specific configuration.

Testing Actual Functionality vs Just Endpoints

Don't just verify endpoints exist — use auth flow actions to trigger actual functionality:

❌ Wrong: Only query the endpoint

steps:
  - action: admin_api_graphql
    admin_api_request:
      query: |
        query { user { lockoutStatus } }

✅ Right: Trigger functionality first, then verify state changed

steps:
  # 1. Trigger actual failure (authentication attempt)
  - name: failed_login
    action: input
    input: |
      {"authentication": "primary_password", "password": "wrong"}
    output:
      error: |
        {"reason": "TooManyRequest"}  # Error shows it's locked

  # 2. Verify state changed via query
  - name: verify_locked
    action: admin_api_graphql
    admin_api_request:
      query: |
        query { user { accountLockout { isLocked } } }
    admin_api_output:
      result: |
        {"data": {"node": {"accountLockout": {"isLocked": true}}}}

Setting Up Test Data with SQL Fixtures

Always use SQL fixtures in the before hooks for initial data setup, not API mutations:

before:
  - type: custom_sql
    custom_sql:
      path: fixtures.sql  # Direct database INSERT statements

steps:
  # Now test functionality that operates on this data
  - action: admin_api_graphql

This is faster and more reliable than creating data via API in tests.