SGLang CI Workflow Orchestration Guide

This skill covers the CI infrastructure layer — how tests are dispatched, gated, and aborted on failure across stages. For test authoring (templates, fixtures, registration, which tier a test belongs to), see the write-sglang-test skill. To run the tests a change touches on a PR, see rerun-test.

The workflow files settle any disagreement with this guide. Counts that change with the test tree (shard counts, job counts, suite lists) are left out on purpose; read them from the files named below.

Naming Conventions

• Suite: {stage}-test-{runner_config}, generated from each test's register_cuda_ci(stage=..., runner_config=...) (e.g. base-b-test-1-gpu-small, nightly-test-8-gpu-b200)
• runner_config: a runner-pool key (1-gpu-small, 4-gpu-h100, 2-gpu-gb300); scripts/ci/runner_configs.yml maps it to the physical runner label. Runner configs are independent of tiers: any tier can run any runner config.
• Test group: directory-level group under test/registered/ (e.g. hicache maps to test/registered/hicache/test_*.py)

Tiers

pr-test-finish (base) and lint are the required checks for merging. Which tier a test belongs to is in write-sglang-test, Choosing a Tier. /rerun-test runs any registered test on a PR, whatever its tier.

Key Files

Architecture Overview

pr-test.yml on a PR:

1. check-changes detects which packages changed (main_package, sgl_kernel, jit_kernel, multimodal_gen, ...) and computes the partitions; call-gate runs pr-gate.yml (draft, label, rate limit). Kernel and rust-ext builds start alongside.
2. Base A (pre-flight): base-a-test (1-gpu-small, one shard) and base-a-test-cpu.
3. wait-for-base-a, then Base B: the base-b-test matrix, one row per runner config.
4. wait-for-base-b, then Base C: the base-c-test matrix and base-c-test-aarch64 (GB300 rows).
5. Kernel, JIT kernel, and multimodal-gen tests run as called workflows beside the stages.
6. pr-test-finish aggregates every job and fails on any failure or cancellation.

Each stage job is a matrix over { runner_config, run_timeout, timeout_per_file } rows that calls _pr-test-stage.yml; that job runs run_suite.py --suite {stage}-test-{runner_config} once per shard. Extra, nightly, and weekly reuse the same reusable stage.

Every stage job runs check-pr-test-health after checkout: if any job in the run has already failed, it fails fast with the root cause. Scheduled runs skip the wait-for-base-* jobs and the health check, and run all stages in parallel.

Fail-Fast Layers

4 layers of fail-fast, from fine to coarse:

• Layer 1: -f flag appended to all python3 -m pytest / unittest invocations in ci_utils.py
• Layer 2: --continue-on-error flag in run_suite.py — off for PRs, on for scheduled runs
• Layer 3: skipped when SKIP_PR_TEST_HEALTH_CHECK is set (scheduled, run_all_tests, test_parallel_dispatch); filters out cascade failures to show only root cause jobs
• Layer 4: wait-for-base-* jobs run only on pull_request events without test_parallel_dispatch

The control labels bypass-fail-fast, parallel-stages, max-concurrency, and highest-priority (all three) relax these limits for one PR; .github/scripts/ci-labels.cjs resolves them, and the contribution guide describes each.

Execution Modes

A single test, class, or method on a PR runs through /rerun-test, not a stage dispatch.

Stage Gating (wait-for-jobs action)

wait-for-base-a and wait-for-base-b are lightweight ubuntu-latest jobs that poll the GitHub Actions API.

1. Calls listJobsForWorkflowRun to list all jobs in the current run
2. Matches jobs by name prefix (matrix shards are named like base-b-test-1-gpu-small (3)), so the stage jobs' name: is load-bearing
3. If any matched job has conclusion === 'failure' → fail immediately
4. If all matched jobs are completed and the count matches expected_count → success
5. Otherwise → sleep and retry, up to max-wait-minutes (240 for base-a, 350 for base-b). Keep budgets under the 6h hard cap on GitHub-hosted runners: a wait killed by the cap ends cancelled, not timeout. The base-b wait sets fail-on-timeout: 'false', so a timeout lets base-c start; pr-test-finish still fails the run on any base-b failure

Each spec entry reads its expected_count from check-changes.outputs.partitions['<suite>'].size, so shard counts never need a hand edit. A new runner row in base-a or base-b still needs its own {"prefix": ..., "expected_count": ...} entry in the wait job.

Cross-Job Fail-Fast (check-pr-test-health action)

Composite action called after checkout in the reusable stage workflows and in the kernel and multimodal-gen workflows.

1. Queries listJobsForWorkflowRun for the current workflow run
2. Keeps root cause failures only — failed jobs whose failing step is not the health check itself (a job that failed fast is excluded)
3. If any are found → core.setFailed() with their names; later steps skip through the default if: success()
4. Otherwise the step succeeds

It checks every job in the run, not only the current stage. Error message example:

Fail-fast: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)

Within-Suite Failure Handling

Controlled by run_unittest_files() in python/sglang/test/ci/ci_utils.py.

Retry classification: a failure matching NON_RETRIABLE_PATTERNS (import, syntax, type errors, OOM, segfault, ...) is not retried; one matching RETRIABLE_PATTERNS (accuracy and performance assertions) or INFRA_RETRIABLE_PATTERNS is. An unknown AssertionError is retried; other unknown failures are not. The lists are in ci_utils.py.

check-changes sets continue_on_error (true for scheduled runs and run_all_tests), and _pr-test-stage.yml passes it to run_suite.py as --continue-on-error.

Test Partitioning

scripts/ci/utils/compute_partitions.py runs in check-changes and emits, per dispatched suite, {size, arr, max_parallel}:

• Per-file time: the live estimate from the sglang-ci-stats model (model.json), falling back to the file's est_time.
• Shard count: enough shards that each fits 0.75 * run_timeout of its stage row (0.75 is the inverse of LPT's 4/3 worst case). Lowering a row's run_timeout buys more shards, not shorter ones. base-a-test-1-gpu-small is pinned to one shard.
• max_parallel: size // 3 (at least 1) for GPU suites; all shards for CPU suites, scheduled runs, and max-concurrency.

Within a suite, ci_register.py:auto_partition() assigns files to shards with the LPT heuristic: sort by time descending (filename as tie-breaker), then give each file to the shard with the least total time.

The Compute partitions step summary lists every suite's size and max_parallel for that run. Multimodal diffusion uses python/sglang/multimodal_gen/test/run_suite.py, not test/run_suite.py.

check-changes Job

main_package gates the base stages; sgl_kernel builds the PR-local kernel wheel, runs the kernel suites, and switches B200 jobs to the kernel-build runner labels; jit_kernel and multimodal_gen gate their workflows. The full output list is in _pr-test-check-changes.yml.

Concurrency Control

pr-test.yml groups runs by pr-test-{event_name}-{PR number or ref name}-{git_ref or 'all'}:

• event_name keeps scheduled runs apart from fork PRs whose branch is named main
• a PR keys on its number, since head_ref has no owner and two forks with one branch name would cancel each other

cancel-in-progress is true except for workflow_call, so a new push cancels the old PR run.

How To: Add a Runner to a Tier

Adding a test to a runner that its tier already runs needs no workflow edit: stage= and runner_config= on the registration are enough. When the tier lacks that runner, add one row; never move the test to another runner to fit a tier.

1. Add { runner_config: <key>, run_timeout: ..., ... } to the tier's matrix: the stage job in pr-test.yml (base-a-test, base-b-test, base-c-test, or base-c-test-aarch64 for aarch64 runners), the extra-test matrix in pr-test-extra.yml (with its stage and, for GB300, arch: aarch64), or the matrix in nightly-test-nvidia.yml / weekly-test-nvidia.yml.
2. Add the suite name {stage}-test-{runner_config} to PER_COMMIT_SUITES or NIGHTLY_SUITES in test/run_suite.py; validate_all_suites rejects a registration whose suite is not listed.
3. For a base-a or base-b row, add its prefix to the wait-for-base-* spec.
4. A new runner config (a new machine type) also needs its entry in scripts/ci/runner_configs.yml.

A suite left with no tests makes its stage row fail (the partition lookup returns null): remove the row when moving the last test off a runner.

How To: Debug CI Failures

Slash Commands

Handled by scripts/ci/utils/slash_command_handler.py → .github/workflows/slash-command-handler.yml. The commands, their permissions, and their limits are in the contribution guide.

Base CI covers only the core default paths; tests in the extra, nightly, and weekly tiers do not run on a normal PR. To cover what a change touches, run those tests with /rerun-test; the rerun-test skill describes how to pick them.

Opt-in CI tiers

An opt-in tier is a set of workflows a PR enters by carrying run-ci-<tier> on top of run-ci: nv-extra selects pr-test-extra.yml (with its Xeon CPU coverage), and amd-extra selects pr-test-amd-extra.yml. Each tier is one entry in scripts/ci/opt_in_ci_tiers.json; the slash handler reads it for /tag-run-ci-label <tier>, /tag-and-rerun-ci <tier> and /run-<tier>-ci, so adding a tier needs no handler change. A tier's workflows follow one convention, enforced by scripts/lint/check_opt_in_ci_tiers.py (pre-commit):

1. Gate through pr-gate.yml with require-label: run-ci-<tier>. The gate live-fetches labels, so a missing opt-in fails the gate job (red run) and a rerun after labeling passes. Keep this decision out of event-payload expressions, which do not refresh on rerun.
2. No labeled trigger. GitHub's rerun API reuses the original event payload, so a run started by an unrelated label event and skipped by an if on that payload stays skipped on rerun. Adding a label after a push takes a rerun instead (/tag-and-rerun-ci <tier> or /run-<tier>-ci).
3. Every directly selectable workflow that accepts the label is listed in the tier, so /run-<tier>-ci re-runs all of them. Reusable jobs chained from a listed workflow stay part of that run.

Tier names base, cpu, extra, failed and full are reserved: they collide with the base aliases, /run-cpu-ci, the hardware-specific Extra syntax, /rerun-failed-ci and /run-full-ci.