docker-bench-expert
GitHub专为AMT Docker Bench环境设计的专家技能，用于调试T264卡车仿真容器的启动、网络、版本兼容及DDS通信问题。提供故障排查流程，如FX-8520加油调度诊断，并支持查询版本兼容性矩阵。
Trigger Scenarios
Install
npx skills add NeverSight/learn-skills.dev --skill docker-bench-expert -g -y
SKILL.md
Frontmatter
{
"name": "docker-bench-expert",
"description": "Specialised skill for the AMT Docker Bench — a multi-container simulation of a T264 autonomous truck. Use when asked about docker_bench containers (rpk1, rpk2, rpk3, platform-sim, avi-radio, test-manager, vas-aht), Docker Compose configuration, network topology (onboard\/offboard\/CAN\/local-fms-network), NAT setup, CIC\/RPK version selection, T264 simulator, VAS:AHT integration, DDS QoS profiles, IP address remapping, overlay files, run_tests.sh, Docker Bench version compatibility, compatible FMS\/AMT version pairs, or any issue involving the Docker Bench environment. Also use for debugging container startup failures, networking problems, build errors, image pull failures, version mismatch issues, or service readiness issues in the bench. Can fetch the Docker Bench Versions Compatibility Matrix from Confluence to determine compatible version combinations."
}
Docker Bench Expert
A specialised skill for understanding, maintaining, and debugging the AMT Docker Bench environment — a collection of Docker containers that emulate the separate physical computers on a T264 autonomous haul truck for testing and validation purposes.
This skill extends the docker-expert skill with domain-specific knowledge of the Docker Bench architecture, its containers, networking, and configuration.
When to Use This Skill
- Understanding how the Docker Bench containers fit together
- Debugging container build failures, startup issues, or service readiness problems
- Analysing or modifying the
docker-compose.yml, Dockerfiles, or overlay configuration - Diagnosing networking issues between containers (onboard, offboard, CAN, FMS)
- Understanding or modifying the NAT setup (avi-radio)
- Selecting or changing CIC/RPK/T264 Sim versions
- Determining compatible FMS and AMT Docker Bench version combinations
- Resolving image pull failures or version mismatch errors
- Understanding the IP address remapping system
- Troubleshooting DDS communication or QoS profile issues
- Working with
run_tests.shor the test-manager container - Understanding the relationship between Docker Bench and the real truck hardware
FX-8520 Refuel Dispatch Playbook (Validated)
Use this workflow when a Docker Bench refuel scenario reaches Dispatched but the
asset does not move, or the assignment is quickly cancelled.
High-signal checks
Run these checks together so geometry, mode, and dispatch state are evaluated in the same time window:
- TM area state:
GET /areas/{areaId}/state(Traffic Manager) - Robot mode/restrictions:
GET /RobotControl/AssetIdentifier/{asset} - Assignment list:
GET /Assignments(Assignment Aggregator) - TM logs: search for
DispatchCancelledEvent,No MoveToArea instruction found,LoadAreaId,DumpAreaId
Copy/paste diagnostic command (local Imperium + Docker Bench)
Use this one-liner to capture the key state in one snapshot:
TOKEN=$(curl -s -X POST http://localhost:4020/Auth/locallogin -H "Content-Type: application/json" -d '{"username":"641693","password":"Autonomy@1"}' | python3 -c "import json,sys; print(json.load(sys.stdin)['tokenResponse']['token'])") && \
echo '--- TM area state ---' && \
curl -s "http://localhost:7100/areas/e387c3ae-8bc4-4278-8fdf-83943f6be9ea/state" -H "Authorization: Bearer $TOKEN" | python3 -m json.tool && \
echo '--- RobotControl AHT002 ---' && \
curl -sk "https://robot-control-service.dev.localhost/RobotControl/AssetIdentifier/AHT002" -H "Authorization: Bearer $TOKEN" | python3 -m json.tool && \
echo '--- Assignment Aggregator ---' && \
curl -s -o /tmp/aa_root.out -w "%{http_code}\n" "http://localhost:5660/Assignments" -H "Authorization: Bearer $TOKEN" && \
head -c 1200 /tmp/aa_root.out && echo && \
echo '--- traffic-manager relevant logs ---' && \
docker logs traffic-manager 2>&1 | grep -Ei "AHT002|ELI_REFUEL|no route|route|assignment|Dispatched|DispatchCancelled|MoveToArea|LoadAreaId|DumpAreaId" | tail -120
Interpret this snapshot before making geometry changes.
Known failure signature
If TM logs show both of the following, treat it as a dispatch/assignment interpretation failure (not only a mode problem):
No MoveToArea instruction found for "LoadAreaId"(and/orDumpAreaId)Handling "DispatchCancelledEvent" ...immediately after dispatch is added
Lane-aligned spawn guidance
For scripted spawn positions, derive pose from the lane geometry instead of map eyeballing:
- Get the lane segment id from TM
entryPathPoint.laneSegmentId. - Query Mine Model
GET /PathSegmentand select the matching segment. - Use the midpoint
PathSegmentPoint(indexcount//2) forXandY. - Derive heading from midpoint neighbors (
mid-1->mid+1) withatan2.
Using the midpoint centerpoint avoids truck overhang at lane ends and gives more stable initial routing.
Important note for debugging priorities
If midpoint lane-aligned spawn and autonomy mode are both correct but cancellation persists, prioritise assignment payload/dispatch contract checks over further spawn angle tuning.
Communication Style
- Default: Provide concise, plain-language explanations. Focus on what is happening, why it matters, and how to fix it. Do not assume the reader has deep Docker or networking knowledge.
- When asked for more detail: Give in-depth explanations covering the underlying Docker and networking mechanisms.
- Always provide actionable solutions — not just descriptions of problems.
Docker Bench Architecture Overview
The Docker Bench simulates a T264 autonomous haul truck using seven containers, each representing a physical computer or network device on the real truck:
| Container | Role | What It Emulates |
|---|---|---|
| rpk1 | Autonomy Management Toolkit (AMT) | The main onboard management computer running the RTK launcher, diagnostics watchdog, mine model adapter, blob sync, and FMS bridge |
| rpk2 | Compute IC (CIC) | The control computer running the Control IC software with a web interface on port 8009 |
| rpk3 | Robot Perception Kernel (RPK) | The perception computer running obstacle detection and point cloud processing |
| vas-aht | Vehicle Awareness System (VAS:AHT) | The Motium unit running VAS for position/heading reporting, notifications, and sound management via DDS |
| platform-sim | T264 Simulator | Simulates the vehicle dynamics and CAN bus of the physical truck, with a web interface on port 8123 |
| avi-radio | NAT Gateway | Emulates the AVI radio that bridges the truck's internal network to the external FMS/OT network using iptables NAT rules |
| test-manager | Test Automation (optional) | Runs ARCU automation tests; enabled via --profile test-manager |
Multi-Asset Support
Docker Bench supports running multiple instances simultaneously, each representing a different truck. The ASSET_ID variable in .env (default: AHT000) controls:
- Compose project name:
amt-docker-bench-${ASSET_ID_LOWER}(e.g.,amt-docker-bench-aht000) - Container names:
<service>-${ASSET_ID}(e.g.,rpk1-AHT000) - Network names:
<network>-${ASSET_ID}(e.g.,onboard-AHT000,offboard-AHT000)
Two conventions exist for running commands inside containers:
- Primary:
docker exec <service>-AHT000 <command>(e.g.,docker exec rpk1-AHT000 bash) - Alternative:
docker compose exec <service> <command>(e.g.,docker compose exec rpk1 bash)
The primary convention uses the full container name (service + asset ID). The alternative uses Compose service names and is useful when you do not know or want to type the asset ID. Both are valid; the documentation uses the primary convention.
Key Files
| File | Purpose |
|---|---|
| docker_bench/docker-compose.yml | Orchestrates all containers, networks, volumes, and build arguments |
| docker_bench/.env | Environment variables: versions, IPs, asset ID, patterns |
| docker_bench/run_tests.sh | Automated test orchestration script |
| docker_bench/utils/helpers.sh | Shell utility functions for logging, Docker version checks, container readiness |
| docker_bench/dockerfile-rpk1 | AMT container Dockerfile |
| docker_bench/dockerfile-rpk2 | CIC container Dockerfile |
| docker_bench/dockerfile-rpk3 | RPK container Dockerfile |
| docker_bench/dockerfile-vas-aht | VAS:AHT container Dockerfile |
| docker_bench/dockerfile-platform-sim | T264 Simulator Dockerfile |
| docker_bench/dockerfile-test-manager | Test manager Dockerfile |
| docker_bench/dockerfile-avi-radio | NAT gateway Dockerfile |
| docs/docker_bench/index.md | User-facing documentation for Docker Bench |
Network Architecture
The Docker Bench uses four networks to mirror the real truck's network topology:
Networks
| Network | Subnet | Purpose | Real-World Equivalent |
|---|---|---|---|
onboard (onboard-${ASSET_ID}) |
10.10.10.0/24 |
Internal communication between onboard computers | The truck's internal Ethernet connecting all RPKs |
offboard (offboard-${ASSET_ID}) |
10.10.1.0/24 |
Connects onboard computers to the AVI radio | The link from onboard computes to the external-facing radio |
| local-fms-network | 172.18.0.0/16 |
External FMS/cloud connectivity | The OT (Operational Technology) mine network |
can (can-${ASSET_ID}) |
Docker CAN plugin | CAN bus between RPK2 and platform-sim | Physical CAN bus on the truck |
IP Address Assignments
| Container | Onboard (eno1) | Offboard (eno2) | Other |
|---|---|---|---|
| rpk1 | 10.10.10.110 |
10.10.1.110 |
— |
| rpk2 | 10.10.10.111 |
10.10.1.111 |
CAN network |
| rpk3 | 10.10.10.112 |
10.10.1.112 |
— |
| vas-aht | — | 10.10.1.115 |
— |
| platform-sim | 10.10.10.55 |
10.10.1.55 |
CAN network |
| test-manager | 10.10.10.56 |
10.10.1.56 |
local-fms-network |
| avi-radio | — | 10.10.1.2 |
172.18.50.1 (primary), plus dedicated IPs per container |
NAT and Port Forwarding (avi-radio)
The avi-radio container performs Network Address Translation using per-container dedicated IPs on the local-fms-network. Each onboard container has its own external IP, so traffic is routed per-container rather than requiring port-specific forwarding:
| Container | Offboard IP | Dedicated FMS IP |
|---|---|---|
| rpk1 | 10.10.1.110 |
172.18.50.110 |
| rpk2 | 10.10.1.111 |
172.18.50.111 |
| rpk3 | 10.10.1.112 |
172.18.50.112 |
| vas-aht (motium) | 10.10.1.115 |
172.18.50.115 |
| platform-sim | 10.10.1.55 |
172.18.50.55 |
| test-manager | 10.10.1.56 |
172.18.50.56 |
For each container, SNAT rewrites the source address of outgoing traffic to its dedicated FMS IP, and DNAT rewrites the destination of incoming traffic to its offboard IP.
Explicit port forwarding is additionally set up for web interfaces and SSH access (especially for Windows users who cannot directly access local-fms-network):
| Port | Protocol | Destination | Service |
|---|---|---|---|
| 8009 | TCP | rpk2 | CIC web interface |
| 8123 | TCP | platform-sim | T264 Sim web interface |
| 2020 | TCP | rpk1:22 | SSH |
| 2021 | TCP | rpk2:22 | SSH |
| 2022 | TCP | rpk3:22 | SSH |
| 2025 | TCP | motium:22 | SSH |
Interface Naming System
Docker assigns default interface names like eth0, eth1 to containers. The RTK service apply-systemd-networkd-config.service (part of the base images) runs at container boot to rename these interfaces to match the systemd-networkd configuration files in /etc/systemd/network/.
The script (/usr/bin/apply_systemd_networkd_config.py) works by:
- Reading all
.networkfiles in/etc/systemd/network/ - Extracting
[Match] Name=as the target interface name - Finding the existing Docker interface (
eth0, etc.) by matching the[Network] Address=IP - Renaming the Docker interface to the target name using
ip link set
Interface Name Convention
| Container | Onboard Interface | Offboard Interface |
|---|---|---|
| rpk1 | eno1 |
eno2 |
| rpk2 | eno1 |
eno2 |
| rpk3 | eno1 |
eno2 |
| vas-aht | — | enp0s29f1 |
| platform-sim | eno1 |
eno2 |
Important: Glob Patterns in Name= Are Not Supported
The apply-systemd-networkd-config.service script reads Name= values literally. On real hardware, systemd-networkd treats Name=enp* as a glob pattern. In Docker Bench, the script would rename the interface to the literal string enp* (with asterisk). This was the root cause of FX-8560.
All Docker Bench network configs must use exact interface names (e.g. Name=eno2, Name=enp0s29f1), not glob patterns. The RPK containers handle this by removing the base image's 20-wired.network (which contains Name=enp*) and replacing it with exact-name configs (10-eno1.network, 20-eno2.network). The VAS AHT Dockerfile patches 20-wired.network in place with sed to set Name=enp0s29f1 (matching site configuration).
IP Address Remapping System
Each container runs an overwrite-ip-address-config.service at boot that rewrites IP addresses in configuration files. This allows the bench to work with different subnet configurations without manually editing every config file.
- Controlled by overlay/common/etc/overwrite_ip_address_config/overwrite_ip_address_config.xml
- Executed by overlay/common/usr/bin/overwrite_ip_address_config.py
- Searches through
/etc/systemd/network/,/etc/systemd/system/,/opt/amt/resources/,/opt/vas/config/,/opt/vas/share/,/opt/lme/cic/,/opt/lme/rpk/,/usr/share/t264_simulator/, and/persistent/ - Explicitly excludes
/opt/field-commissioning/,/opt/field-sync/,/opt/field-signal/, and/opt/field-telemetry/(field services use DDS on localhost only; no IP rewriting needed) - Replaces default IPs with actual container IPs using environment variables
Version Selection
CIC Version (rpk2)
- CIC tarballs are stored in
docker_bench/overlay/rpk2/persistent/ - The
.envvariableCIC_FIND_PATTERNcontrols which version is installed (e.g.cic*0.65*.tgz) - Set to
cic*.tgzto always install the latest available version - The
upgradeCIC.shscript in the overlay handles installation
RPK Version (rpk3)
- RPK tarballs are stored in
docker_bench/overlay/rpk3/persistent/ - The
.envvariableRPK_FIND_PATTERNcontrols which version is installed (e.g.rpk*FS65*.tgz) - Set to
rpk*.tgzto always install the latest available version
VAS:AHT Version (vas-aht)
- The
.envvariableVAS_AHT_DOCKER_BENCH_VERSIONcontrols the VAS:AHT base image version - VAS uses a separate base image (
rtk_os_docker_bench_vas_aht) built by the VAS repository, independent of the AMT base images - Default: pinned to a specific version (e.g.
4.2.1-master.6-0.x1a79a05d)
T264 Sim Version (platform-sim)
- The
.envvariableT264_SIM_VERSIONforces a specific version (in DPKG format) - If unset, the version bundled with the base image is used
Docker Bench Compatibility Matrix
The authoritative source for compatible version combinations is the Confluence page: Docker Bench Versions Compatibility Matrix
Important ordering note: The newest compatible combinations are at the bottom of the "Master" table, not the top. When scanning manually, verify this by checking that version numbers increase as you go down.
How to Fetch and Use the Compatibility Matrix
When a user asks about version compatibility, version selection, or encounters image pull/startup errors related to version mismatches, fetch the matrix from Confluence using the Atlassian MCP tool:
mcp_com_atlassian_getConfluencePage(
cloudId: "fmgl-autonomy.atlassian.net",
pageId: "1226080308",
contentFormat: "markdown"
)
The response contains a markdown table under the "## Master:" heading with these columns:
| Column | Description |
|---|---|
| FMS Version | Imperium Docker image tag (e.g. 4.1.0-master.776-0.xe4bdb8fd) |
| Imperium Repo Commit Hash | Git commit to check out the Imperium repo at |
| AMT Version | AMT Docker Bench image tag (e.g. 0.33.0-master.47-0.x531ee13b) |
| AMT Repo Commit Hash | Git commit to check out the AMT repo at |
| T264 Sim Version | T264 Simulator version (DPKG format); "(default)" means use the version bundled in the base image |
| CIC Version | CIC tarball filename pattern; "(default)" means use the latest in overlay/rpk2/persistent/ |
| RPK Version | RPK tarball filename pattern; "(default)" means use the latest in overlay/rpk3/persistent/ |
| Workarounds/Comments | Known issues, required workarounds, or empty if clean |
There is also a "## Release Candidates:" table with the same columns for RC builds.
Version Compatibility Queries
When the user asks a version-related question, fetch the page and answer using these patterns:
"What is the latest compatible version pair?"
- Look at the last row of the Master table.
- Prefer rows where the Workarounds/Comments column is empty or contains only minor notes (not "can't get the truck to move" or similar blockers).
- Report the FMS Version, AMT Version, and any default component versions (T264 Sim, CIC, RPK).
"Is FMS version X compatible with AMT version Y?"
- Search the table for a row matching both the FMS and AMT versions.
- If found, report the row and any workarounds.
- If not found, report that the combination is not in the matrix and suggest the nearest compatible pair(s).
"What AMT version works with FMS version X?"
- Search the table for rows matching the FMS version.
- There may be multiple rows (same FMS version paired with different AMT versions).
- Report all matches, noting which has the fewest workarounds.
"What FMS version works with AMT version X?"
- Search the table for rows matching the AMT version.
- Report all matches with their workarounds.
"Give me a clean version pair with no workarounds"
- Scan from the bottom of the table upward.
- Return the first (newest) row where the Workarounds/Comments column is empty.
Version String Anatomy
Understanding version strings helps identify compatibility:
- FMS:
4.1.0-master.776-0.xe4bdb8fd→ major.minor.patch-branch.buildNumber-0.xCommitHash - AMT:
0.33.0-master.47-0.x531ee13b→ same pattern - T264 Sim:
0.0.0~master.58~0.x83d258a2→ uses~instead of-(Debian package format) - CIC:
cic_0.62.0.tgz→ tarball filename - RPK:
rpk-533_20260301-175657-release-63152_FS62.1.tgz→ tarball with date and FS (Functional Safety) version
The commit hash suffix (after 0.x) can be used to check out the corresponding Git repository at the exact commit that was tested.
Applying Versions
Once a compatible pair is identified, apply them as follows:
FMS version (Imperium side):
# In imperium/Tools/local-environment/
./setVersion.ps1 <FMS_VERSION>
# e.g. ./setVersion.ps1 4.2.1-master.207-0.xc5d2f714
AMT Docker Bench version (AMT side):
- Set
AMT_DOCKER_BENCH_VERSIONindocker_bench/.env, OR - Pass
-dockerBenchVersion <AMT_VERSION>to Imperium'sstartAsset.ps1
T264 Sim version (if not "(default)"):
- Set
T264_SIM_VERSIONindocker_bench/.env
CIC/RPK versions (if not "(default)"):
- Ensure the correct tarball exists in
overlay/rpk2/persistent/(CIC) oroverlay/rpk3/persistent/(RPK) - Set
CIC_FIND_PATTERN/RPK_FIND_PATTERNindocker_bench/.envto match
Imperium repo commit (if the matrix specifies a different commit from the FMS version tag):
- Some rows list an Imperium Repo Commit Hash that differs from the hash in the FMS version string. This means the Docker images are from one commit but the local tooling scripts (PowerShell scripts, compose files) must be from a different commit for compatibility.
- Check out the specified commit:
cd ~/workspace/imperium && git checkout <commit_hash>
Common Version Mismatch Symptoms
| Symptom | Likely Cause |
|---|---|
| Image pull fails ("manifest not found") | FMS version tag doesn't exist in Artifactory for one or more services. Check the matrix for a version that has all images published. |
| Truck won't move / map sync fails | FMS↔AMT version incompatibility. The matrix notes often describe this (e.g. "can't get the truck to move"). |
ALL_POSE_TIMEOUT in FMS Bridge |
Known issue with several version pairs. Restart amt.service or asset-dds-gateway. Check the matrix Workarounds column. |
| RPK MAP LANE BREACH fault on startup | Known RPK issue with some versions. Clear via CIC Test Page or RCS Health Reset. |
| Path permission / hash faults (797, 849) | Do a Health Reset in RCS and delete remaining diagnostic IDs with controlled stop latched. |
DEADLINE_MISSED_PATH_SEGMENT_PERMISSIONS |
Restart mine-model-distributor and mine-model-service. |
Container Capabilities and Permissions
The containers require elevated Linux capabilities to function:
| Capability | Used By | Why |
|---|---|---|
CAP_NET_ADMIN |
All RPKs, vas-aht, platform-sim, test-manager | Network configuration, IP remapping, interface management |
CAP_SYS_MODULE |
All RPKs, vas-aht, platform-sim, test-manager | Kernel module loading (CAN drivers) |
CAP_SYS_PTRACE |
rpk1 | Process tracing for diagnostics |
CAP_IPC_LOCK |
rpk2, rpk3, vas-aht, platform-sim | Lock memory for real-time performance |
CAP_SYS_NICE |
rpk2, rpk3, vas-aht, platform-sim | Set real-time scheduling priorities |
CAP_SYS_TIME |
vas-aht | System time modification (required by chrony) |
NET_ADMIN |
avi-radio | NAT and iptables rules |
NET_RAW |
avi-radio | Raw socket access for network translation |
DDS Communication
The bench uses RTI Connext DDS for inter-container communication.
DDS Domain IDs
| Domain | Name | Used By |
|---|---|---|
| 20 | CIC | FMS Bridge (rpk1), CIC (rpk2), RPK (rpk3), ARCU tests |
| 30 | VAS | VAS Bridge (vas-aht), field-sync, field-signal |
| 50 | IAI | Asset Shadow (Imperium), VAS Bridge telemetry route |
| 51 | Field | field-commissioning, field-sync, field-signal, field-telemetry |
DDS Configuration
- QoS profiles are defined in overlay/common/persistent/USER_QOS_PROFILES.xml
- FMS Bridge communicates externally via unicast through the avi-radio NAT
- The
ASSET_SHADOW_DDS_PARTICIPANT_IDenvironment variable configures the FMS Bridge participant - VAS:AHT communicates via DDS over the offboard network interface (UDP, no TCP ports required)
- Field services bind DDS to
127.*(localhost) only, routing through VAS Bridge
Spoof Scripts
Located in docker_bench/overlay/common/persistent/, these scripts simulate data that would come from real truck sensors or systems:
| Script | What It Simulates |
|---|---|
spoof_aci_pose.sh |
Vehicle position and attitude (published every 100ms) |
spoof_aci_robot_mode.sh |
Robot operational mode |
spoof_health_event_*.sh |
Various health/diagnostic events |
spoof_actioned_diagnostics_perception_stop.sh |
Perception system diagnostic actions |
spoof_health_reset_perception_stop.sh |
Perception health status reset |
VAS:AHT Container
The vas-aht container runs the Vehicle Awareness System for Autonomous Haul Trucks. On the real truck, VAS:AHT runs on the Motium unit (10.10.1.115). It acts primarily as a network bridge — receiving commands and publishing status via DDS, with the centralised OCS handling all positioning and decision-making.
VAS:AHT Key Details
- Base image:
rtk_os_docker_bench_vas_aht(built by the VAS repository, not AMT) - Version variable:
VAS_AHT_DOCKER_BENCH_VERSION(independent ofAMT_DOCKER_BENCH_VERSION) - Network: Offboard only (
10.10.1.115) — not connected to onboard or CAN - No exposed ports: VAS:AHT communicates exclusively via DDS over UDP
- IP remapping:
overwrite-ip-address-config.servicescans/opt/vas/config/and/opt/vas/share/for hardcoded IPs
Disabled Imperium Field Services
One Imperium Field .NET service is pre-installed in the VAS base OS image but is disabled in dockerfile-vas-aht because it is not required in Docker Bench:
| Service | Why Disabled |
|---|---|
field-commissioning-ui.service |
Web UI for commissioning; confirmed not required in Docker Bench context (Craig Brogle) |
Enabled Imperium Field Services
field-commissioning.service is enabled in Docker Bench to support manned mode commissioning workflows. It is a .NET service providing an HTTP API and DDS participant for calibration, survey import, and operator login.
Communication path:
- DDS domain 51 bound to
127.*(localhost) - routes through VAS Bridge in the same container - HTTP (Kestrel) bound to
localhost:4050- not externally reachable - The
172.18.*entry inAllowInterfacesListis a no-op (no matching interface in Docker Bench) /opt/field-commissioning/is explicitly excluded from IP address remapping
Verification:
docker exec vas-aht-AHT000 systemctl status field-commissioning.service
docker exec vas-aht-AHT000 curl -s -w "\n%{http_code}" http://localhost:4050/Health/liveness
docker exec vas-aht-AHT000 curl -s http://localhost:4050/Health/version
docker exec vas-aht-AHT000 journalctl -u field-commissioning --no-pager -n 30
field-sync.service is enabled in Docker Bench to support manned mode mine model and asset data synchronisation. It provides a DDS participant for distributing mine model, area, zone, and asset state data to the in-cab display (Centurion).
Communication path:
- DDS domain 51 bound to
127.*(localhost) - routes through VAS Bridge in the same container - DDS domain 30 (additional VAS domain bridge participant)
- HTTP (Kestrel) bound to
localhost:5510- not externally reachable - Redis connection to
localhost:6379(port corrected from base image default of 6380) /opt/field-sync/is explicitly excluded from IP address remapping
Expected warnings (non-fatal in standalone Docker Bench):
- gRPC
Unavailableerrors for Asset Shadow and Mine Model Distributor (FMS Core not reachable) - RabbitMQ connection warnings (no AMQP broker in container)
Verification:
docker exec vas-aht-AHT000 systemctl status field-sync.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:5510/
docker exec vas-aht-AHT000 journalctl -u field-sync --no-pager -n 30
field-signal.service is enabled in Docker Bench to support manned mode real-time data delivery to Centurion (the in-cab display). It hosts a SignalR hub serving operational data (assignments, materials, notifications, asset state, etc.) from Redis to connected SignalR clients.
Communication path:
- DDS domain 51 bound to
127.*(localhost) - routes through VAS Bridge in the same container - DDS domain 30 (additional VAS domain bridge participant)
- HTTP (Kestrel) bound to
localhost:5000(default, not externally reachable) - Redis connection to
localhost:6379(default port, no fix needed) - SignalR hub hosted on the Kestrel port (Centurion connects here)
/opt/field-signal/is explicitly excluded from IP address remapping
Dependencies: redis-server.service (hard), field-sync.service (ordering only)
Expected behaviour (normal in standalone Docker Bench):
- No connected SignalR clients (Centurion is not running)
- Redis reads return empty data (until field-sync populates it)
- No gRPC or RabbitMQ errors (field-signal does not connect to external services)
Verification:
docker exec vas-aht-AHT000 systemctl status field-signal.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:5000/
docker exec vas-aht-AHT000 journalctl -u field-signal --no-pager -n 30
field-telemetry.service is enabled in Docker Bench to support the health event pipeline from vehicle telemetry to the in-cab display. It subscribes to telemetry data via DDS (domain 51), extracts health events, and stores them in Redis for consumption by field-signal.
Communication path:
- DDS domain 51 bound to
127.*(localhost) - subscribes to/telemetrytopic on the field domain - HTTP (Kestrel) bound to
localhost:4200(set via systemd drop-in to avoid port 5000 conflict with field-signal) - Redis connection to
localhost:6379- stores health events atLocalAsset:FieldHealth /opt/field-telemetry/is explicitly excluded from IP address remapping
Operating mode: DDS input mode (UseDdsForTelemetryData: true) is the production default. No RaptorCore hardware required.
Dependencies: redis-server.service (hard)
Expected behaviour (normal in standalone Docker Bench):
- DDS participant on domain 51 subscribes to
/telemetrytopic - Redis key
LocalAsset:FieldHealthexists but is empty (ZCARD = 0) because telemetry data arrives on VAS domain 30 via VAS Bridge but field-telemetry subscribes on field domain 51; no domain 30→51 bridge exists in standalone Docker Bench - Health events would be populated when running with Imperium local-env (which bridges telemetry to domain 51)
- No hardware errors, no gRPC errors, no RabbitMQ dependency
Verification:
docker exec vas-aht-AHT000 systemctl status field-telemetry.service
docker exec vas-aht-AHT000 curl -s -o /dev/null -w "%{http_code}" http://localhost:4200/
docker exec vas-aht-AHT000 journalctl -u field-telemetry --no-pager -n 30
The remaining disabled service (field-commissioning-ui) is not required in Docker Bench.
In Imperium, the FMS-to-VAS integration path is: FMS Core -> Asset Shadow (separate container) -> DDS via avi-radio NAT -> VAS:AHT. Field services are handled externally by dedicated Imperium containers on local-fms-network.
VAS:AHT Dockerfile Setup
- Installs Netskope CA certificates
- Copies shared overlay files (SSH keys, QoS profiles, spoof scripts)
to
/persistent/ - Sets up SSH access for
rootandrtkuseraccounts - Installs and enables
overwrite-ip-address-config.service - Patches
20-wired.networkto replace globName=enp*with exactName=enp0s29f1(FX-8560 — the base image uses a glob thatapply-systemd-networkd-config.servicereads literally) - Sets
registration_state="Measured"in/etc/robot_model/robot_model.xml(base image ships with"Default", causing nodes to exit) - Sets
robot_class="DumpTruck"in/etc/robot_model/robot_model.xml(base image ships with"Undefined") - Disables
field-commissioning-ui.service(not required in Docker Bench) - Fixes field-sync Redis port (base image uses 6380, VAS:AHT Redis runs on 6379)
- Fixes field-telemetry Kestrel port (base image defaults to 5000, conflicts with field-signal; set to 4200 via systemd drop-in)
Troubleshooting Workflows
Bench Won't Start
- Check Docker version: requires >= 27.0.3. Run
docker --version. - Verify the Docker CAN plugin is installed:
docker plugin ls. Therun_tests.shscript installs it automatically. - Check that base images are available:
docker images | grep rtk_os_docker_bench. - Verify
.envhas validAMT_DOCKER_BENCH_VERSION(e.g.latest-master). - Check for port conflicts on the host:
ss -tlnp | grep -E '8009|8123|2020|2021|2022'.
Container Crashes or Won't Stay Running
- Check logs:
docker compose logs <service>(e.g.docker compose logs rpk1). - Check exit code:
docker inspect <container> --format='{{.State.ExitCode}}'. - Verify the
overwrite-ip-address-configservice ran successfully — IP mismatches cause failures downstream. - For rpk2/rpk3: check that the CIC/RPK tarball matched the
FIND_PATTERNand installed correctly. - Shell into a failed container interactively:
docker run -it --entrypoint /bin/bash <image>.
Services Not Ready (run_tests.sh Timeout)
The test script waits for these processes:
rpk1: rtk_launcher, rtk_log_recorder, amt_diagnostics_watchdog, mine_model_adapter, task_spooler, blob_sync_node, fms_bridge
platform-sim: /usr/bin/t264_simulator
If the readiness check times out:
- Shell into the container:
docker compose exec rpk1 bash. - Check
systemctl status amt.service(rpk1) or the relevant service. - Check
journalctl -u amt.service --no-pager -n 50for errors. - Verify DDS licence is present and valid (RTI Connext).
Networking Issues Between Containers
- Verify containers are on the correct networks:
docker network inspect <network>. - Test connectivity:
docker compose exec rpk1 ping 10.10.10.111(rpk2 onboard). - Check that applications bind to
0.0.0.0, not127.0.0.1. - For DDS issues: verify QoS profiles match across containers and domain ID is consistent.
- For FMS connectivity: check avi-radio NAT rules with
docker compose exec avi-radio iptables -t nat -L -n. - If an interface has an unexpected name (e.g.
enp*with a literal asterisk), check/etc/systemd/network/for glob patterns inName=. Theapply-systemd-networkd-config.servicereadsName=literally — use exact names only. See the Interface Naming System section.
CAN Bus Issues
- Verify the Docker CAN plugin is running:
docker plugin ls. - Check CAN interfaces inside containers:
docker compose exec rpk2 ip link show. - Decode CAN messages from platform-sim:
docker compose exec platform-sim bash -l -c "candump vcan10 | cantools decode --single-line /t264-sim/libraries/dbc/DBW_OAL.dbc"
Build Failures
- Check if base images exist: the Dockerfiles expect pre-built
rtk_os_docker_bench_*images. - Before changing
AMT_DOCKER_BENCH_VERSION, consult the Docker Bench Compatibility Matrix section above. Fetch the matrix from Confluence and pick a known-compatible combination (newest compatible entries are at the bottom of the table). - Verify Netskope CA certificate is accessible (used for SSL in all containers).
- For test-manager: ensure
USER_GITHUB_PATis set and the GitHub token hasrepo,read:org,read:userpermissions with SSO enabled. - Check that overlay files referenced in
COPYdirectives exist at the expected paths.
Useful Commands
| Task | Command |
|---|---|
| Start the bench | cd docker_bench && docker compose up --build -d |
| Start (force pull latest images) | cd docker_bench && docker compose up --build --pull always -d |
| Start with test-manager | docker compose --profile test-manager up --build -d |
| Stop the bench | docker compose down |
| View all logs | docker compose logs -f |
| View one service's logs | docker compose logs -f rpk1 |
| Run automated tests | ./run_tests.sh |
| SSH into rpk1 | ssh -p 2020 rtkuser@localhost or docker compose exec rpk1 bash |
| SSH into rpk2 | ssh -p 2021 rtkuser@localhost |
| SSH into rpk3 | ssh -p 2022 rtkuser@localhost |
| SSH into vas-aht (motium) | ssh -p 2025 rtkuser@localhost or docker compose exec vas-aht bash |
| CIC web UI | Open http://localhost:8009 |
| T264 Sim web UI | Open http://localhost:8123 |
| Run ARCU tests manually | docker exec -itu rtkuser rpk1-AHT000 /opt/amt/bin/arcu -d 20 -i /opt/amt/share/arcu/arcu_config.xml |
| Check container readiness | docker ps --format 'table {{.Names}}\t{{.Status}}' |
| Inspect NAT rules | docker exec avi-radio-AHT000 iptables -t nat -L -n |
Imperium Integration
Docker Bench can integrate with the Imperium local environment for end-to-end testing:
- Use
startAsset.ps1 -useDockerBenchto connect Imperium to the bench - The asset ID must be configured in
.envto match the Imperium environment (e.g.,ASSET_ID=AHT001) withIsAutonomousAssetset totrue - The Imperium startup script (
startAssetDockerBench.ps1) generates an.env-asset-{ASSET_ID}-docker-benchfile with asset-specific network settings fromassets.json - An Asset Shadow container (on
local-fms-network) bridges FMS Core services to Docker Bench via DDS through avi-radio - VAS:AHT integrates via the same avi-radio NAT path — its offboard IP (
OFFBOARD__MOTIUM__IP) is SNAT/DNAT'd to a dedicated FMS IP (LOCAL_FMS_NETWORK__AVI_RADIO_MOTIUM__IP) - SSH access to VAS:AHT is available on port 2025 via avi-radio port forwarding
Known Bugs
RPK Map Lane Breach Stuck / Unclearable (APADA-14702)
Details: https://fmg-lme-autonomy.atlassian.net/browse/APADA-14702
Observed with: FMS 4.1.0-rc.130 (April 2026)
Symptom: CIC raises a repeated "RPK map lane breach" diagnostic that activates and deactivates repeatedly while the truck is driving manually inside a locked AOZ. Once the AOZ is unlocked, the diagnostic clears from CIC — but the FMS retains the health event as unclearable (IsClearable = false, event type BOUNDARY_INTERACTION). The "RPK map lane breach" persists in the Asset Details panel and survives mode transitions (Manual → Autonomy). Standard RCS Health Reset and latching/unlatching do not clear it.
Root cause: Under investigation. The health event published by CIC/RPK is recorded in RCS with IsClearable = false. Once in this state, the normal "Clear to Proceed" action is not available in the RCS UI.
Workaround (requires FMS database access):
-
Stop the RCS service:
docker service scale fms_robot-control-service=0 -
Clear restrictions in the RCS database:
UPDATE [RobotHealth_Eli_Int].[dbo].[RobotControls] SET [RestrictionState] = N'{"Restrictions":[]}' WHERE [AssetIdentifier] = '<ASSET_ID>'; -
Clear health events in the RCS database:
UPDATE [RobotHealth_Eli_Int].[dbo].[HealthEvents] SET [IsClearable] = 1 WHERE [AssetIdentifier] = '<ASSET_ID>' AND [EventType] = 'BOUNDARY_INTERACTION'; -
Restart the RCS service:
docker service scale fms_robot-control-service=1
Notes:
- Replace
<ASSET_ID>with the actual asset identifier (e.g.DT5395) - A simpler but less targeted alternative is to spoof a clearable diagnostic via RCS with
"isResettable": trueand"responseAction": "Proceed"— but this was observed to not work for this specific event - This issue may also appear during Docker Bench integration testing with Imperium when AOZ locking scenarios are tested
FMS Integration Lessons (FX-8520)
Lessons learned from the FX-8520 refuelling integration test development, relevant to any Docker Bench + FMS testing:
Eliwana Seed Data Issues
The Eliwana mine model seed data (seedFMS.ps1) has several problems for refuelling tests:
| Problem | Impact | Workaround |
|---|---|---|
ELI_REFUEL area type is RefuelBay (enum=1) |
TM's IsDynamicArea() filter ignores it entirely |
Delete and recreate as Refuel (enum=6) |
| Entry/exit lanes share the same endpoint (co-located) | Drive-through path planning is mathematically impossible | Create new N→S drive-through lanes via mine model DEBUG API |
| Area polygon ~52 m × 26 m | Fails minimum inscribed circle requirement (20 m radius) for Refuel type |
Use 120 m × 120 m boundary |
Coordinate System
All mine model coordinates are in centimetres (mine grid). NTS geometry functions divide by 100 internally when computing metric distances. When creating lanes or boundaries via the API, all coordinate values must be in centimetres.
Traffic Manager Seeding Behaviour
- TM seeds its area read model once on startup using Polly retry (exponential backoff, 15s cap)
- TM's health check passes independently of seeding completion — a healthy TM does NOT mean areas are loaded
- If
seedFMS.ps1runs AFTER TM starts, TM never ingests the areas. Always restart TM after bulk seeding:docker restart traffic-manager - Poll
GET /areas/{areaId}/statefor up to 180s to confirm ingestion
Spatial Association Requires State Change
The spatial-association-service only fires entry/exit events on position changes. If an asset was already at the target coordinates from a previous run:
- Teleport asset outside the area (offset by ≥ 200 m)
- Wait 3-5 seconds for the exit event
- Teleport asset to the target position inside the area
- Wait for the entry event
Without this pattern, the asset is never detected in the area.
Mine Model DEBUG API Notes
POST /DEBUG/Lanecreates a lane AND auto-generates child LaneSegment(s)- The response structure is
.lane.segments[]— NOT.laneSegments[] - Lane segment IDs in the response are what you pass to
POST /Areaas entry/exit segment references NTS Polygon.Contains()uses DE-9IM Interior model: points exactly on the boundary are excluded. Queue points must be offset ≥ 1 m (100 cm) from edges.POST /ArearequirescontrolBehaviorsas a JSON-encoded string field within the request body, not a nested object
PowerShell Serialization Gotchas
When scripting mine model REST calls from PowerShell:
Invoke-RestMethodauto-deserializes and flattens nested JSON (GeoJSON, lane segments). UseInvoke-WebRequest+ConvertFrom-Jsonfor complex responses.- GeoJSON coordinate arrays must be built as here-strings, NOT via
ConvertTo-Json(which double-serializes nested arrays). - The
contentTypeparameter must be"application/json"— PowerShell does not default to this.
-dockerBenchUseLatest Artifactory Risk
The -dockerBenchUseLatest parameter queries the Artifactory Storage REST API, not the Docker registry. docker login success does NOT guarantee Storage API access (different backends). The API frequently returns 504 Gateway Timeout during CI load.
Recommendation: Always use hardcoded default versions from the compatibility matrix.
ACI V1.0 → V1.1 Breaking Change Boundary
| FMS Version Threshold | Minimum AMT Version | ACI Version |
|---|---|---|
< 4.2.1-master.278 |
Any AMT version | V1.0 |
>= 4.2.1-master.278 |
0.34.0-master.24 |
V1.1 |
FMS and AMT versions must be updated together when crossing this boundary. The default test versions (FMS .207 + AMT .16) are below the threshold (V1.0).
VAS:AHT Mode-Gated Telemetry
VAS:AHT only forwards fuel telemetry to FMS when the truck is in MANUAL mode:
- In AUTONOMY mode: fuel telemetry is processed but NOT forwarded (mode gate blocks it)
- In MANUAL mode: fuel telemetry flows VAS:AHT → DDS domain 50 → Asset Shadow → FMS
This is intentional — autonomous refuelling detection requires the truck to be stationary in MANUAL mode. If fuel telemetry never appears in FMS, verify the truck's robot mode state is MANUAL.
SQL Workarounds for Docker Bench FMS
seedFMS.ps1does NOT reliably setPasswordHash— write it directly via SQL UPDATESET QUOTED_IDENTIFIER ONis required before any UPDATE onAspNetUsers(filtered indexes)stopAll.ps1does NOT remove Docker networks — orphanedlocal-fms-networkblocks subsequentstartSql.ps1. Clean up withdocker network rm local-fms-network
References
Version History
- e0220ca Current 2026-07-05 23:24
