Skills Eval Triage

Purpose

Read the committed evaluation report, identify actionable findings per skill, apply targeted fixes to SKILL.md or scripts, and verify locally — without re-running the full eval suite.

Scope: T1 (schema / security / quality / lint) and T2 (context dedup). T3 neutral verdicts have an interpretation guide but no automated fix path; they require authoring judgment.

Prerequisites

• Repo checkout with _logs/ populated from a recent local run, or the committed tools/ci/gitlab/skills/report/ tree from the last pipeline.
• skillevaluator installed (for local re-validation): pip install skillevaluator==0.8.3
• NVIDIA_INFERENCE_KEY set (needed only if re-running T1 with --llm-verify).

Reading the report

Locate the report using the first source that exists:

If tools/ci/gitlab/skills/report/ does not exist yet (first ever run), regenerate it from the local logs:

python3 tools/ci/gitlab/skills/update_scorecard.py \
  --logs-dir _logs --skills-root skills \
  --state tools/ci/gitlab/skills/report/SKILL_HEALTH.json \
  --origin local --commit "$(git rev-parse HEAD)"

python3 tools/ci/gitlab/skills/consolidate_reports.py \
  --logs-dir _logs \
  --state tools/ci/gitlab/skills/report/SKILL_HEALTH.json \
  -o tools/ci/gitlab/skills/report/index.html

Then open tools/ci/gitlab/skills/report/index.html. Sort by Block descending to surface blocking skills first. For each blocking or failing skill:

1. Click its T1 link → per-skill static report with full finding list and fix suggestions.
2. Click its T3 link → agent eval report with per-dimension scores and trial traces.
3. Cross-reference SKILL_HEALTH.json for the last-known commit and execution status.

Key columns:

• Block — YES (T1) means a critical or high T1 finding; YES (T3) means T3 execution failed or verdict is fail. Both gate a merge.
• T1 findings — worst-severity pill. clean means no C/H findings; advisories are shown in the sub-label (3M 2L).
• T3 verdict — pass / neutral / fail / ⚠ failed (infrastructure error).
• T3 score / lift — neutral with a good score (>80%) means the skill is valid but did not measurably beat the no-skill baseline.

T1 triage decision table

Work findings in severity order: Critical → High → Medium. Low findings are advisory only.

Schema (SCHEMA-*)

Security (SECURITY-HIGH → blocking)

Security findings that survive --llm-verify are more likely genuine. Read the Fix block in the T1 report — it is generated per-instance and names the exact file and line.

Before fixing, classify the finding:

Common false-positive signatures for Isaac Sim skills:

• "The script's core function is to send arbitrary Python code to a running server" — design characteristic of isaac-sim-remote.
• "executes any registered omni.kit.commands command by name" — design characteristic of command-dispatching skills.
• "accepts and executes arbitrary Python" — design characteristic of the python_server architecture.

These should have been downgraded by --llm-verify. If they weren't, add a ## Security section to the SKILL.md explaining the threat model and why the design is intentional, then re-run.

Code Risk (Code Risk Analysis)

Quality (QUALITY-MEDIUM → advisory, not blocking)

Quality advisories reduce the T1 grade but don't block unless the overall grade drops to D/F. Address them when the grade is B or lower or when fixing other issues in the same skill.

Script lint (SCRIPT_LINT-MEDIUM → advisory)

These never block a merge. Fix when they appear alongside other issues in the same script.

T2 triage

T2A — intra-skill context dedup

DUPLICATE-HIGH in the context-optimization-check log means two sections of the SKILL.md are semantically near-identical. Consolidate: keep the more detailed version, remove or link the duplicate.

T2B — inter-skill similarity

SIMILARITY-HIGH in the Tier 2B log means this skill overlaps heavily with another. Options:

• Add a ## Related Skills section naming the overlapping skill and explaining when to choose one over the other.
• If the overlap is substantial, consider merging the skills or making one call the other via meta-skills composition patterns.

T3 interpretation

T3 verdicts are not auto-fixed — they require authoring judgment. Use this table to decide whether to act and what to change.

Dimension scores (visible in the T3 detail report): focus on the lowest-scoring dimension.

Verification workflow

After applying fixes, validate locally without running the full suite:

# T1 only — fast (<2 min per skill), no GPU needed
SKILL_FILTER=<skill-name> SKIP_TIER2=1 RUN_TIER3=0 \
  tools/ci/gitlab/skills/validate-skills-local.sh

# T1 + T2A — catches context dedup regressions
SKILL_FILTER=<skill-name> RUN_TIER3=0 \
  tools/ci/gitlab/skills/validate-skills-local.sh

# Multiple skills at once
SKILL_FILTER=isaac-sim-installation,motion-generation SKIP_TIER2=1 RUN_TIER3=0 \
  tools/ci/gitlab/skills/validate-skills-local.sh

The per-skill T1 report is written to tools/ci/gitlab/skills/report/tier1/<skill>.html. Open it to confirm the specific finding is resolved.

Do not re-run T3 to verify T1 fixes — T3 takes hours and T1 is independent. Re-run T3 only when instructions or examples changed substantively.

Handoff pattern

After T1/T2 fixes are applied and verified:

1. If the fixes revealed a generalizable pattern (e.g., all skills need a ## Security section for design-level notes), invoke skill-distillation to capture that as a library-wide convention.
2. If a T3 neutral verdict led to authoring changes, re-run T3 for that skill alone:

   SKILL_FILTER=<skill-name> SKIP_TIER1=1 SKIP_TIER2=1 FORCE_TIER3_ALL=1 \
     tools/ci/gitlab/skills/validate-skills-local.sh
   

3. Commit changes to the skill files; the next MR pipeline will update the report.

Inputs

Output Format

For each triaged skill, produce:

1. Finding summary — validator, severity, finding code, affected file and line.
2. Classification — Genuine / False positive / Advisory (non-blocking).
3. Applied fix — what was changed and why, or why no change was made.
4. Verification result — output of the targeted validate-skills-local.sh re-run confirming the finding is resolved.

Limitations

• LLM verify (--llm-verify) is required to reliably downgrade Isaac Sim architecture false positives (python_server, command dispatch). Without NVIDIA_INFERENCE_KEY the fallback keeps them as HIGH and they will block.
• T3 neutral verdicts require authoring judgment — there is no deterministic fix path.
• T3 execution failures (⚠ failed) cannot be diagnosed from the SKILL.md alone; they require the Harbor job log.
• This skill covers public and internal skills equally; _internal/ schema advisories (nesting depth) are expected and inert.

Troubleshooting

What not to fix

• T3 execution failures (⚠ failed): these are infra/config issues, not authoring problems. Check the Harbor job log, not the SKILL.md.
• SCHEMA-MEDIUM: Unexpected nesting depth for internal skills: internal skills live under skills/_internal/ which is a non-standard hierarchy; this is expected and inert.
• Security findings describing Isaac Sim's architecture: the python_server, command dispatch, and remote-control patterns are inherent to the domain. Document, don't remove.
• Low SCRIPT_LINT advisories on scripts that already work: flat scripts and deep nesting are advisory only. Fix only when the script is already being edited for another reason.