integrations-lifecycle

This skill is the single place to learn how Netdata's integrations pipeline works end to end. It documents:

• the generator pipeline rooted in integrations/gen_integrations.py;
• the collector taxonomy pipeline rooted in integrations/gen_taxonomy.py;
• the JSON-Schema contracts every metadata.yaml and taxonomy.yaml is validated against;
• every artifact the pipeline produces (gitignored runtime files AND committed .md documentation);
• the <!--startmeta banner conventions and DO-NOT-EDIT rules;
• the two CI workflows that auto-PR or gate the regenerated docs (generate-integrations.yml, check-markdown.yml);
• the secondary ibm.d generation chain (contexts.yaml -> metadata.yaml);
• the contract by which downstream dashboard code consumes integrations.js and, when opted in, integrations/taxonomy.json;
• the collector-consistency rule (taxonomy.yaml moves with metadata and docs) and what is and is NOT enforced by tooling;
• every surprising/dead/edge-case behavior an assistant or maintainer is likely to hit.

After reading SKILL.md plus the per-domain guides linked below, an assistant should never need to ask "how does metadata.yaml work?", "are these integrations/*.md files generated?", "what fields does the schema support?", "what runs in CI?", "where does the in-app integrations page get its data?".

Key concepts (read first)

1. metadata.yaml is the normalized pipeline input. Every per-integration page on every surface (Learn site, in-app dashboard, the umbrella COLLECTORS.md / SECRETS.md / SERVICE-DISCOVERY.md pages) is rendered from metadata.yaml by the pipeline. Most metadata files are authoritative hand-authored input and are edited directly. ibm.d metadata is generated from contexts.yaml, config.go, and module.yaml; NPM catalog metadata is generated by integrations/gen_npm_catalog.py from its profiles, catalogue, and generator-owned copy. Change the authoritative source for the file kind, regenerate producer-owned metadata, and then run the integration pipeline locally to validate the normalized input. Do not include producer-generated metadata or generated documentation in a source PR; the post-merge workflow owns those committed artifacts as described in item 3.

   src/collectors/COLLECTORS.md is the source page for Learn's "Monitor anything with Netdata" page. It is generated from integrations/integrations.js by integrations/gen_doc_collector_page.py; never hand-edit its integration tables. To update that page, change the source metadata/categories or the generator, then run gen_integrations.py and gen_doc_collector_page.py.

   Treat short descriptions as public product copy. Catalog descriptions must say what the integration is and what it monitors, enriches, exports, authenticates, or discovers. For collector-like metadata, the first sentence of overview.data_collection.metrics_description is the catalog sentence used by generated pages such as COLLECTORS.md. Start that sentence with a user-facing action phrase such as Monitor..., Collect..., Enrich network flows with..., or Annotate network flows with.... Do not use the catalog description for variables, defaults, option names, setup instructions, limits, or troubleshooting. Put those details in setup, default-behavior, examples, or troubleshooting fields. See description-authoring.md.

2. integrations/*.md files are GENERATED. DO NOT EDIT. Every per-integration .md opens with a <!--startmeta ... endmeta--> block that ends with message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S/EXPORTER'S/...'S metadata.yaml FILE". See artifacts-and-banners.md for the full banner spec. Edit ordinary hand-authored metadata.yaml, or the authoritative IBM/NPM producer source when metadata itself is generated; never hand-edit the generated page.

3. The CI workflow auto-opens a "Regenerate integrations docs" PR. After a metadata.yaml change merges to master, .github/workflows/generate-integrations.yml first regenerates NPM and ibm.d metadata from their authoritative inputs, then regenerates every per-integration .md and the umbrella pages, and opens a PR for a maintainer to merge. Source PRs contain the authoritative inputs, generators, contracts, and tests; the generated-artifact PR contains the derived files. The pull-request check-markdown.yml workflow regenerates pages for link validation but does not fail merely because that regeneration changes the checkout. Earlier guidance incorrectly described a clean generated diff as a pull-request gate.

4. The collector consistency rule. Anything that touches a collector's runtime behavior MUST land in one PR with matching changes to:

   • metadata.yaml (the integration page driver),
   • taxonomy.yaml (dashboard TOC placement for chart contexts),
   • config_schema.json (the dashboard's DYNCFG editor),
   • the stock .conf (what /etc/netdata/... ships),
   • health.d/*.conf (the alert definitions),
   • the authoritative documentation source. Generated integration pages and umbrella pages are validated locally but committed by the post-merge generated-artifact PR. See consistency.md for what is and is NOT automatically enforced.

5. ibm.d is different. ibm.d generates runtime files (zz_generated_contexts.go, config_schema.json) and documentation files (metadata.yaml, README.md) from contexts.yaml, config.go, and module.yaml. Never hand-edit them. Runtime outputs stay with runtime source changes; documentation outputs are validated locally and committed by the post-merge generated-artifact PR. See ibm-d.md.

6. The dashboard consumes generated integration artifacts. The cloud-frontend at ${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/ runs gen_integrations.py in its own CI to copy integrations.js into its source tree. The historical contract is that .js file's exact shape: export const categories = [...]; export const integrations = [...]. Collector taxonomy is emitted separately as integrations/taxonomy.json by gen_taxonomy.py; downstream consumers opt in to that JSON contract. See in-app-contract.md.

Table of contents

Live how-to rule (mandatory)

If an assistant is asked a concrete question about the integrations pipeline that is NOT already documented under how-tos/ or one of the per-domain guides above, AND answering it requires non-trivial analysis (reading multiple scripts, running the pipeline, cross-referencing schemas), the assistant MUST author a new how-to under how-tos/<slug>.md and add a one-line entry to how-tos/INDEX.md BEFORE completing the task. This rule is durable. Skipping it means the next assistant repeats the analysis from scratch.

Path discipline

This skill follows <repo>/.agents/sensitive-data-discipline.md:

• Files in this repo: repo-relative (integrations/gen_integrations.py, <repo>/integrations/..., src/...).
• Files in sibling Netdata-org repos: ${NETDATA_REPOS_DIR}/<repo-name>/... (env-key from .env).
• No literal home-directory or workstation-root paths anywhere (use the env-keyed placeholder above instead).

Sources of truth referenced by this skill

• <repo>/integrations/ -- generators, schemas, templates, shared metadata files (categories.yaml, deploy.yaml).
• <repo>/integrations/schemas/*.json -- all 12 schemas.
• <repo>/integrations/templates/ -- all Jinja templates.
• <repo>/.github/workflows/generate-integrations.yml and <repo>/.github/workflows/check-markdown.yml -- the CI.
• <repo>/.github/data/distros.yml -- platform table fed into deploy rendering.
• <repo>/src/go/plugin/ibm.d/ -- the secondary generator chain (docgen/main.go, metricgen/main.go).
• <repo>/AGENTS.md -- the "Collector Consistency Requirements" policy text.

Related skills

• project-writing-collectors -- the broader collector authoring context (NIDL contexts, dashboard shaping, plugin landscape). Read FIRST when authoring a brand-new collector; read THIS skill when working with the integration metadata side.
• learn-site-structure -- how the per-integration .md files ultimately get published on learn.netdata.cloud. The Learn-side mapping is driven by <repo>/docs/.map/map.yaml; for integration pages, the relevant <!--startmeta block inside each generated .md is what Learn's ingest reads.