Agent Skills › maester365/maester › maester-test-expert

maester-test-expert

GitHub

用于创建、编辑、验证和文档化 Maester 安全测试,涵盖 Microsoft 365 租户评估、属性配置及调试。

.github/skills/maester-test-expert/SKILL.md maester365/maester

触发场景

创建新的 Maester 安全检查 编辑或重构现有检查 编写或更新检查的 Markdown 文档 调试失败的 Maester 测试

安装

npx skills add maester365/maester --skill maester-test-expert -g -y
更多选项

非标准路径

npx skills add https://github.com/maester365/maester/tree/main/.github/skills/maester-test-expert -g -y

不安装直接使用

npx skills use maester365/maester@maester-test-expert

指定 Agent (Claude Code)

npx skills add maester365/maester --skill maester-test-expert -a claude-code -g -y

安装 repo 全部 skill

npx skills add maester365/maester --all -g -y

预览 repo 内 skill

npx skills add maester365/maester --list

SKILL.md

Frontmatter
{
    "name": "maester-test-expert",
    "description": "Write, validate, and document Maester security checks for Microsoft 365 tenants. Use when asked to create, edit, review, or debug a Maester check (a native test: Test.<ID>.ps1 with a [MaesterTest] attribute plus Test.<ID>.md), its attributes, or a user's custom test, including converting Pester-format tests. Covers the [MaesterTest] attribute, choosing Product, Feature, Area, ResourceType, Severity, UserImpact, ImplementationCost, Mitre and Control (CIS, CISA and NIST SP 800-53 mappings) with the attribute rubric and the taxonomy, Service\/License gates, test parameters, families, Graph API data retrieval, Add-MtTestResultDetail formatting, the MT, CIS, CISA, EIDSCA and ORCA suites, remediation guidance, Entra ID, Exchange, SharePoint, Teams, Defender, Conditional Access, and the validation checklist for new checks."
}

Maester Test Expert

Create, edit, validate, and document Maester security checks -- the unit of work that assesses a Microsoft 365 tenant's security posture.

From Maester 3.0 every check is a native test: two files, Test.<ID>.ps1 and Test.<ID>.md. There are no *.Tests.ps1 wrappers, no separate helper function in powershell/public/, no maester-config.json row and no FunctionsToExport entry. Maester's own engine discovers the test from its attribute, checks that the services and licences it declares are available, runs it, catches errors and writes the result row.

The user-facing guide with every rule is website/docs/writing-tests/index.mdx. This skill is the contributor's condensed version; if they disagree, the code and that page win. How to choose the value of each classification, priority and ATT&CK attribute is in website/docs/writing-tests/attribute-rubric.md (the rubric); this skill links to it and does not repeat it.

Agent Selection

Use maester-test-expert as the primary agent for all normal workflows.

Specialized planner and issue-manager agents are optional workflow components and should generally stay in the background to avoid user confusion.

When to Use This Skill

  • Creating a new Maester security check
  • Editing or refactoring an existing check
  • Adding or correcting a check's attributes: product, feature, area, resource type, severity, user impact, implementation cost, ATT&CK techniques, service or licence declarations
  • Writing or updating a check's .md (description, remediation, result template)
  • Reviewing a check for correctness, attributes that fit the rubric, or documentation completeness
  • Debugging a failing, erroring or skipped check
  • Writing or converting a user's custom test (custom/ folder)

Do NOT use this skill for:

  • Editing EIDSCA tests by hand -- tests/eidsca/Test.EIDSCA.* are generated by build/eidsca/Update-EidscaTests.ps1 from templates in build/eidsca/templates/. Change the generator or its data.
  • Editing ORCA tests by hand -- tests/orca/Test.ORCA.* (header "Generated by ./build/orca/Update-OrcaTests.ps1") and powershell/internal/generated/orca/check-ORCA*.ps1 are generated. Change the generator.
  • Editing website/docs/tests/ -- test pages are generated from the test files by website/scripts/generate-test-docs.mjs.
  • Unit tests of the module itself (powershell/tests/).

Prerequisites

  • PowerShell 7.4 or later.
  • A source checkout: Import-Module ./powershell/Maester.psd1 loads the module and defines every built-in test from ./tests. No build and no .NET SDK are needed unless you change src/Maester.Engine.
  • A working Connect-Maester session to run a check against a tenant.
  • Review the Maester contribution guidelines before submitting changes.

Anatomy of a Check

File Location Purpose
Test.<ID>.ps1 tests/<suite>/[<service>/] One function with a [MaesterTest(...)] attribute. Reads the tenant, decides, calls Add-MtTestResultDetail, returns $true/$false.
Test.<ID>.md Same folder, same base name Description, #### Remediation action, optional #### Related links, then <!--- Results ---> and %TestResult%. Shown in the report.
suite.json Each suite folder (already exists) The suite's ID, name, Source and help URL template. Do not edit for a single check.
frameworks/<key>.json Beside the suite.json of the CIS and CISA suites The catalogue of the benchmark: its name and version, and one entry per requirement with its level, profiles or criticality. It does not list tests. A CIS or CISA check names its requirement in Control; add the requirement here in the same pull request if it is new, or the build fails.

Suite folders: tests/maester/<service>/ (MT.xxxx; services such as entra, exchange, teams, defender, azure, intune, and xspm for exposure management, which has its own suite.json), tests/cisa/<service>/ (CISA.MS.*), tests/cis/ (CIS.M365.*), tests/ad/<area>/ (AD-*), and the generated tests/eidsca/ and tests/orca/. Every folder name is lowercase kebab-case; see the repository layout in the contributing guide.

The function keeps a descriptive name (Test-MtCaMfaForRiskySignIn). Function names must be unique across powershell/internal, powershell/public and every test file, because built-in tests share the module scope. Code shared by several checks goes in powershell/internal/checks/<suite>/<service>/ as an internal helper; reusable data access for a service goes in powershell/internal/services/<service>/.

The Test File

Template

Scaffold it, then edit:

New-MtTest -Id MT.1234 -Title 'At least one Conditional Access policy blocks device code flow.' `
    -Product entra -Feature conditional-access -Area authentication -ResourceType Tenant `
    -Severity High -UserImpact Low -ImplementationCost Low -Mitre T1078.004, T1528, T1566 `
    -Service Graph -Path ./tests/maester/entra

Only -Id and -Title are required, so you can scaffold first and set the other values once the .md is written (see Choosing the attribute values). New-MtTest does not write Author; add it.

function Test-MtCaBlockDeviceCodeFlow {
    <#
    .SYNOPSIS
    Checks if at least one enabled Conditional Access policy blocks the device code flow.

    .DESCRIPTION
    {Detailed description of the control and what the check evaluates.}
    #>
    [MaesterTest(
        Id                 = 'MT.1234',
        Title              = 'At least one Conditional Access policy blocks device code flow.',
        Product            = 'entra',
        Feature            = 'conditional-access',
        Area               = 'authentication',
        ResourceType       = 'Tenant',
        Severity           = 'High',
        UserImpact         = 'Low',
        ImplementationCost = 'Low',
        Mitre              = ('T1078.004', 'T1528', 'T1566'),
        Control            = ('cisa-scuba:MS.AAD.3.1', 'nist-800-53-r5:IA-2'),
        Service            = 'Graph',
        License            = 'AAD_PREMIUM',
        Author             = 'your-github-handle'
    )]
    [CmdletBinding()]
    [OutputType([bool])]
    param()

    $policies = Get-MtConditionalAccessPolicy | Where-Object { $_.state -eq 'enabled' }
    $blocking = @($policies | Where-Object {
            $_.conditions.authenticationFlows.transferMethods -match 'deviceCodeFlow' -and
            $_.grantControls.builtInControls -contains 'block'
        })

    if ($blocking.Count -gt 0) {
        $result = "Well done. These policies block the device code flow:`n`n%TestResult%"
        Add-MtTestResultDetail -Result $result -GraphObjects $blocking -GraphObjectType ConditionalAccess
        return $true
    }

    Add-MtTestResultDetail -Result 'No enabled Conditional Access policy blocks the device code flow.'
    return $false
}

Note what is not there: no Test-MtConnection guard, no Get-MtLicenseInformation check, no outer try/catch, no -Description, no -Severity, no -TestTitle, and no Tag.

File rules (enforced by Read-MtNativeTest; violations are InvalidMetadata)

  • File name is exactly Test.<Id>.ps1; the .md sits beside it.
  • Exactly one function with [MaesterTest(...)]. Other functions (private helpers, a family's instance source) may live in the same file.
  • Top level contains only function definitions: no statements, no script param() block.
  • The test function has [CmdletBinding()].

The [MaesterTest] attribute

Arguments are named and constant: quoted strings, $true/$false, a parenthesised list ('a', 'b'), a single string for a list property, or a bare flag (LongRunning). No variables, expressions, @() or hashtables. The schema lives in powershell/assets/MaesterTestSchema.psd1. How to choose the values from Product to Mitre is in Choosing the attribute values.

Property Type Values Required for built-ins Notes
Id string ^(?=.*[0-9.\-])[A-Za-z][A-Za-z0-9]*([.\-][A-Za-z0-9_]+)*$, max 64 Yes Never change an existing ID.
Title string One line Yes The only title.
Product string A product key from the taxonomy: entra, exchange, defender, ... Yes Write the key, not an alias such as Entra ID. Groups the console progress and the run summary.
RelatedProduct string[] Product keys No Never the same key as Product. -Product matches it too.
Feature string A feature key of the test's product: conditional-access, ... Yes
Area string An area key: authentication, privileged-access, ... Yes
ResourceType string A resource type key: Tenant, User, ConditionalAccessPolicy, ... Yes
Severity string Critical, High, Medium, Low, Info Yes A TestSettings row in a user's config overrides it.
UserImpact string Low, Medium, High Yes, unless Severity is Info Leave it out of an Info check.
ImplementationCost string Low, Medium, High Yes, unless Severity is Info Leave it out of an Info check.
Mitre string[] ATT&CK technique IDs (T1110, T1110.003), or the single value None Yes At most five. Never a technique beside one of its own sub-techniques.
Control string[] <framework-key>:<requirement-id> CIS and CISA checks The requirements the check implements or supports. A CIS check sets Control = 'cis-m365:5.1.5.1', a CISA check Control = 'cisa-scuba:MS.AAD.3.1'; any other check may name the same requirements. The first one that a framework file knows becomes the Benchmark of the result. For a shipped framework the requirement must exist in its file.
Service string[] Graph, Azure, ExchangeOnline, SecurityCompliance, SharePointOnline, Teams, Dataverse, AzureDevOps, GitHub, ActiveDirectory, or None Yes All listed must be connected. Registry: powershell/assets/MaesterServiceRegistry.psd1.
License string[] Tokens from powershell/assets/MaesterLicenseTable.psd1; A&B = both No Tenant needs any one element.
Permission string[] One permission per element, no commas No The permissions the check needs, for example Policy.Read.All. Recorded; not enforced in 3.0.
TenantType string[] Workforce, External No Default Workforce. Recorded; enforced only when a run turns it on.
Cloud string[] Commercial, GCC, GCCHigh, DoD, China, Bleu, Delos, GovSG No Default all. Recorded; enforced only when a run turns it on.
Platform string[] Windows, Linux, MacOS No Declare Windows for Windows-only commands (lint checks common ones).
Lifecycle string Preview, Stable, Deprecated No Default Stable. Use Preview for a check that depends on preview APIs or is still being validated; it is not run by default. A Deprecated check still runs.
ReplacedBy string The ID of an existing test No Only with Lifecycle = 'Deprecated'.
LongRunning bool No Slow in large tenants. Not run by default.
InstanceSource string Function in the same file No Makes the check a family (below).
Exclusive bool No Must not run concurrently (for example it writes to the tenant).
Tag string[] No commas No Built-in checks do not use it. See Tags.
Author string[] GitHub handles Yes The initial author(s).
Contributor string[] GitHub handles No Add yourself when you substantially change a check.
HelpUrl string https://... No Defaults to https://maester.dev/docs/tests/<Id>.

Category and Preview were removed in 3.0. A test that still has one is InvalidMetadata, with the replacement in the message: Product and Feature instead of Category, Lifecycle = 'Preview' instead of Preview. Reserved names (rejected today): OptionalService, TimeoutSeconds.

Licence tokens: AAD_PREMIUM (Entra ID P1 or better), AAD_PREMIUM_P2, Entra_Identity_Governance, AAD_WRKLDID_P1, AAD_WRKLDID_P2, ATP_ENTERPRISE (MDO P1 or P2), THREAT_INTELLIGENCE (MDO P2), EOP_ENTERPRISE, EXCHANGE_S_STANDARD, EXCHANGE_DLP, M365_ADVANCED_AUDITING, DEFENDER_XDR, LOCKBOX_ENTERPRISE, INTUNE_A. Built-in checks use only these; add a token to the table if you need a new one.

Return values and outcomes

The check does Result ReasonCode
return $true / return $false Passed / Failed
Add-MtTestResultDetail -Investigate Investigate (whatever it returns)
Add-MtTestResultDetail -SkippedBecause <code> Skipped; the call ends the test TestSkipped, or NotApplicable
throws, or -SkippedBecause Error Error TestError
returns $null without skipping Skipped NoResult
returns a non-boolean or several values Error InvalidReturn
  • Return exactly one boolean. Uncaptured output ($list.Add($x), a stray expression) becomes a second return value and an InvalidReturn error; send it to $null or Out-Null.
  • The engine owns the try: any terminating or statement-terminating error ends the check as Error with the message. A try/catch is fine for a specific condition (for example a 403 reported as -SkippedBecause NotAuthorized), not as a blanket wrapper. A skip raised inside your own try still skips.
  • Use -SkippedBecause NotApplicable when the tenant state makes the check meaningless (for example no policies of that kind exist). Never return $null to mean "not applicable".
  • Never use NotConnected* or NotLicensed* skip codes in a built-in check; declare Service / License.
  • Checks that need delegated permissions skip with -SkippedBecause NotSupportedAppPermission when (Get-MgContext).AuthType -ne 'Delegated'.

Parameters

Tunable values are function parameters, never hard-coded and never read from wrapper arguments.

param(
    # Longest validity period, in days, that a certificate may be issued for.
    [ValidateRange(1, 3650)]
    [int] $MaximumValidityDays = 365,

    # Members of these groups are exempt from this check.
    [MaesterParameter(Kind = 'Entra.Group')]
    [string[]] $ExcludedGroups
)
  • Types: int, bool, switch, string, string[]. Constant defaults. Nothing mandatory: the check must run with no config.
  • Every parameter has a one-line # comment directly above it (only the nearest comment line is read), or .PARAMETER help. The unit tests require a description.
  • Thresholds get ValidateRange; choices get ValidateSet.
  • [MaesterParameter(Kind = ...)] for IDs: Entra.User, Entra.Group, Entra.ServicePrincipal, Entra.Application, Entra.ConditionalAccessPolicy, Entra.NamedLocation, Entra.Domain, Entra.DirectoryRole, Azure.Subscription, Azure.ResourceGroup, Azure.Resource (powershell/assets/MaesterParameterKinds.psd1).
  • Instance and names starting with Mt are engine-owned.
  • Users set values in maester-config.json under TestSettings[].Parameters, or with Invoke-MtTest -Parameter.

Families (one check, several results)

When a check produces one result per item, declare InstanceSource = '<function in the same file>'. The engine calls that function at run time (never at discovery) and runs the check once per returned object, passed as $Instance (declare param($Instance) with a description comment).

Each instance object has Id (required, unique, ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$; the row ID becomes <Id>.<suffix>) and optional Title, Severity, Tag, Data. See tests/maester/drift/Test.MT.1060.ps1 and tests/maester/entra/Test.MT.1024.ps1.

Settings

Read tenant-wide settings with Get-MtSetting -Name <name> (for example EmergencyAccessAccounts). New settings Maester itself reads go in powershell/assets/MaesterSettings.psd1 with a type, default and description.

Data Retrieval

Always prefer Invoke-MtGraphRequest over Invoke-MgGraphRequest or Get-Mg* cmdlets. It provides:

  • Built-in caching -- reduces API calls when multiple checks query the same endpoint.
  • Automatic pagination -- no need for -All.
  • ConsistencyLevel header -- included by default for read-only calls.
  • Batching -- pass an array of IDs via -UniqueId.
  • Named parameters -- Select, Filter, QueryParameters.
$policies = Invoke-MtGraphRequest -RelativeUri 'policies/authorizationPolicy' -ApiVersion v1.0
$users = Invoke-MtGraphRequest -RelativeUri 'users' -Filter "userType eq 'Member'" -Select id, displayName
$users = Invoke-MtGraphRequest -RelativeUri 'users' -UniqueId $userIds -Select id, displayName, onPremisesSyncEnabled

$policySplat = @{
    ApiVersion      = 'beta'
    RelativeUri     = 'policies/roleManagementPolicyAssignments'
    Filter          = "scopeId eq '/' and scopeType eq 'DirectoryRole' and roleDefinitionId eq '$($role.id)'"
    QueryParameters = @{ expand = 'policy(expand=rules)' }
}
$policy = Invoke-MtGraphRequest @policySplat

Other helpers: Get-MtConditionalAccessPolicy, Get-MtUser, Get-MtRole, Get-MtRoleMember, Get-MtGroupMember, Get-MtExo, Get-MtSpo, Invoke-MtAzureRequest, Invoke-MtAzureResourceGraphRequest. Caches are reached only through these helpers; never assign $script:, $global: or $__MtSession state in a check (lint).

New Graph permission scopes go in Get-MtGraphScope.

Result Formatting

# Result text; the .md supplies the description
Add-MtTestResultDetail -Result $resultMarkdown

# Objects rendered as a table with admin portal deep links; -Result must contain %TestResult%
Add-MtTestResultDetail -Result "Found issues:`n`n%TestResult%" -GraphObjects $failing -GraphObjectType ConditionalAccess

# Needs a human to review
Add-MtTestResultDetail -Result $result -Investigate

# Custom skip
Add-MtTestResultDetail -SkippedBecause Custom -SkippedCustomReason 'All alerts have been suppressed.'

-GraphObjectType: Users, Groups, Devices, ConditionalAccess, AuthenticationMethod, AuthorizationPolicy, ConsentPolicy, Domains, IdentityProtection, UserRole.

Output messages: a pass message starts with "Well done." and describes the positive outcome; a fail message says what is wrong and what to change; tables have header and separator rows; link to the admin portal where possible.

The Markdown File (Test.<ID>.md)

No front matter. Template:

{One-line summary of what the check verifies.}

{Rationale: explain WHY this configuration matters for security.}

#### Remediation action

1. In **Entra ID**, navigate to **{Section}** > **[{Page Name}]({direct admin portal URL})**.
2. {Step-by-step instructions to fix the issue.}
3. Click **Save**.

#### Related links

* [{Admin portal link description}]({URL})
* [{Benchmark or baseline reference}]({URL})

<!--- Results --->
%TestResult%
  • Exactly one <!--- Results ---> followed by %TestResult%; everything above it is the description.
  • Do not pass -Description to Add-MtTestResultDetail, so the .md is used.
  • The .md is also shown for rows that did not run (not selected, service not connected), so it must make sense on its own.
  • The website page website/docs/tests/<suite>/<ID>.md is generated from this file and the attribute.

Choosing the Attribute Values

A built-in check says what it is about, how urgent a failure is and which attack techniques it addresses. Each of those values is chosen with the attribute rubric. Read the rubric's section for an attribute before you set, change or review its value, and do not take the values of a neighbouring check without checking them against it.

Attribute The question it answers
Product In which product is the fix made?
RelatedProduct Under which other products would an admin expect to find the check? Optional.
Feature Which part of the product, a settings page or an object type, is the check about?
Area Which security theme is it, whatever the product?
ResourceType What does a failing result point at: the tenant, or a kind of object it lists?
Severity How much risk does the organization carry while the check fails?
UserImpact How much does the recommended fix disrupt end users?
ImplementationCost How much administrator work does the recommended fix take?
Mitre Which MITRE ATT&CK techniques does the checked control prevent or detect?
Control Which requirements of CIS, CISA SCuBA or NIST SP 800-53 does a passing result support?

The rubric rates a check by the fix its own .md recommends, so write the .md before you choose the values. The values of the generated EIDSCA and ORCA checks are in build/eidsca/eidsca-test-metadata.json and build/orca/orca-test-metadata.json: change a value there and regenerate.

Framework requirements and NIST controls (Control)

Fill in Control on every new check. Do not leave it out because it is optional: decide, and leave it empty only when nothing applies. It is the only way a check is mapped to a framework. Work through these in order:

  1. The requirement the check is written for. A CIS check lists cis-m365:<id> or cis-github:<id> and a CISA check lists cisa-scuba:<id>, first. If the requirement is not in the suite's framework file yet, add it there with its facts (level and profiles, or criticality) in the same pull request.
  2. A CIS or CISA requirement that asks for the same setting. A Maester check that verifies what cisa-scuba:MS.AAD.3.1 asks for lists it. Look the requirement up in tests/cis/frameworks/*.json and tests/cisa/frameworks/cisa-scuba.json, or in the lookup of the test builder. Get-MtTest -Suite CISA | Format-Table Id, Title, Control shows what the existing checks map to.
  3. NIST SP 800-53 Rev. 5 controls, written nist-800-53-r5:<control>, chosen with the NIST part of the rubric's Control section from the pinned list powershell/assets/MaesterNistControls.json. At most three, best fit first, and only when a passing result is direct evidence for the control. Skip this step when step 1 or 2 gave a cisa-scuba reference: CISA publishes the NIST mapping of its policies and the result gets it from the framework file.

The first value that names a requirement in a framework file becomes the Benchmark of the result, so the order matters. An inventory count, or a health, licensing or tidiness check, usually lists nothing.

Where the keys come from

  • Product, Feature, Area and ResourceType take keys from powershell/assets/MaesterTaxonomy.psd1: the products with their features, the areas and the resource types. A feature key belongs to one product.
  • Product follows the folder: a check in tests/maester/entra or tests/cisa/entra is entra. The folder map is in powershell/tests/general/NativeTestLint.Tests.ps1; a new service folder needs an entry there.
  • Mitre takes technique IDs from the pinned ATT&CK release, powershell/assets/MaesterAttackTechniques.json. An ID that the release retired or does not know fails the unit tests. Tactics are looked up from the ID and never typed.
  • Control takes requirement IDs from the framework files (tests/<suite>/frameworks/<key>.json) and NIST control IDs from powershell/assets/MaesterNistControls.json. The unit tests fail on an ID that is in neither.
  • Get-MtTest -Product entra -Feature conditional-access | Format-Table Id, Title, Severity, Area, ResourceType lists the existing checks of a feature, to compare your values with.
  • A new key is added to the taxonomy in the same pull request as the first test that uses it. Prefer an existing key. Do not add a key ahead of its test: the unit tests fail on a product, feature, area or resource type that no test uses. A new area also needs a row in the rubric's area table, which the unit tests compare with the taxonomy.

Tags

Tag holds free labels for the owner of a test folder. It is not how Maester classifies a check.

  • Built-in checks carry no tag for product, feature, area, suite, benchmark, licence, severity or their own ID. Each of those has an attribute or is derived, and the unit tests fail a built-in check that tags one. Remove such a tag when you edit an older check.
  • Do not put Preview or LongRunning in Tag; use Lifecycle = 'Preview' and the LongRunning flag.
  • Users select checks with -Product, -Feature, -Area, -Severity and -Suite on Invoke-Maester and Get-MtTest. The tags of Maester 2.x, such as CA or CIS, still select the checks that carried them, with a warning, through powershell/assets/Maester.LegacyTags.json. That file is generated and never edited, and a check added after 3.0 is not in it, so do not add a tag to make a new check selectable.
  • A user's custom test may tag freely.

How the values are checked

The unit tests in powershell/tests/general/NativeTestAttributes.Tests.ps1 check what is mechanical. The attribute-review check on the pull request has a reviewer model judge each new or changed value against the rubric. The person or agent who reviews the pull request applies the same rubric. See How the rubric is enforced.

When the attribute review fails, change the value, or explain in the pull request why the declared value fits the rubric. When you review a check yourself, follow "Reviewing a test's attributes" in AGENTS.md; it also shows how to run the attribute review before you push.

Step-by-Step Workflow: Creating a New Check

  1. Reserve an ID. Check issue #697 and comment to reserve the next MT.xxxx (the maester-issue-manager agent can do this). Get-MtTest MT.12* shows which IDs exist.
  2. Scaffold with New-MtTest -Id <ID> -Title '...' -Service <service> -Product <product key> -Path ./tests/<suite>/<service>.
  3. Fill in the attribute: License, Author, flags as needed.
  4. Write the function: read with Invoke-MtGraphRequest or the helpers, decide, Add-MtTestResultDetail, return a boolean. Add parameters for tunable values.
  5. Write the guide, the .md, in full: a one-line summary, why the setting matters, numbered remediation steps with a direct link to the admin portal page, related links (Microsoft Learn, and the benchmark when the check maps to one), and the results footer. Replace every placeholder that New-MtTest wrote; a check with scaffold text in its guide is not finished. Verify product facts and links with Microsoft Learn.
  6. Choose the attribute values with the rubric: Feature, Area, ResourceType, Severity, UserImpact, ImplementationCost, Mitre, Control, and RelatedProduct when one applies. See Choosing the attribute values and Framework requirements and NIST controls. The test builder shows the allowed values and the rubric for each attribute and writes the attribute; use it to check a value you are unsure of.
  7. Validate offline: Import-Module ./powershell/Maester.psd1 -Force; Get-MtTest <ID> must show IsValid = True and no Errors. Name the ID and leave out -Path: Get-MtTest -Path <file> applies the rules of a custom test, so it does not report a missing attribute or a key that is not in the taxonomy.
  8. Run against a tenant: Connect-Maester; Invoke-MtTest -Path <file> (or Invoke-MtTest -Id <ID>). Try -Parameter @{...} for each parameter. Check the row's Result, ReasonCode, ResultDetail.TestResult.
  9. Run the unit tests: ./powershell/tests/pester.ps1. powershell/tests/general/NativeTestLint.Tests.ps1 and powershell/tests/general/NativeTestAttributes.Tests.ps1 check every built-in test.

Converting a User's Pester-format Test

Users' 2.x custom tests (*.Tests.ps1) still run when Pester 5.7.1+ is installed. To convert:

Convert-MtTest -Path ./custom -WhatIf
Convert-MtTest -Path ./custom | Format-Table Id, Status, Notes

It writes Test.<ID>.ps1 + .md per It, turns a Test-MtConnection guard into Service, removes a blanket -SkippedBecause Error catch, and rewrites a trailing | Should -Be $true to return. The Describe name becomes Feature, so the row's Block stays what it was. The Pester tags are kept in Tag, except that a Preview tag becomes Lifecycle = 'Preview' and a LongRunning tag the LongRunning flag. Rows marked NeedsReview have TODO comments; tests with names built at run time are Skipped and must be rewritten as families. Once converted, delete the Pester file: the native test runs and the Pester test with the same ID no longer does.

A user's custom native test lives in their custom/ folder, uses their own ID prefix (not MT., CISA., CIS., EIDSCA., ORCA., AD-, AZDO., MT1060., or MP., which is reserved for Maester Packs), needs a .md, and needs only Id and Title in its attribute. It may set Product, Feature, Area and ResourceType to values of its own (a value that matches the taxonomy is stored as the key), may tag freely and may use Control; the rubric applies to it only if its owner wants it to. It can call only Maester's exported commands; to reuse a built-in check it calls Invoke-MtTest -Id <ID>.

Validation Checklist

Test logic

  • Returns exactly one $true/$false; no stray pipeline output.
  • No Test-MtConnection guard, no Get-MtLicenseInformation check; Service and License declared instead.
  • No blanket try { } catch { Add-MtTestResultDetail -SkippedBecause Error ... }.
  • -SkippedBecause NotApplicable (not $null) when the check does not apply.
  • Data retrieval uses Invoke-MtGraphRequest or Maester helpers.
  • No hard-coded GUIDs, tenant IDs or names; tunables are described parameters.
  • No writes to $script:, $global:, $env: or $__MtSession.
  • Platform = 'Windows' if it uses Windows-only commands.

Attribute and files

  • Test.<Id>.ps1 and Test.<Id>.md in the right suite folder; ID unique and not reused.
  • Id, Title, Product, Feature, Area, ResourceType, Severity, Mitre, Service, Author set; values constant.
  • UserImpact and ImplementationCost set, or both left out when Severity is Info.
  • Product, Feature, Area and ResourceType are keys from powershell/assets/MaesterTaxonomy.psd1; a new key is added there in this pull request.
  • Each of those values, and RelatedProduct if set, checked against its section of the attribute rubric, with the reason visible in the description or the remediation text.
  • No Category or Preview; no Tag for product, feature, area, suite, benchmark, licence, severity or the ID. Lifecycle = 'Preview' and the LongRunning flag where they apply.
  • A CIS or CISA check sets Control to its requirement (cis-m365:<id>, cis-github:<id> or cisa-scuba:<id>), first, and the requirement is in the suite's framework file.
  • Control was considered for every check: a CIS or CISA requirement that asks for the same setting, and NIST SP 800-53 controls from powershell/assets/MaesterNistControls.json (none when a cisa-scuba reference already brings them). Left empty only when nothing applies.
  • [CmdletBinding()] and [OutputType([bool])]; comment-based help inside the function.
  • Get-MtTest <ID> (without -Path) reports no errors.

Documentation

  • .md has summary, rationale, #### Remediation action with portal links, and ends with <!--- Results ---> + %TestResult%.
  • No scaffold placeholder is left in the .md ("Describe what this test checks", "Explain how to fix"), and its links open the page they name.
  • Pass message starts with "Well done."; fail message is actionable.

Code style

  • One True Brace Style; Pascal Case; full cmdlet names (no aliases); 4-space indentation.
  • UTF-8 with BOM for .ps1 files.

Pre-pull request

  • ./powershell/tests/pester.ps1 passes (PSScriptAnalyzer included).
  • Ran against a tenant with Invoke-MtTest.
  • Commit messages follow Conventional Commits.

Common Mistakes

Mistake Why it matters Fix
Adding a Test-MtConnection guard or licence check Duplicates the engine's gate; lint fails Declare Service / License
Wrapping the body in try/catch that reports -SkippedBecause Error Lint fails; the engine already does it Remove it; catch only specific conditions
Returning $null for "not applicable" Reported as Skipped/NoResult Add-MtTestResultDetail -SkippedBecause NotApplicable
Uncaptured output ($list.Add(...)) Error/InvalidReturn $null = $list.Add(...)
Variables or @() in the attribute InvalidMetadata Constants and ('a', 'b') lists only
Category or Preview in the attribute InvalidMetadata; both were removed in 3.0 Product and Feature; Lifecycle = 'Preview'
A Tag such as 'CA', 'Entra' or 'Maester' on a built-in check Unit tests fail; tags are not the taxonomy Use the attribute that says it; see Tags
A product alias ('Entra ID') or a key that is not in the taxonomy Unit tests fail Use the key from powershell/assets/MaesterTaxonomy.psd1, or add the key in the same pull request
Copying Severity, Area or Mitre from a similar check The value may not fit this check, and the attribute review fails a value that does not fit Choose each value with the rubric
Validating a built-in check with Get-MtTest -Path only It applies the rules of a custom test and reports no missing attribute Get-MtTest <ID>
File name does not match Id InvalidMetadata Rename to Test.<Id>.ps1
Missing [CmdletBinding()] InvalidMetadata Add it before param()
Hard-coded thresholds or object IDs Users cannot tune the check Make them described parameters
Multi-line # comment above a parameter Only the last line is read as its description One-line comment or .PARAMETER
Invoke-MgGraphRequest / Get-Mg* Bypasses caching Invoke-MtGraphRequest
Using -Severity or -TestTitle on Add-MtTestResultDetail Ignored for single-result checks Set them in the attribute
Editing generated EIDSCA/ORCA files or website/docs/tests/ Overwritten Change the generator or the test files
Adding the function to FunctionsToExport Check functions are internal in 3.0 Don't
Missing %TestResult% in -Result with -GraphObjects The object table is not inserted Include the placeholder

Reference Examples

  • tests/maester/entra/Test.MT.1012.ps1 + .md -- simple Graph check with a licence gate.
  • tests/maester/entra/Test.MT.1198.ps1 -- check with a ValidateRange parameter, LongRunning.
  • tests/maester/drift/Test.MT.1060.ps1 -- family with InstanceSource and $Instance, Service = 'None'.
  • tests/cisa/entra/Test.CISA.MS.AAD.3.5.ps1 -- CISA check.
  • tests/cisa/exchange/Test.CISA.MS.EXO.14.1.ps1 -- check with a RelatedProduct.
  • tests/maester/azure-devops/Test.AZDO.1045.ps1 -- Info check with Mitre = 'None' and Lifecycle = 'Preview'.
  • tests/eidsca/Test.EIDSCA.AF01.ps1 -- generated thin check comparing a tenant value.
  • tests/orca/Test.ORCA.100.ps1 -- generated ORCA check.
  • website/docs/writing-tests/index.mdx -- the full native test guide.
  • website/docs/writing-tests/attribute-rubric.md -- how to choose each attribute value.
  • powershell/assets/MaesterTaxonomy.psd1 -- the product, feature, area and resource type keys.
  • website/docs/configuration/applicability.md -- gate order and reason codes.

Troubleshooting

Symptom Likely cause Resolution
Error / InvalidMetadata Attribute or file rule broken Get-MtTest <ID> lists each problem with its line (Get-MtTest -Path <file> for a custom test)
Attribute review fails on the pull request A value does not fit the rubric Change it to the expected value, or explain in the pull request why the declared value fits
Error / InvalidReturn Non-boolean or multiple outputs Capture stray output; return one boolean
Error / InvalidConfiguration Bad parameter value in config or -Parameter Check Get-MtTest <ID> parameters; fix the value
Error / TestError Exception in the body (often a permission) Read ReasonDetail / ErrorRecord
Skipped / ServiceNotConnected Declared service not connected Connect-Maester -Service <service>
Skipped / LicenseNotFound Tenant lacks every License element Expected; check the token
Skipped / NoResult Returned $null Return a boolean or skip explicitly
NotRun / LongRunning or Preview Excluded by default -IncludeLongRunning / -IncludePreview, or Invoke-MtTest -Id
NotRun / OptInServiceNotConnected Active Directory check without AD connection Connect-Maester -Service ActiveDirectory
Report shows no description .md missing, misnamed, or -Description passed Name it Test.<ID>.md; drop -Description
%TestResult% appears literally Placeholder missing from -Result with -GraphObjects Add it
403 from Graph Missing scope Add it to Get-MtGraphScope

版本历史

  • bd9ab9d 当前 2026-10-20 00:56

    新增关于 Control 框架映射、Maester Packs 保留前缀警告及实施裁决的详细文档说明。

  • 6bf1d98 2026-08-20 14:03

同 Skill 集合

.claude/skills/maester-test-expert/SKILL.md

元信息

文件数
0
版本
bd9ab9d
Hash
9ba91b43
收录时间
2026-08-20 14:03

首页 - Wiki
Copyright © 2011-2026 iteam. Current version is 2.156.0. UTC+08:00, 2026-11-21 16:20
浙ICP备14020137号-1