Claude Code plugin eval: an acceptance workflow for reliable AI systems

Claude Code 2.1.269, dated September 11, 2026, adds `claude plugin eval`, a command that runs a plugin evaluation suite and returns scored, reproducible results through JSON and HTML reports. That is the confirmed capability. Anthropic does not say that the command certifies security, production readiness, business accuracy or regulatory compliance.

The practical value is therefore governance: a team can preserve inspectable test output before deciding whether a plugin should advance. CreatikLab’s operational interpretation is to separate execution, evidence and approval. The command executes the suite; the reports preserve results; an accountable owner decides whether the evidence satisfies the release conditions.

• Confirmed: a plugin evaluation suite can be run from the Claude Code command line.
• Confirmed: results are scored and described as reproducible.
• Confirmed: reports are available as JSON and HTML.
• Not confirmed: a universal passing threshold, mandatory test catalogue or production certification.

To connect this topic with execution, continue with custom digital presence, AI marketing solutions and Creatiklab services.

For the next operational reads, use Parallel Claude Code agents: a controlled workflow for custom web development, Claude Code release readiness: a practical reliability audit for AI development workflows, Claude Code restricted mode: a security audit for business workflows and Claude Code release governance: model switches, cost signals and safer delivery.

The same release adds output-style selection across local, Remote Control, cloud and other headless sessions. It also adds a diff of files changed by a Bash command when the Bash tool performs edits, controlled by the `bashEditDiffEnabled` setting. These capabilities can improve inspection, but Anthropic does not claim that they capture every possible side effect.

Anthropic also documents optional repository attributes for OpenTelemetry metrics and events through `OTEL_METRICS_INCLUDE_REPOSITORY`. Commit events can receive repository-reference attributes. For gateway users, `CLAUDE_CODE_GATEWAY_MODEL_DISCOVERY_TIMEOUT_MS` can extend the model-discovery timeout, whose documented default is three seconds.

Workflow concurrency can be raised with `CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS` within the documented range of one to 256 for inference-bound fan-outs. This is a capacity control, not evidence that higher concurrency improves quality. The release also fixes cache-reuse and background-agent status issues, reinforcing why the exact installed version belongs in every evaluation record.

A useful evaluation starts with the failure the buyer cannot afford, not with a convenient prompt. The following CreatikLab matrix is a methodology, not a Claude Code product promise. It converts business risk into inspectable evidence and names the person who must act.

• Functional risk — Evidence: expected output and allowed variation. Action: compare observed behaviour with an approved fixture. Owner: product owner.
• Tool-use risk — Evidence: tools requested, commands executed and changed-file diff where available. Action: reject unexplained access or mutation. Owner: technical lead.
• Data risk — Evidence: sanitized inputs, output handling and logs. Action: remove unnecessary sensitive fields and verify retention. Owner: data owner.
• Integration risk — Evidence: dependency versions, gateway configuration and failure response. Action: test degraded and unavailable dependencies. Owner: platform engineer.
• Business risk — Evidence: downstream decision affected by the output. Action: require human review when an error could create material harm. Owner: process owner.

The decision rule is simple: do not accept a plugin when a material risk has no named evidence, action and owner. A strong aggregate score cannot compensate for a missing test of the workflow’s most consequential failure.

Reproducibility requires more than saving the final score. Record the Claude Code version, plugin revision, evaluation-suite revision, permitted tools, relevant settings, test inputs, expected outcomes and execution environment. If any of these changes, label the run as a new condition rather than presenting it as a direct comparison.

Run `claude plugin eval` only after the fixtures and acceptance statements have been reviewed. Preserve the JSON report as structured evidence and the HTML report for human inspection. The release note does not specify report schemas, storage duration or access controls, so verify those details directly and apply your organisation’s retention and security requirements.

1. Freeze the plugin and suite revisions under review.
2. Remove or replace secrets and personal data in fixtures.
3. Document permitted tools and expected file mutations.
4. Execute the evaluation in a controlled environment.
5. Store JSON and HTML outputs with version metadata.
6. Record anomalies, reviewer comments and the final disposition.
7. Link remediation work to the failed case before rerunning.

A failed case should be classified before it is fixed. Determine whether the plugin failed, the expectation was ambiguous, the fixture was unrealistic, the environment differed or an external dependency behaved unexpectedly. Editing the test merely to improve the score destroys the audit trail unless the reason is documented and approved.

Passing cases also deserve sampling. A plugin may reach an expected final answer through an unacceptable command, unnecessary data access or an uncontrolled file change. Where Bash performs file edits, the newly documented diff can support review. Treat it as additional evidence rather than assuming it represents every action in the session.

Keep the original report, the diagnosis, the remediation reference and the rerun together. This creates a chain from failure to decision. CreatikLab recommends rejecting silent exceptions: if a reviewer accepts a known deviation, the record should state the business rationale, affected scope, temporary control and accountable owner without implying that Claude Code supplied those governance rules.

The release score is a platform-generated result; the acceptance decision is organisational. Measure both separately. At suite level, preserve the score and report identity. At case level, record expected behaviour, observed behaviour, failure class and reviewer status. At operational level, monitor incidents, interventions and rollback triggers after deployment using definitions agreed by the business.

• Evaluation coverage: which material workflow risks have an explicit case, without turning case count into a quality claim.
• Repeatability: whether equivalent controlled runs produce materially consistent decisions.
• Mutation review: whether expected and unexpected file changes are distinguishable.
• Human intervention: which outputs require correction or escalation before downstream use.
• Business validity: whether accepted outputs satisfy the process owner’s documented criteria.
• Release traceability: whether every approval links to the plugin, suite, environment and reports used.

Do not compare headline scores across changed suites as if they represented identical conditions. Do not infer qualified leads, revenue, safety or reliability from the score unless the evaluation cases explicitly test a defined connection to that outcome and the downstream measurement system is independently validated.

The official release note is narrow. It does not specify bundled evaluation cases, a universal threshold, pricing, plan eligibility, report schema, data retention, deployment scope or a guarantee that results will remain identical across every environment. Those details should not be invented during procurement or implementation.

• Do not assume reproducible means deterministic under every dependency and configuration.
• Do not assume JSON output makes an evaluation automatically auditable.
• Do not assume an HTML report proves that permissions were appropriate.
• Do not raise concurrent-agent limits merely because the documented control permits it.
• Do not treat a Bash changed-file diff as a complete security log.
• Do not expose production secrets merely to make fixtures realistic.
• Do not let the plugin author be the only release approver for a material workflow.

Concurrency deserves particular caution. Anthropic describes the control for inference-bound fan-outs and gives its permitted range, but does not claim that the maximum is suitable for every system. More simultaneous work can change cost, rate-limit, observability and review demands; assess those effects in the actual environment before changing the setting.

A buyer comparing providers should request inspectable deliverables: a risk register tied to plugin behaviour, versioned evaluation fixtures, an execution manifest, retained JSON and HTML reports, failure triage, permission review, observability mapping, acceptance criteria and a rollback procedure. Ask who owns each decision and how exceptions are documented—not merely whether the provider can run the command.

For workflows connected to acquisition or sales, qualified leads should be measured in the downstream system using an agreed business definition, such as sales acceptance or another validated lifecycle status. Plugin scores, task completion and generated output volume are diagnostic measures; they are not substitutes for lead quality or commercial outcomes.

CreatikLab’s AI automation service can deliver a Claude Code plugin acceptance audit and implementation plan covering evaluation design, permissions, changed-file review, telemetry, failure handling and release evidence. To continue the diagnosis with context, tell Lia what the plugin does, which systems it can affect, what evidence already exists and which failure would be most costly.

What did Claude Code 2.1.269 add for plugin testing?

Anthropic says the release adds the claude plugin eval command. It runs a plugin evaluation suite and produces scored, reproducible results in JSON and HTML formats.

Does a passing plugin evaluation prove production readiness?

No. The documented command provides evaluation evidence, not a production guarantee. CreatikLab treats it as one acceptance input alongside security, permissions, integration, rollback and human-review checks.

Why keep both JSON and HTML reports?

JSON supports controlled processing and comparison, while HTML is easier for reviewers to inspect. Their exact schemas and retention requirements are not specified in the release note, so teams should verify them in their environment.

Can the evaluation command replace human review?

Anthropic does not state that it replaces review. Human owners should examine failed cases, unexpected passes, changed files, tool use and business consequences before approving a release.

What should be compared between plugin versions?

Compare the evaluation definition, environment, input fixtures, outputs, score, failure categories and reviewer decision. A score is not comparable when the underlying test conditions changed without documentation.

How can CreatikLab help?

CreatikLab can deliver a plugin acceptance audit covering evaluation design, permissions, observability, failure triage, release evidence and rollback criteria through its AI automation service. To continue with context, describe the plugin, workflow and risk in Lia.