An OpenAPI document that passes validation isn't necessarily one an AI agent can use. Grammar is one thing; semantic clarity, safety, and discoverability are another. The Jentic API Scorecard scores your OpenAPI document against the Jentic API AI Readiness Framework (JAIRF) across six dimensions and returns a single grade — so you know exactly where to improve. Pair it with the companion api-improve agent skill to apply those improvements automatically: non-breaking edits, a reusable OpenAPI Overlay, and a before/after changelog.
- Quick start
- What it scores
- How it works
- Requirements
- Install
- Try it now
- Control output depth
- Machine-readable output
- HTML report
- LLM analysis
- Anonymous vs keyed access
- Skills
- CLI reference
- GitHub Action
- Prefer a browser?
- Enterprise-ready by default
- Status
- License
-
Get a free key at the Jentic Scorecard API Keys page.
-
Score an OpenAPI document:
JENTIC_API_KEY=<your-key> npx @jentic/api-scorecard-cli score ./openapi.yaml
Each OpenAPI document is evaluated across six lenses — small, targeted improvements in any of them tend to produce outsized gains for both human developers and AI agents:
- Foundational Compliance (FC) — structural validity and conformance to OpenAPI itself.
- Developer Experience & Jentic Compatibility (DXJ) — documentation quality and how well the OpenAPI document plays with downstream tooling.
- AI-Readiness & Agent Experience (ARAX) — semantic clarity and the context an LLM needs to reason about each operation.
- Agent Usability (AU) — predictable, safe multi-step orchestration.
- Security (SEC) — declared auth schemes and trust boundaries.
- AI Discoverability (AID) — how easily an AI system can find and parse the OpenAPI document.
Scoring runs locally inside a Docker container in two phases. Analysis runs a battery of validators and structural checks against the OpenAPI document to produce a set of diagnostics and observations. Scoring maps those into ~35 signals across the six JAIRF dimensions, aggregates them into per-dimension scores, and rolls those up into a single weighted score and grade.
- Node.js 20 LTS or newer (
>= 20.19.0) with npm/npx. See Node.js downloads. - Docker installed and running. See Docker installation. The CLI pulls the scoring image automatically on first run.
- Network access to
ghcr.io(to pull the image) and to whatever URL hosts the OpenAPI document you're scoring (the engine fetches it from inside the container).
npm install -g @jentic/api-scorecard-cliThis installs the CLI globally. The scoring engine (Docker image) is downloaded automatically
the first time you run score — allow a minute or two on a typical connection.
For local files or non-OAK URLs you'll also need a JENTIC_API_KEY — see
Anonymous vs keyed access.
Verify the install:
jentic-api-scorecard --versionPrefer zero-install? You can skip the global install and use
npx— every example in this README works withnpx @jentic/api-scorecard-cliin place ofjentic-api-scorecard. Pin to a specific release withnpx @jentic/api-scorecard-cli@<version>(e.g.@1.0.0); the unpinned form resolves to whatever thelatestdist-tag points at on each invocation, whilenpm install -gpins you to the installed version until you explicitly update.
OpenAPI documents from Jentic Public APIs (OAK) score without any key, uncapped — no signup, no config:
npx @jentic/api-scorecard-cli@latest score \
https://raw.githubusercontent.com/jentic/jentic-public-apis/refs/heads/main/apis/openapi/swagger-api/petstore/1.0.27/openapi.jsonFor URLs outside OAK or local files, set the API key:
JENTIC_API_KEY=<your-key> npx @jentic/api-scorecard-cli@latest score \
https://petstore3.swagger.io/api/v3/openapi.jsonJENTIC_API_KEY=<your-key> npx @jentic/api-scorecard-cli@latest score ./openapi.yamlImportant
Free keys come with 100 scorings per month (resets at the start of each calendar month). See Anonymous vs keyed access for signup and quota details.
That's it. The CLI pulls the scoring engine automatically on first run.
The --detail flag lets you zoom in:
# Just the headline score and grade
npx @jentic/api-scorecard-cli@latest score --detail summary ./openapi.yaml
# Per-dimension breakdown (default)
npx @jentic/api-scorecard-cli@latest score --detail dimensions ./openapi.yaml
# Individual signals within each dimension
npx @jentic/api-scorecard-cli@latest score --detail signals ./openapi.yaml
# Full diagnostics with top 5 findings per severity
npx @jentic/api-scorecard-cli@latest score --detail diagnostics ./openapi.yamlAdd --format json to emit engine-verbatim JSON on stdout (filtered by whatever
--detail level you pick). Pretty stays the unconditional default; --format json
is the canonical way to get a stable machine-readable channel for CI gating, archival,
or LLM-assisted review.
# Gate on the headline score in CI
npx @jentic/api-scorecard-cli@latest score ./openapi.yaml --format json | jq .summary.score
# Capture the full evidence bundle to a file
npx @jentic/api-scorecard-cli@latest score ./openapi.yaml \
--format json --detail diagnostics --output report.json--output <file> (-o) writes the report to a path instead of stdout; the spinner stays on stderr.
--quiet (-q) suppresses the stderr spinner even in interactive terminals (the spinner already
auto-suppresses when stderr isn't a TTY). Engine warnings still pass through stderr.
Add --format html to render an interactive, self-contained HTML scorecard — a single file
with all JS and CSS inlined (no external CDN, works offline), suitable for CI artifacts and
dashboards. It honours --detail, so --detail diagnostics embeds the full evidence bundle.
Because the output is a full HTML document, the CLI refuses to print it straight into an
interactive terminal — redirect it or use -o:
# Redirect to a file
npx @jentic/api-scorecard-cli@latest score ./openapi.yaml --format html > scorecard.html
# Or write it with -o, at full detail
npx @jentic/api-scorecard-cli@latest score ./openapi.yaml \
--format html --detail diagnostics -o scorecard.htmlAdd --with-llm to unlock LLM-backed signals — deeper semantic reasoning about whether your API
descriptions are actionable for agents, whether error responses support autonomous recovery, and
more. Requires an LLM provider: cloud (OpenAI / Anthropic / Gemini / AWS Bedrock) or a local
OpenAI-compatible endpoint (Ollama, LM Studio, vLLM, …).
Without --with-llm, the LLM-backed signals are not evaluated — they hold an assumed-perfect
baseline score until --with-llm actually assesses them. A default (no---with-llm) scorecard
therefore reflects only the deterministic signals, with the LLM-backed ones sitting at that perfect
baseline; turn on --with-llm to have them genuinely assessed (which can lower your score).
export OPENAI_API_KEY=sk-...
export LLM_PROVIDER=OPENAI
export LIGHT_LLM_PROVIDER=OPENAI
export LLM_LIGHT_MODEL=gpt-4o-mini
JENTIC_API_KEY=<your-key> npx @jentic/api-scorecard-cli@latest score ./openapi.yaml --with-llmToken cost is low — the engine uses a lightweight model (e.g. Claude Haiku, GPT-4o-mini), processes operations in small batches, and caps at 7 batches regardless of spec size. Local models (Ollama) cost nothing per call.
If the LLM calls fail (bad credentials, an inaccessible model, an unreachable endpoint), the
affected LLM-backed signals get scored as perfect — which would inflate their dimension(s) and the
overall score. Rather than print a misleading scorecard, the CLI suppresses the report, names the
affected signals and the provider error on stderr, and exits 8 — so a CI job running
--with-llm fails loudly instead of passing on an inflated score. Fix the provider error and
retry, or re-run without --with-llm for a valid score from the non-LLM signals.
See LLM Signals guide for all provider recipes (including local Ollama), the full environment variable reference, and troubleshooting.
OpenAPI documents from Jentic Public APIs (OAK) score without any key and stay on the free tier — those URLs bypass key validation entirely. For everything else (local files, URLs outside OAK), get a key from the Jentic Scorecard API Keys page. Then set it:
export JENTIC_API_KEY=<your-key>Real keys are validated live by the container against api.jentic.com. The same call doubles
as the per-key usage / rate-limit accounting hit. Each free key gets 100 scorings per month,
resetting at the start of each calendar month. Once that quota is exhausted the CLI exits with
code 7 and prints the Retry-After value along with a link to upgrade your plan.
This repository ships two complementary agent skills — install either or both through whichever path fits your agent.
| Skill | Purpose |
|---|---|
jentic-api-scorecard |
Teaches coding agents to use the scoring CLI: scoring files and URLs, producing JSON/HTML, wiring into CI, and interpreting exit codes. |
jentic-api-improve |
Closes the loop: applies non-breaking improvements to a spec automatically, producing an improved spec, an OpenAPI Overlay, and a before/after changelog. |
A versioned agent skill that teaches AI coding agents how to use the CLI correctly — installing it, scoring files and URLs, producing JSON/HTML, wiring it into CI, enabling LLM analysis, and interpreting exit codes.
Claude Code users install it as a plugin — this repository doubles as a plugin marketplace:
/plugin marketplace add jentic/jentic-api-scorecard
/plugin install api-scorecard@jentic-api-scorecard
Once installed, the skill loads automatically when you ask Claude to score an OpenAPI document — no explicit invocation needed:
> Score ./openapi.yaml for AI-readiness
> How AI-ready is https://petstore3.swagger.io/api/v3/openapi.json?
To force it into context regardless of phrasing, invoke it explicitly with
/api-scorecard:jentic-api-scorecard.
Install it straight from this repository with the
skills CLI:
npx skills add jentic/jentic-api-scorecard --skill jentic-api-scorecardIt's also listed in the skills.sh directory.
The @jentic/api-scorecard-cli npm package also ships this skill inside its published
tarball, so it's discoverable by TanStack Intent for
projects that already depend on the CLI and want version-aligned agent guidance.
Scoring tells you what to fix; the jentic-api-improve skill closes the loop and
fixes it. Point an AI coding agent at an OpenAPI document and it runs a baseline
score, identifies the weak dimensions and the semantic diagnostics, applies
non-breaking improvements (adding summary/description/example/tags — never
changing existing paths, parameters, or response shapes), and produces three artifacts:
an improved spec, an OpenAPI Overlay
(the reusable, non-breaking delta), and a changelog with before/after scores.
This skill orchestrates the scorecard CLI plus a few extra command-line tools, so it
needs more than the CLI alone. In addition to the scorecard CLI's
Requirements (Node.js ≥ 20.19, a running Docker daemon, and a
JENTIC_API_KEY for local-file scoring), install:
pipx install jentic-openapi-tools # validation
pipx install jentic-apitools-cli # overlay verification (command: jentic-apitools)
pipx install check-jsonschema # overlay schema validation
# plus python3 and jq, which most systems already haveInstall the skill through whichever path fits your agent — the same three channels as the scoring skill:
The improve skill is a separate plugin in this repository's marketplace:
/plugin marketplace add jentic/jentic-api-scorecard
/plugin install api-improve@jentic-api-scorecard
Installing the plugin also registers the companion jentic-api-improve subagent (used
for multi-iteration improvement loops). Once installed, ask Claude to improve a spec:
> Improve ./openapi.yaml for AI-readiness
> Raise the JAIRF score of my API and give me an overlay
npx skills add jentic/jentic-api-scorecard --skill jentic-api-improveThe @jentic/api-scorecard-cli npm package ships this skill (and the companion agent
definition) inside its published tarball, so it's discoverable by
TanStack Intent for projects that already depend on the
CLI. The agent file lands at agents/jentic-api-improve.md; copy it into your
.claude/agents/ to enable the declarative subagent (optional — the skill falls back to
spawning a subagent from an inline brief).
jentic-api-scorecard [-V | --version] [-h | --help]
jentic-api-scorecard <command> [options]
| Command | Description |
|---|---|
score <input> |
Score an OpenAPI document by URL or local file path. |
Score an OpenAPI document by URL or local file path.
jentic-api-scorecard score <input> [options]
| Name | Description |
|---|---|
<input> |
https:// URL or local file path to an OpenAPI document. Required. |
| Flag | Default | Choices | Description |
|---|---|---|---|
--with-llm |
off | — | Enable LLM-backed analysis. Requires an LLM provider (see LLM analysis). |
--bundle |
off | — | Force CLI-side bundling for URL inputs: the CLI fetches the URL on the host, bundles with Redocly, and pipes to the container via stdin. Use for URLs only the host can reach (internal networks, VPN-gated specs, auth-required URLs). Requires JENTIC_API_KEY. No-op for local files. |
-d, --detail <level> |
dimensions |
summary, dimensions, signals, diagnostics |
Payload depth (see Control output depth). |
-f, --format <fmt> |
pretty |
pretty, json, html, markdown, sarif |
Output encoding (markdown is a GitHub-flavored projection for $GITHUB_STEP_SUMMARY / PR comments; sarif projects diagnostics only as SARIF 2.1.0 for GitHub code-scanning and always emits full diagnostics regardless of --detail; see also Machine-readable output and HTML report). |
-o, --output <file> |
stdout | — | Write the formatted report to <file>. The spinner stays on stderr. |
-q, --quiet |
off | — | Suppress the stderr spinner regardless of TTY. |
-h, --help |
— | — | Show usage for score. |
| Variable | When | Purpose |
|---|---|---|
JENTIC_API_KEY |
URLs outside OAK and local files | Real key issued at the Jentic Scorecard API Keys page; validated live against api.jentic.com (see Anonymous vs keyed access). Free quota: 100 scorings per calendar month. |
| LLM provider + routing vars | With --with-llm |
The CLI auto-detects credentials (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, AWS keys) and routing (LLM_PROVIDER, LIGHT_LLM_PROVIDER, LLM_MODEL, LLM_LIGHT_MODEL, *_API_URL, LLM_MAX_TOKENS) and forwards them to the container; loopback URLs are rewritten so a host-side Ollama is reachable. Full reference: LLM Signals guide. |
| Code | Meaning |
|---|---|
| 0 | Scoring completed (regardless of the score itself). |
| 1 | Generic error (bad input, unexpected container failure, output write failure). |
| 2 | Auth: JENTIC_API_KEY is set to a value the Jentic backend does not recognize, or a local file / stdin input was used without the key set. |
| 3 | Anonymous gate rejected: URL outside the OAK allowlist and no key set. |
| 4 | Docker not installed or daemon unreachable. |
| 5 | Spec fetch, parse, or host-side bundling failure (local files and --bundle URLs). |
| 6 | Engine invocation failure. |
| 7 | Rate limit reached: the key is valid but the user is over quota. Message includes the server-provided detail and the Retry-After header when present. |
| 8 | LLM analysis failed under --with-llm: the provider call failed, so the LLM-derived signals would be scored as perfect and inflate the result. The CLI suppresses the report and prints the affected signals + provider error on stderr. Re-run without --with-llm for a valid non-LLM score. |
A composite GitHub Action wraps the CLI for CI: it scores an OpenAPI document, gates the build on the score, uploads SARIF findings to the Security tab, attaches the HTML scorecard as a downloadable artifact, and renders a Markdown summary on the run page.
name: API readiness
on: [pull_request]
permissions:
contents: read
security-events: write # required so SARIF uploads to the Security tab
jobs:
scorecard:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: jentic/jentic-api-scorecard@v1 # floating major tag (or pin a full version / commit SHA)
with:
input: ./openapi.yaml
api-key: ${{ secrets.JENTIC_API_KEY }}
min-score: '70'To enable LLM-backed analysis, set with-llm: true and provide the provider credentials and
routing as job-level env: — the action forwards them to the engine but does not turn raw secrets
into those variables for you, so the run fails fast if none are present:
jobs:
scorecard:
runs-on: ubuntu-latest
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
LLM_PROVIDER: OPENAI
LIGHT_LLM_PROVIDER: OPENAI
LLM_LIGHT_MODEL: gpt-4o-mini
steps:
- uses: actions/checkout@v4
- uses: jentic/jentic-api-scorecard@v1
with:
input: ./openapi.yaml
api-key: ${{ secrets.JENTIC_API_KEY }}
with-llm: 'true'See the LLM Signals guide for the full provider matrix (cloud and local Ollama) and the variable reference.
| Input | Default | Description |
|---|---|---|
input |
— | Required. https:// URL or local file path to an OpenAPI document. |
api-key |
— | Jentic API key. Required for local files and non-OAK URLs; never logged. |
github-token |
workflow token | Token for the SARIF upload (see fork PRs below). |
min-score |
— | Fail when the score is below this. Unset = no gate. |
max-errors |
— | Fail when error-level findings exceed this. Unset = no gate. |
max-warnings |
— | Fail when warning-level findings exceed this. Unset = no gate. |
severity |
warning |
Minimum level kept in the SARIF. |
max-findings |
5000 |
Cap on SARIF results, lowest-severity dropped first. |
with-llm |
false |
Enable LLM-backed analysis. |
summary-detail |
dimensions |
Markdown run-summary depth: summary, dimensions, signals, diagnostics. |
cli-version |
action's release | CLI version to run (pins the engine image); defaults to the version shipped with the action ref you pinned. |
Choosing your gates. Set min-score to fail PRs below a readiness bar, and/or max-errors /
max-warnings to fail on too many findings. Each finding carries one of three levels — error,
warning, note — which is how it's labelled on its Security-tab alert. (If you also read
--format json, those map from its 1–4 severity: 1 is error, 2 is warning, 3 and 4 are both
note.) The severity input only trims which findings reach the Security tab; it never changes your
gate, so quieting the Security tab to errors-only won't let a warning-heavy spec slip through.
Outputs land even on a failing build. When a gate fails, you still get the SARIF findings, the HTML artifact, and the Markdown summary — a failing PR is exactly when you want to see them.
Choose your pin deliberately. The @v1 major tag (used in the examples above) floats to the
newest v1.x.x on every release, so you auto-receive fixes without a workflow edit and stay within
one major — no breaking changes. The trade-off: a release can ship a new scoring engine, and the
same document can score differently across engine versions, so a green PR can turn red when @v1
moves under it. If you gate on a hard threshold and need that build reproducible, pin a full version
(@v1.8.3) or a commit SHA instead, and bump it deliberately when you're ready to adopt a new
engine.
On fork PRs, GitHub gives the workflow a read-only token, so the Security-tab upload can't run;
the action skips it with a notice and still publishes the HTML artifact and Markdown summary.
(Uploading fork findings anyway needs a writable token from a base-context workflow, supplied via
github-token.) In the Security tab, findings link to the document but not yet to a specific line.
jentic.com/scorecard offers the same scoring in a web UI — paste a URL or drop a file, no Docker or Node required.
For teams that need to know exactly what's running, verify exactly what was shipped, and run without a runtime dependency on Jentic.
Scoring runs entirely in a container on your own machine. Local files are
piped in over stdin; URLs are fetched on your side — by the container's engine,
or host-side by the CLI when you pass --bundle — never by Jentic. Either way,
your spec is never uploaded.
The only call to Jentic is a key-check round-trip against api.jentic.com —
it carries your key, never any part of your spec, and OAK URLs (jentic-public-apis)
skip even that. The one exception is --with-llm, which sends spec context to the
LLM provider you choose (point it at a local Ollama to keep that on-machine too).
Every component in the scoring stack — runner, CLI, release pipeline, and engine — is Apache 2.0 licensed and source-readable. No proprietary blobs, no closed-source shims. Read the code that's about to grade your specs before you adopt it; audit any line, redistribute under the license terms, fork if you ever need to.
Every npm tarball and every GHCR image is signed by Sigstore
with SLSA provenance and an SPDX SBOM. Signing happens inside an OIDC-driven
GitHub Actions workflow with no long-lived publishing secrets — there is no
NPM_TOKEN, no PAT, and no human keyholder in the release chain. One command
verifies an artifact end-to-end before you install it:
- npm package supply chain → —
npm provenance, SPDX SBOM, trusted publishing, and the
gh attestation verifyrecipes. - Docker image supply chain → —
per-platform SBOMs, dual-store attestations (BuildKit OCI referrers + Sigstore), and
verification via either
docker buildx imagetools inspectorgh attestation verify.
The image is a closed system at scoring time: every Python wheel, Node.js binary, and validator tarball it needs is baked in at build time, so scoring pulls no runtime packages from PyPI or npmjs. Multi-arch images (linux/amd64 + linux/arm64) ship from the same release, so the same guarantees hold on Apple Silicon dev machines, ARM CI runners, and x86 servers alike.
CLI version, image tag, and engine version are locked one-to-one. Pinning
@jentic/api-scorecard-cli@<version> resolves to a specific image tag, which
in turn pins an exact engine release and exact validator versions. Last
month's score is reproducible from last month's pin.
The CLI ships stable under the latest npm dist-tag — release cadence is driven by
Conventional Commits. Track in-flight work in
specs/roadmap.md.
The :unstable Docker image is rebuilt on every push to main for direct docker run users.
Versioned images are published alongside each CLI release.
To see which Jentic API AI Readiness Framework signals are active in the current release, check out the scoring engine implementation status.
Jentic API Scorecard is licensed under the Apache 2.0 license. Jentic API Scorecard comes with an explicit NOTICE file containing additional legal notices and information.