diff --git a/.claude/agents/software-pack-auditor.md b/.claude/agents/software-pack-auditor.md new file mode 100644 index 0000000..4ef0879 --- /dev/null +++ b/.claude/agents/software-pack-auditor.md @@ -0,0 +1,181 @@ +--- +name: software-pack-auditor +description: >- + Audits a Nebari software-pack repository against the nebari-dev + software-pack-template release-readiness checklist and produces a readiness + score, the achieved maturity level (Experimental/Alpha/Beta/GA), and a + prioritized fix list. Use when asked to "audit this pack", "score pack + readiness", "is this pack ready for alpha/beta/GA", "review a software pack + against the template", or to compare several packs. Read-only on the pack; + writes reports only under the audit tool's reports/ directory. +tools: Bash, Read, Glob, Grep, Write, Edit +model: inherit +--- + +You are the software-pack auditor. You judge a Kubernetes application +repository against the Nebari software-pack contract and tell its owners +exactly what stands between them and the next maturity level. + +# Modes + +**Audit mode (default) is read-only on the pack.** You never create, edit, +delete, check out, or commit anything inside the pack repository. The only +files you write are `reports/.json` and `reports/.md` under the +audit tool's directory. Branch content is read through `git archive` or +`git show`, never a checkout. If a check would need a change to the pack to +proceed (for example a values file that does not render), record the finding +and work around it with `--values` / `--set` on the scanner command line. + +**Fix mode is opt-in.** When the audit is done, you may offer, in one line, to +apply specific fixes from the Top fixes list. You enter fix mode only after +the user explicitly says yes in this conversation; a request to "audit", +"score", or "review" is never permission to edit. In fix mode: + +- apply only the fixes the user approved, one checklist id at a time; +- stay inside the pack repository the user named, on a branch they named or + a new branch you announce; never commit or push unless asked; +- re-run the scanner after each fix and report the before/after status; +- for a generated repository (a `GENERATED.md` or "do not edit" banner), + say that the fix must land in the generator and ask where that is before + editing anything. + +The contract is `nebari-dev/software-pack-template`: a pack is any deployment +that carries a `NebariApp` custom resource so the nebari-operator can give it a +route, TLS, and Keycloak OIDC. The promotion rubric is that repo's +`docs/release-readiness-checklist.md`, encoded item-for-item in +`checklist.yaml` next to `audit.py`. + +# Where the tooling lives + +Resolve `TOOL_DIR` as the first of these that contains `audit.py`: +`tools/pack-audit/` (inside software-pack-template), `software-pack-audit/` +(this workspace), or the path the user gives you. Everything below is relative +to `TOOL_DIR`. + +- `audit.py` deterministic scanner and scorer +- `checklist.yaml` the rubric (ids, levels, categories, check type, expectation) +- `reference/upstream-release-readiness-checklist.md` upstream rubric text +- `reference/upstream-nebariapp-crd-reference.md` NebariApp field reference +- `reference/pack-metadata.schema.json` dashboard schema +- `reference/template-conventions.md` what a healthy pack looks like, distilled + from the template and the first-party packs +- `reports/` output + +# Procedure + +1. **Identify the target.** Confirm the repo path and which git ref holds the + pack. If the default branch is a stub and the content is on another branch, + audit that branch with `--ref` and say so in the report. Never check out + or modify the user's working tree. + +2. **Run the scanner.** + ``` + python3 -I TOOL_DIR/audit.py scan [--ref ] [--name ] [--out TOOL_DIR/reports] + ``` + If `helm template` failed in both modes, read the error, find the values + file or `--set` the chart needs (look at the chart README, `ci/`, `examples/`, + `.gitlab-ci.yml`), and rerun with `--values` / `--set`. Record what you + needed; a chart that cannot render from its documented example is itself a + finding. + +2b. **Verify on a cluster when asked or when the environment allows.** If the + user asked for cluster verification, or `docker`, `kind`, `kubectl`, `helm` + and `make` are all available and the user has not objected to a local kind + cluster being created, run + `python3 -I TOOL_DIR/audit.py verify reports/.json [--chart ]` + after the scan. It installs the pack on a throwaway kind cluster running the + Nebari stack, records NA-01 / NA-04, and sets "Runs on Nebari" to verified or + no. Run `--dry-run` first and show the plan if the user has not seen it + before. Never run it against a cluster the user did not name as disposable. + Without a cluster, the field stays at "likely" or "no" from the static + check, and you say so in the verdict. + +3. **Read the pack like a platform engineer.** Read, at minimum: the README, + `Chart.yaml`, `values.yaml`, every `templates/*.yaml` that renders a + workload or the NebariApp, the CI definitions, the examples, and any docs the + README links for prerequisites, auth, troubleshooting, sizing, upgrade. Use + the rendered manifests under the `rendered_dir` the scanner reports. + +4. **Decide every JUDGMENT item.** The scanner leaves these pending: + OI-05, IN-02, NA-02, DOC-03, DOC-05, TEL-01, TEL-02, RE-03, RE-05. For each, + set `PASS`, `PARTIAL`, `FAIL`, or `NA` with one sentence of evidence that + cites a file path or heading. Rules: + - PASS only when the expectation in `checklist.yaml` is met in substance, + not merely by a heading with the right name. + - PARTIAL when the material exists but is incomplete, scattered across + docs, or wrong for the Nebari deployment path (for example auth docs that + describe a Traefik forward-auth path the pack no longer uses). + - NA needs a justification that the item cannot apply (for example TEL-01 + when the app exposes no metrics endpoint and the docs say so). + - Never upgrade an automated FAIL to PASS without naming the evidence the + scanner missed; you may downgrade an automated PASS when the evidence is + hollow. + +5. **Review the automated results for false positives.** The scanner uses + heuristics (heading regexes, tag names, grep). Where you override a status, + keep the scanner's evidence and append `override: `. + +6. **Write the verdict back and rescore.** Edit `reports/.json` + (`checks..status` and `.evidence`), then run + `python3 -I TOOL_DIR/audit.py report reports/.json` to regenerate the + markdown with the final score. + +7. **Write the narrative.** Prepend to `reports/.md` (above the + generated content) a short section with: + - **Verdict**: one paragraph. Achieved level, score, and the single biggest + obstacle. + - **Top fixes**: numbered, ordered by level gate then effort. Each one + names the checklist id, the file to change, and what "done" looks like. + Group what unblocks Experimental, then Alpha, then Beta, then GA. + - **Risks outside the checklist**: anything from the scanner's + observations or your reading that would bite in production (double + ingress, unpinned images, placeholder secrets, operator field misuse, + generated-repo workflow, branch not merged). + - **Needs cluster verification**: the MANUAL items, with the exact + command or check a platform engineer should run. + +# Judgment guidance + +- The template is the source of truth. When a pack deviates (hand-written + NebariApp instead of the `nebari-app` library chart, GitLab CI instead of + GitHub Actions, a vendored upstream chart instead of a dependency), ask + whether the deviation still delivers the checklist's intent. Equivalent + delivery is PASS with a note; a deviation that drops a guarantee is FAIL. +- First-party-only items (dashboard listing, nebari-dev helm-repository sync, + pre-sales sign-off) stay NA or MANUAL for vendor packs. Say what the vendor + equivalent is. +- Routing: a NebariApp whose `spec.routing` is omitted gets no HTTPRoute on + operator v0.1.1. Treat that as blocking. +- Auth: `enforceAtGateway: false` is legitimate for apps that do OAuth + natively (Harbor, Grafana) only when the docs explain how the app consumes + the operator-provisioned client Secret. +- Secrets: `secrets.existingSecret`, `required`, or `lookup`-based generation + are acceptable. Literal placeholder passwords in default-rendered Secrets are + not, even when labelled CHANGEME, because `helm install` succeeds with them. +- Images: `latest`, `prod`, `main`, or a missing tag is unpinned. A digest or an + immutable tag passes. A documented offline bundle that pins digests at install + time is PARTIAL until the repo carries the pins. +- Telemetry: NIC's OpenTelemetry collector discovers metrics through + `prometheus.io/scrape` pod annotations or OTLP push, and the LGTM pack ships + every container's stdout to Loki. TEL-01 passes on annotations/OTLP (a bare + ServiceMonitor is PARTIAL); TEL-02 passes on stdout logging, with plain-text + format noted rather than deducted. +- Generated repositories: fixes must land in the generator. Say so, but score + the repository as it is. +- Multi-chart repositories without one pack chart: score the chart that would + front the user-facing app, and record under Observations that the pack has + no single installable unit. + +# Output discipline + +- Every status needs evidence a reader can verify: a path, a heading, a + rendered resource name, or a command output. +- Do not invent cluster behaviour. If it needs a cluster, it is MANUAL. +- Keep the fix list concrete: "add `routing: {routes: [{pathPrefix: /}], tls: + {enabled: true}}` under `nebariapp:` in chart/values.yaml", not "improve + routing". +- Report completion with the path to `reports/.md`, the score, the + achieved level, the "Runs on Nebari" state (and whether it is verified or + static), and the top three fixes. Mention `audit.py export --format sarif|junit` + when the user's CI or review tooling could ingest the findings. End with one line offering fix + mode for named items; do not start it. diff --git a/.github/workflows/pack-audit.yaml b/.github/workflows/pack-audit.yaml new file mode 100644 index 0000000..10cd9ea --- /dev/null +++ b/.github/workflows/pack-audit.yaml @@ -0,0 +1,291 @@ +name: Pack audit + +# Reusable workflow: score a software pack against docs/release-readiness-checklist.md +# with tools/pack-audit, publish the report, optionally decide the judgment items +# with a language model, and optionally gate on a maturity level. +# +# Call it from a pack repository: +# +# permissions: +# contents: read +# security-events: write # SARIF upload (needs code scanning; skipped otherwise) +# pull-requests: write # summary comment on PRs +# copilot-requests: write # only if judge: true with the Copilot runner +# jobs: +# audit: +# uses: nebari-dev/software-pack-template/.github/workflows/pack-audit.yaml@main +# with: +# set: | +# my-chart.secrets.password=ci-only +# judge: ${{ github.event_name == 'push' }} +# secrets: inherit +# +# The scan needs no secrets and no AI. The judge job runs only when `judge` is +# true and is ONE model request per audit (~9k input tokens, <1k output). Two +# runners: +# - default: GitHub Copilot CLI authenticated with the workflow token and billed +# to the organization's AI credits (org policy "Allow use of Copilot CLI billed +# to the organization" + permission copilot-requests: write). +# - `judge-endpoint`: any OpenAI-compatible /chat/completions base URL (OpenRouter, +# Azure, OpenAI, Ollama, vLLM, LiteLLM, Anthropic's compatibility endpoint...) +# with the PACK_AUDIT_LLM_API_KEY secret and `judge-model`. +# Either way the verdicts land in the results JSON with the model named, and a +# person can override them and rerun `audit.py report`. + +on: + workflow_call: + inputs: + name: + type: string + default: "" + description: Pack name for the report (default repository name) + values: + type: string + default: "" + description: Values files to render with, whitespace separated + set: + type: string + default: "" + description: helm --set key=value pairs the chart needs to render, whitespace separated + min-level: + type: string + default: "" + description: "Fail unless the repo-readiness level is at least this: E|A|B|GA (or experimental|alpha|beta|ga). Empty = report only" + judge: + type: boolean + default: false + description: Decide the judgment items with a language model (one request per run) + judge-model: + type: string + default: "" + description: Model for the judge. Copilot runner a model id the CLI accepts; endpoint runner the provider model id. Empty = runner default + judge-endpoint: + type: string + default: "" + description: OpenAI-compatible base URL to use instead of Copilot (needs secret PACK_AUDIT_LLM_API_KEY) + comment: + type: boolean + default: true + description: Post or update a summary comment on pull requests from this repository + tool-repo: + type: string + default: nebari-dev/software-pack-template + description: Repository that carries tools/pack-audit + tool-ref: + type: string + default: main + description: Branch or tag of tool-repo to take tools/pack-audit from + secrets: + PACK_AUDIT_LLM_API_KEY: + required: false + +env: + PACK_NAME: ${{ inputs.name != '' && inputs.name || github.event.repository.name }} + +jobs: + scan: + name: Scan + runs-on: ubuntu-latest + permissions: + contents: read + security-events: write + steps: + - uses: actions/checkout@v5 + + - name: Fetch tools/pack-audit from ${{ inputs.tool-repo }}@${{ inputs.tool-ref }} + env: + TOOL_REPO: ${{ inputs.tool-repo }} + TOOL_REF: ${{ inputs.tool-ref }} + run: | + git clone --quiet --depth 1 --branch "$TOOL_REF" --filter=blob:none --sparse \ + "https://github.com/$TOOL_REPO.git" "$RUNNER_TEMP/software-pack-template" + git -C "$RUNNER_TEMP/software-pack-template" sparse-checkout set tools/pack-audit + echo "TOOL_DIR=$RUNNER_TEMP/software-pack-template/tools/pack-audit" >> "$GITHUB_ENV" + + - uses: azure/setup-helm@v5 + + - name: Install kubeconform + run: | + curl -sSL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | \ + sudo tar -xz -C /usr/local/bin kubeconform + + - name: Install Python dependencies + run: python3 -m pip install --quiet --user pyyaml jsonschema + + - name: Scan + env: + VALUES: ${{ inputs.values }} + SETS: ${{ inputs.set }} + run: | + args=() + for f in $VALUES; do args+=(--values "$f"); done + for kv in $SETS; do args+=(--set "$kv"); done + python3 -I "$TOOL_DIR/audit.py" scan . --name "$PACK_NAME" --out reports "${args[@]}" + + - name: Export SARIF + run: python3 -I "$TOOL_DIR/audit.py" export "reports/$PACK_NAME.json" --format sarif --out reports/pack-audit.sarif + + - name: Upload SARIF to code scanning + uses: github/codeql-action/upload-sarif@v3 + continue-on-error: true # private repositories need code scanning enabled; the report is still in the artifact and summary + with: + sarif_file: reports/pack-audit.sarif + category: pack-audit + + - name: Job summary + run: cat "reports/$PACK_NAME.md" >> "$GITHUB_STEP_SUMMARY" + + - uses: actions/upload-artifact@v4 + with: + name: pack-audit-reports + path: reports/ + retention-days: 30 + + judge: + name: Judge (one model request) + needs: scan + if: ${{ inputs.judge }} + runs-on: ubuntu-latest + continue-on-error: true # a judge failure (org policy off, model retired, endpoint down) must not hide the scan result + permissions: + contents: read + copilot-requests: write + steps: + - uses: actions/checkout@v5 + + - name: Fetch tools/pack-audit from ${{ inputs.tool-repo }}@${{ inputs.tool-ref }} + env: + TOOL_REPO: ${{ inputs.tool-repo }} + TOOL_REF: ${{ inputs.tool-ref }} + run: | + git clone --quiet --depth 1 --branch "$TOOL_REF" --filter=blob:none --sparse \ + "https://github.com/$TOOL_REPO.git" "$RUNNER_TEMP/software-pack-template" + git -C "$RUNNER_TEMP/software-pack-template" sparse-checkout set tools/pack-audit + echo "TOOL_DIR=$RUNNER_TEMP/software-pack-template/tools/pack-audit" >> "$GITHUB_ENV" + + - uses: actions/download-artifact@v4 + with: + name: pack-audit-reports + path: reports + + - name: Install Python dependencies + run: python3 -m pip install --quiet --user pyyaml jsonschema + + - name: Build the judge prompt + run: python3 -I "$TOOL_DIR/audit.py" judge "reports/$PACK_NAME.json" --repo . --prompt-out reports/judge-prompt.md + + - name: Judge with an OpenAI-compatible endpoint + if: ${{ inputs.judge-endpoint != '' }} + env: + PACK_AUDIT_LLM_API_KEY: ${{ secrets.PACK_AUDIT_LLM_API_KEY }} + PACK_AUDIT_LLM_MODEL: ${{ inputs.judge-model }} + ENDPOINT: ${{ inputs.judge-endpoint }} + run: python3 -I "$TOOL_DIR/audit.py" judge "reports/$PACK_NAME.json" --repo . --endpoint "$ENDPOINT" + + - uses: actions/setup-node@v4 + if: ${{ inputs.judge-endpoint == '' }} + with: + node-version: 22 + + - name: Judge with GitHub Copilot (billed to the organization) + if: ${{ inputs.judge-endpoint == '' }} + env: + GITHUB_TOKEN: ${{ github.token }} + COPILOT_AUTO_UPDATE: "false" + JUDGE_MODEL: ${{ inputs.judge-model }} + run: | + npm install -g --silent @github/copilot + model_arg=() + if [ -n "$JUDGE_MODEL" ]; then model_arg=(--model="$JUDGE_MODEL"); fi + copilot -p "$(cat reports/judge-prompt.md)" -s --no-ask-user --no-color "${model_arg[@]}" > reports/judge-answer.txt + python3 -I "$TOOL_DIR/audit.py" judge "reports/$PACK_NAME.json" --repo . \ + --ingest reports/judge-answer.txt --decided-by "copilot:${JUDGE_MODEL:-default}" + + - name: Job summary + run: cat "reports/$PACK_NAME.md" >> "$GITHUB_STEP_SUMMARY" + + - uses: actions/upload-artifact@v4 + with: + name: pack-audit-reports + path: reports/ + retention-days: 30 + overwrite: true + + comment: + name: PR comment + needs: [scan, judge] + if: ${{ always() && needs.scan.result == 'success' && inputs.comment && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository }} + runs-on: ubuntu-latest + permissions: + pull-requests: write + steps: + - uses: actions/download-artifact@v4 + with: + name: pack-audit-reports + path: reports + + - name: Post or update the audit comment + env: + GH_TOKEN: ${{ github.token }} + PR: ${{ github.event.pull_request.number }} + run: | + marker="" + headline=$(python3 - <<'PY' + import json, os + d = json.load(open(f"reports/{os.environ['PACK_NAME']}.json")) + s = d["score"] + line = f"{s['overall_pct']}% · repo-readiness level: {s['repo_level'] or 'none'} · runs on Nebari: {s['runs_on_nebari']['state']}" + if d.get("judge"): + u = d["judge"].get("usage") or {} + line += f" · judged by {d['judge']['decided_by']}" + (f" ({u.get('prompt_tokens')} in / {u.get('completion_tokens')} out tokens)" if u else "") + print(line) + PY + ) + { + echo "$marker" + echo "## Pack audit: $headline" + echo + echo "
Full report" + echo + head -c 60000 "reports/$PACK_NAME.md" + echo + echo "
" + } > comment.md + cid=$(gh api "repos/$GITHUB_REPOSITORY/issues/$PR/comments" --paginate \ + --jq ".[] | select(.body | startswith(\"$marker\")) | .id" | head -n1) + if [ -n "$cid" ]; then + gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$cid" -F body=@comment.md > /dev/null + else + gh api "repos/$GITHUB_REPOSITORY/issues/$PR/comments" -F body=@comment.md > /dev/null + fi + + gate: + name: Gate + needs: [scan, judge] + if: ${{ always() && needs.scan.result == 'success' && inputs.min-level != '' }} + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Fetch tools/pack-audit from ${{ inputs.tool-repo }}@${{ inputs.tool-ref }} + env: + TOOL_REPO: ${{ inputs.tool-repo }} + TOOL_REF: ${{ inputs.tool-ref }} + run: | + git clone --quiet --depth 1 --branch "$TOOL_REF" --filter=blob:none --sparse \ + "https://github.com/$TOOL_REPO.git" "$RUNNER_TEMP/software-pack-template" + git -C "$RUNNER_TEMP/software-pack-template" sparse-checkout set tools/pack-audit + echo "TOOL_DIR=$RUNNER_TEMP/software-pack-template/tools/pack-audit" >> "$GITHUB_ENV" + + - uses: actions/download-artifact@v4 + with: + name: pack-audit-reports + path: reports + + - name: Install Python dependencies + run: python3 -m pip install --quiet --user pyyaml jsonschema + + - name: Gate + env: + MIN_LEVEL: ${{ inputs.min-level }} + run: python3 -I "$TOOL_DIR/audit.py" gate "reports/$PACK_NAME.json" --min-level "$MIN_LEVEL" diff --git a/.gitignore b/.gitignore index 4c8bfa9..7ef399b 100644 --- a/.gitignore +++ b/.gitignore @@ -27,3 +27,6 @@ Thumbs.db # Claude CLAUDE.md + +# pack-audit output +tools/pack-audit/reports/ diff --git a/docs/superpowers/specs/2026-10-09-pack-audit-design.md b/docs/superpowers/specs/2026-10-09-pack-audit-design.md new file mode 100644 index 0000000..35fa7f3 --- /dev/null +++ b/docs/superpowers/specs/2026-10-09-pack-audit-design.md @@ -0,0 +1,170 @@ +# Software-pack readiness auditor (`tools/pack-audit/`) + +- **Date:** 2026-10-09 +- **Repo:** nebari-dev/software-pack-template +- **Status:** Proposed (design) + +This document records how the auditor was derived, so the PR explains the +design rather than only dropping files. + +## Goal + +Give pack authors and the platform team a repeatable, evidence-backed answer to +"how ready is this pack, and what exactly do we fix next?", using the same +rubric the org already promotes packs with. The output is a score, an achieved +maturity level, blockers per level, and a fix list that names files. + +## Sources of truth (in priority order) + +1. `software-pack-template/docs/release-readiness-checklist.md`. The rubric. + Every scored item in `checklist.yaml` maps to one line of it and keeps its + level tag ([E]/[A]/[B]/[GA]). +2. `software-pack-template/README.md`, `docs/nebariapp-crd-reference.md`, + `docs/pack-docs-requirements.md`, the five `examples/`, and the CI + workflows. These define the expected layout, the `nebariapp:` values + contract, the lint/template-both-modes/kubeconform CI pattern, and the + docs-site stack. +3. `nebari-dev/software-pack-dashboard/schema/pack-metadata.schema.json` + (vendored) and `tracked-packs.yaml`. Metadata validation and the list of + first-party packs. +4. `nebari-dev/nebari-operator` `api/v1/nebariapp_types.go` and + `internal/controller/nebariapp_controller.go` at v0.1.1/main. Used for the + NebariApp field set and for runtime semantics the docs understate. +5. Eight first-party packs (llm-serving, mlflow, superset, harbor, langfuse, + data-science, lgtm, nebari-nexus) read for the conventions that mature packs + actually follow: `chart/` or root chart, `examples/argocd-application.yaml`, + `standalone-values.yaml`, `dev/` kind Makefile, `SECURITY.md`, + `CODEOWNERS`, docs on `@nebari/starlight`, hand-written NebariApp templates + with explicit `routing`, `check-jsonschema` on `pack-metadata.yaml` in lint. + +Nothing in nebari-dev already does this: a `gh search code` over the org on +2026-10-09 found no `.claude/agents`, skills, or audit tooling for packs, and no +`software-pack-tools` repo. `skillsctl` is a skills registry, not a pack tool. + +## Design decisions + +**Split deterministic from judgment.** Most checklist rows are either +mechanical (file exists, helm lint passes, probes present) or need a reader +(does the README actually explain who the pack is for?). `audit.py` owns the +first class and leaves the second as `JUDGMENT` with gathered evidence; the +agent decides them and rescoring is deterministic again. This keeps scores +reproducible and lets a human run the scanner without Claude. + +**Encode the rubric as data.** `checklist.yaml` carries id, level, category, +check type, applicability (`first-party`, `standalone`, `auth`) and the +expectation text. Changing the rubric upstream means editing the YAML, not the +code. One item was added that the upstream list lacks, `NA-07` (NebariApp spec +validates and resolves), because it is the cheapest predictor of the +cluster-only `NA-01` and caught real defects in the first run. + +**Score by gate, not just by percentage.** The percentage (level-weighted +E=4/A=3/B=2/GA=1) gives a trend line, but promotion is a gate: the report's +headline is the highest level with no FAIL/PARTIAL at or below it. MANUAL and +NA items are excluded from the score so a vendor pack is not penalized for +first-party-only rows (dashboard listing, helm-repository sync, pre-sales +sign-off); they are listed as "needs verification" instead. + +**Render, do not just grep.** Security, probe, image and NebariApp checks run +on `helm template` output in both NebariApp modes. When the default +`values.yaml` cannot render, the scanner searches the repo's own values files +(singles, then pairs) the way the pack's CI would, and records which profile +it needed, because "does not render from defaults" is itself a finding. + +**Validate the NebariApp against the operator, not the docs.** The template's +CRD reference tracks operator alpha.19 and lacks fields the operator now +accepts (`routing.tls.secretName`, `landingPage.iconLight/iconDark`). The +field set in `audit.py` was taken from `nebariapp_types.go` at v0.1.1. +Reading `nebariapp_controller.go` also showed that an omitted `spec.routing` +means no HTTPRoute at all; two template examples omit it, which is a template +bug to raise alongside this PR. + +**Accept equivalent deviations.** Hand-written NebariApp templates (harbor), +GitLab CI (vendor repos), and vendored upstream charts are scored on whether +they deliver the checklist's guarantee, with the deviation recorded under +Observations. The agent prompt spells out the equivalence rules so two auditors +reach the same verdict. + +**Keep the agent read-only on the pack; fixes are a separate, opt-in mode.** +Auditing writes only under `reports/`; branch content is read through +`git archive --ref`, never a checkout. The agent may offer fixes at the end but +enters fix mode only after the user explicitly agrees, applies approved items +one checklist id at a time with a rescan after each, and never commits or +pushes unless asked. Keeping the verdict and the edits in separate steps is what +makes the audit trustworthy to a vendor whose repo it is. + +**Pre-sales and sign-off are reported, never scored.** The upstream checklist +lists pre-sales verification and promotion sign-off as blockers, but they are +process gates owned by people, not properties of the repository. Scoring them +would punish every pack by the same constant and tell the owner nothing to +fix. They remain in the report so a promotion PR has the full list (decision +2026-10-09). + +**Telemetry is judged against what the platform actually provides.** NIC runs +an OpenTelemetry collector DaemonSet whose Prometheus receiver discovers pods +through `prometheus.io/scrape` / `prometheus.io/port` annotations (role: pod) +and accepts OTLP push; the LGTM pack supplies Loki/Tempo/Mimir, ships every +container's stdout to Loki via promtail, and its Grafana sidecar loads any +`grafana_dashboard`-labelled ConfigMap from any namespace. So a pack owes an +annotated `/metrics` endpoint (or a documented reason), a structured log +format, and optionally a dashboard ConfigMap. The upstream checklist's +"ServiceMonitor or PodMonitor" wording describes a mechanism the NIC collector +does not read; the auditor accepts annotations/OTLP as PASS and a bare +ServiceMonitor as PARTIAL. + +**"Runs on Nebari" is a separate three-state answer, not the score.** A pack +can score well on documentation and security while having no NebariApp at +all, and a pack with a perfect NebariApp can still fail on a cluster. So every +report also says `verified` (installed on a cluster with the operator and the +NebariApp reached Ready), `likely` (static NA-07 pre-check passes), or `no`. +`audit.py verify` reuses the template's `dev/Makefile` cluster recipe to move +packs from likely to verified without a person, and writes the operator's own +condition reasons (`NamespaceNotOptedIn`, `ServiceNotFound`, +`CertificateNotReady`) into the report when it fails. + +**Findings export in standard formats.** The results JSON is the source of +truth, and `audit.py export` renders it as SARIF 2.1.0 (rule id = checklist +id, `helpUri` = the rule link), JUnit XML (one suite per maturity level), or +CSV, so GitHub code scanning, GitLab test reports, and spreadsheets can +consume the same findings without a custom parser. + +## Known limits + +- Heading regexes are crude; the agent is expected to correct false + positives/negatives and record `override:` reasons. +- `NA-05` (group membership) and `IN-01` (README-only install) still need a + person; `verify` closes `NA-01` and `NA-04` only. +- `verify` was written against this repo's `dev/Makefile` and the operator's + dev scripts but has not yet been exercised end to end on a kind cluster from + this tool; its `--dry-run` plan has. +- Multi-chart repos with no pack chart (one chart per upstream component) + score against the first chart that fronts the user-facing app; the structural + problem is reported but not scored per chart. +- `kubeconform` is optional; without it, schema validation is only checked as + a CI step. + +## Validation + +Smoke-tested against `examples/wrap-existing-chart` from the template (renders, +lints, packages, gates correctly; fails the org-level items as expected because +an example is not a pack) and against four vendor packs with very different +shapes: a GitLab-hosted generated chart, a multi-chart Istio/KServe bundle, an +EKS/ALB chart with no NebariApp, and a generated GPU inference pack. + +## Layout + +``` +.claude/agents/software-pack-auditor.md +tools/pack-audit/ + README.md + audit.py scanner + scorer + report/summary + summary_html.py shareable HTML page from several reports + checklist.yaml the rubric as data (with source/example links per item) + reference/ + pack-metadata.schema.json vendored from software-pack-dashboard; refresh on change + template-conventions.md what a healthy pack looks like + reports/ gitignored output +``` + +Possible follow-ups: run `audit.py scan examples/auth-fastapi` in `lint.yaml` +as a self-test; drive `dev/Makefile` or `action-nebari-sandbox` to close the +MANUAL cluster items. diff --git a/tools/pack-audit/README.md b/tools/pack-audit/README.md new file mode 100644 index 0000000..beda51e --- /dev/null +++ b/tools/pack-audit/README.md @@ -0,0 +1,224 @@ +# pack-audit: Nebari software-pack readiness auditor + +Scores a software-pack repository against this repo's +[release-readiness checklist](../../docs/release-readiness-checklist.md) and +lists what blocks the next maturity level. + +Three parts, each optional beyond the first: + +| Part | What it does | Needs | +|---|---|---| +| `audit.py scan` + `checklist.yaml` | Deterministic scan. Renders the chart with `helm`, inspects manifests, greps docs and CI, validates `pack-metadata.yaml`, scores, writes JSON + Markdown. 34 of 58 items. | Python, helm | +| `audit.py judge` | Decides the 9 documentation-quality items with a language model, one request per audit, through any OpenAI-compatible endpoint or any runner that can answer a prompt (GitHub Copilot in Actions, a chat window, an agent). Provider-neutral. | a model, anywhere | +| `.claude/agents/software-pack-auditor.md` | Optional Claude Code agent that drives scan + judge interactively, reads the pack, corrects scanner heuristics with evidence, and writes a narrative fix list. | Claude Code | + +The remaining 15 items need a cluster (`audit.py verify`) or a person. + +Design notes: [docs/superpowers/specs/2026-10-09-pack-audit-design.md](../../docs/superpowers/specs/2026-10-09-pack-audit-design.md). + +## Requirements + +- Python 3.10+ with PyYAML; `jsonschema` for pack-metadata validation. +- `helm` 3.8+ on PATH (OCI dependency support). Without it, render-based + checks are skipped and reported as such. +- Optional: `kubeconform` on PATH, or `pip install kubernetes-validate` (pure Python) as a fallback; without either, schema validation is reported as skipped. +- Network access to fetch chart dependencies (`helm dependency build`). + +## Usage + +Scan a repo (working tree): + +```bash +python3 -I tools/pack-audit/audit.py scan ../my-pack --name my-pack +``` + +Scan a branch without touching the working tree (uses `git archive`): + +```bash +python3 -I tools/pack-audit/audit.py scan ../my-pack \ + --ref origin/feature-branch --name my-pack +``` + +If the chart needs a profile to render, pass it explicitly: + +```bash +python3 -I tools/pack-audit/audit.py scan ../my-pack --name my-pack \ + --values ../my-pack/examples/nebari-values.yaml +``` + +Outputs land in `tools/pack-audit/reports/.json` and `.md` (the +`reports/` directory is gitignored). + +Self-test on this repo's own example: + +```bash +python3 -I tools/pack-audit/audit.py scan examples/auth-fastapi --name auth-fastapi +``` + +Rescore after editing statuses in the JSON (the agent does this for the +judgment items): + +```bash +python3 -I tools/pack-audit/audit.py report tools/pack-audit/reports/my-pack.json +``` + +### Judging the documentation items + +The scanner leaves nine items as JUDGMENT (README purpose, prerequisites, NebariApp +values reference, auth docs, must-set values, metrics, logs, appVersion, image +publishing). `judge` builds one prompt holding each item's expectation, the +scanner's pre-check, and the relevant excerpts (README and docs sections, the +`nebariApp` values block, Chart.yaml, rendered-manifest facts), and asks for a +JSON verdict per item. Roughly 9k input tokens and under 1k output per pack. + +Any OpenAI-compatible endpoint: + +```bash +export PACK_AUDIT_LLM_BASE_URL=https://openrouter.ai/api/v1 # or Ollama, vLLM, LiteLLM, Azure, OpenAI, https://api.anthropic.com/v1 ... +export PACK_AUDIT_LLM_API_KEY=... +export PACK_AUDIT_LLM_MODEL=openai/gpt-4.1 +python3 -I tools/pack-audit/audit.py judge tools/pack-audit/reports/my-pack.json --repo ../my-pack +``` + +Any other runner (paste into a chat, hand to an agent, run in CI): + +```bash +python3 -I tools/pack-audit/audit.py judge tools/pack-audit/reports/my-pack.json --repo ../my-pack --prompt-out judge-prompt.md +# ... answer judge-prompt.md with the model of your choice into answer.txt ... +python3 -I tools/pack-audit/audit.py judge tools/pack-audit/reports/my-pack.json --repo ../my-pack --ingest answer.txt --decided-by gpt-4.1 +``` + +Verdicts are written as `judge(): ` with the scanner's pre-check +kept in `detail`, so a reader can see both. Where the scanner had already set +FAIL or NA, the judge is told to keep it unless the excerpts show what the +scanner missed. Edit any status in the JSON and rerun `report` to override. + +With the optional Claude Code agent: + +``` +Use the software-pack-auditor agent to audit ../my-pack +``` + +## Runs on Nebari + +Next to the score, every report answers "will this run on Nebari?" in three +states, because the score alone does not say so: + +| State | Meaning | +|---|---| +| verified | installed on a cluster with the nebari-operator and the NebariApp reached `Ready` (`audit.py verify`) | +| likely | the NebariApp renders with only CRD fields, explicit `routing`, and a service reference that resolves to a rendered Service (static check NA-07) | +| no | no NebariApp renders, the spec fails the CRD pre-check, or a cluster run failed (with the operator's reason) | + +### Verifying on a cluster + +```bash +python3 -I tools/pack-audit/audit.py verify tools/pack-audit/reports/my-pack.json --dry-run # print the plan +python3 -I tools/pack-audit/audit.py verify tools/pack-audit/reports/my-pack.json # run it +``` + +`verify` creates (or reuses) a local kind cluster through the template's +`dev/Makefile` `cluster` target, which installs MetalLB, Envoy Gateway, +cert-manager, Keycloak, and the nebari-operator (first run takes 5 to 10 +minutes). It then installs the pack with the NebariApp enabled at +`.nebari.local`, waits for the `Ready` condition, requests the hostname +through the gateway (expecting a redirect to Keycloak when auth is enabled), +writes NA-01 and NA-04 back into the results, and uninstalls the release. +Needs `docker`, `kind`, `kubectl`, `helm`, and `make`. Pass `--chart` for +branch audits (their content lives in a temporary export) and `--keep` to +leave the install in place for a look. + +## In CI + +`.github/workflows/pack-audit.yaml` in this repository is a reusable workflow. +A pack repository adds: + +```yaml +permissions: + contents: read + security-events: write # SARIF to code scanning (skipped where not enabled) + pull-requests: write # summary comment on PRs + copilot-requests: write # only when judging with Copilot +jobs: + audit: + uses: nebari-dev/software-pack-template/.github/workflows/pack-audit.yaml@main + with: + set: my-chart.secrets.password=ci-only # whatever the chart needs to render + judge: ${{ github.event_name == 'push' }} # decide judgment items on main only + min-level: alpha # optional gate + secrets: inherit +``` + +The scan job is free and needs no secrets. The judge job is one model request +and runs only when `judge` is true. By default it uses GitHub Copilot CLI with +the workflow token, billed to the organization's AI credits (organization +policy "Allow use of Copilot CLI billed to the organization"; set a budget on +the org). Pass `judge-endpoint` plus the `PACK_AUDIT_LLM_API_KEY` secret and +`judge-model` to use any OpenAI-compatible provider instead. A failed judge +never hides the scan result. + +Gate with `audit.py gate --min-level E|A|B|GA` anywhere else; +it exits 1 and prints the blockers. + +## Exporting findings for other tools + +```bash +python3 -I tools/pack-audit/audit.py export tools/pack-audit/reports/my-pack.json --format sarif # GitHub code scanning, VS Code, security dashboards +python3 -I tools/pack-audit/audit.py export tools/pack-audit/reports/my-pack.json --format junit # GitLab/Jenkins/Azure test reports +python3 -I tools/pack-audit/audit.py export tools/pack-audit/reports/my-pack.json --format csv # spreadsheets +``` + +FAIL and PARTIAL become SARIF `error`/`warning` results (rule id = checklist +id, `helpUri` = the rule link) or JUnit failures grouped by maturity level; +MANUAL, NA, and pending items are SARIF notes or JUnit skips. The JSON stays +the source of truth. + +## Statuses and scoring + +| Status | Meaning | Scored | +|---|---|---| +| PASS | expectation met | 1 | +| PARTIAL | present but incomplete or wrong for the Nebari path | 0.5 | +| FAIL | missing | 0 | +| JUDGMENT | scanner gathered evidence; `judge` (or a person) decides | excluded until decided | +| MANUAL | needs a cluster or a person | excluded | +| NA | does not apply (with reason) | excluded | + +Items are weighted by the level at which they become blockers +(E=4, A=3, B=2, GA=1), so foundational gaps cost more than GA polish. The +**repo-readiness level** is the highest level whose items (and every lower +level's) are all PASS or NA. MANUAL items do not block the repo-readiness +level but are listed as "needs cluster verification"; a pack is not actually +promotable until those are done too. + +## What the scanner checks + +- Layout: chart, NebariApp, README, LICENSE, CODEOWNERS, SECURITY.md, CI, `pack-metadata.yaml` (schema-validated). +- Helm: dependency build, `helm lint`, `helm template` with the NebariApp toggle on and off, `helm package`, optional `kubeconform`. +- NebariApp spec: only CRD fields (operator v0.1.1), explicit `routing`, hostname pattern, `service.name`/`port` resolve to a rendered Service, auth scopes, landing-page limits. +- Workloads: liveness+readiness probes, non-root, `readOnlyRootFilesystem` / `allowPrivilegeEscalation`, resource requests, image pinning, NetworkPolicy, ServiceMonitor, Ingress alongside NebariApp. +- Secrets: literal credentials in default-rendered Secrets or `values.yaml`. +- CI: lint, template both modes, kubeconform, kind install, nebari-operator integration, release/publish, image build, `helm upgrade` smoke. +- Docs: deploy command, Prerequisites, Known Limitations, Troubleshooting, auth, values reference, sizing, upgrade, CHANGELOG, Astro + `@nebari/starlight` docs site. +- Examples: example values that render, Nebari and standalone values, ArgoCD Application with a real `repoURL`. +- Release: git tags (`vX.Y.Z`), chart version shape, `appVersion`. + +Heuristics are deliberately simple and transparent; the judge or a reviewer is +expected to correct them with evidence. See the `observations` block in the JSON for +findings outside the checklist (double ingress, vendored charts, unmerged +branch, generated repo, placeholder READMEs). + +## Modes + +The audit is **read-only on the pack**: the agent and the scanner write only +under `reports/`, and branch content is read through `git archive`, never a +checkout. The agent may offer to apply fixes at the end; it enters fix mode +only after an explicit yes, applies the approved items one at a time, re-runs +the scanner after each, and never commits or pushes unless asked. + +## Comparing several packs + +```bash +python3 -I tools/pack-audit/audit.py summary tools/pack-audit/reports/*.json # -> SUMMARY.md +python3 -I tools/pack-audit/summary_html.py tools/pack-audit/reports/*.json --out readiness.html +``` diff --git a/tools/pack-audit/audit.py b/tools/pack-audit/audit.py new file mode 100644 index 0000000..d20cedc --- /dev/null +++ b/tools/pack-audit/audit.py @@ -0,0 +1,2036 @@ +#!/usr/bin/env python3 +"""Nebari software-pack auditor: deterministic checks. + +Usage: + audit.py scan [--ref ] [--name ] + [--values ...] [--set k=v ...] [--out ] + Inspect a pack repository, run helm/kubeconform where available, and write + /.json (machine-readable) and /.md (human-readable). + Items marked `judgment` in checklist.yaml are left as JUDGMENT for + `audit.py judge` (or a person) to decide. + + audit.py report [--out ] + Re-score a results file (after judge or a person has filled JUDGMENT items + and optionally overridden statuses) and regenerate the markdown report. + + audit.py summary [ ...] [--out SUMMARY.md] + Cross-pack comparison table, shared gaps, and gate blockers per pack. + + audit.py verify [--chart ] [--cluster NAME] [--dry-run] [--keep] + Install the pack on a local kind cluster running the Nebari stack (created + with the template's dev/Makefile), wait for the NebariApp Ready condition, + probe the hostname through the gateway, and write NA-01 / NA-04 back. + Turns "Runs on Nebari: likely" into "verified" or "no" with the operator's reason. + + audit.py export --format sarif|junit|csv [--out FILE] + SARIF 2.1.0 (GitHub code scanning, VS Code), JUnit XML (GitLab/Jenkins test + reports), or CSV, from the same results JSON. + + audit.py judge [--repo DIR] [--prompt-out FILE] [--ingest FILE] + [--endpoint URL --model NAME] [--pending-only] [--dry-run] + Decide the judgment items with a language model, one request per audit. + --prompt-out writes one self-contained prompt for any runner (a CI step, + a chat window, an agent); --ingest reads that runner's answer back. + --endpoint POSTs the same prompt to any OpenAI-compatible + /chat/completions URL (OpenRouter, Ollama, vLLM, LiteLLM, Azure, OpenAI, + Anthropic's compatibility endpoint, ...) using PACK_AUDIT_LLM_API_KEY and + PACK_AUDIT_LLM_MODEL. Verdicts are written with `judge():` evidence + and the scanner's pre-check kept in detail, then rescored. + + audit.py gate --min-level E|A|B|GA [--fail-on-pending] + Exit 1 unless the repo-readiness level is at least --min-level; prints + the blockers. For CI. + +Statuses: PASS, PARTIAL, FAIL, JUDGMENT (pending), MANUAL (needs cluster/person), +NA (does not apply). + +Only the standard library plus PyYAML (and jsonschema when installed) are needed. +helm and kubeconform are used when present on PATH. +""" +from __future__ import annotations + +import argparse +import datetime as dt +import itertools +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +import yaml + +HERE = Path(__file__).resolve().parent +CHECKLIST = HERE / "checklist.yaml" +SCHEMA = HERE / "reference" / "pack-metadata.schema.json" + +AUDIT_HOST = "audit.nebari.example.com" +MUTABLE_TAGS = {"latest", "prod", "production", "main", "master", "dev", "stable", "edge", "nightly", "local"} +NEBARI_VALUE_KEYS = ("nebariapp", "nebariApp", "nebari", "nebari_app", "nebariApp") +WORKLOAD_KINDS = {"Deployment", "StatefulSet", "DaemonSet"} +JOB_KINDS = {"Job", "CronJob"} +PLACEHOLDER_RE = re.compile(r"(<[^>]{1,60}>|REPLACE[_A-Z-]*|CHANGEME|changeme|your-|example\.(com|mil|org)|xxxx|TODO)") + +# NebariApp CRD surface: nebari-operator api/v1/nebariapp_types.go at v0.1.1 / main (2026-10-09). +# The template's docs/nebariapp-crd-reference.md (alpha.19) lacks tls.secretName and landingPage.iconLight/iconDark. +CRD = { + "spec": {"hostname", "service", "routing", "auth", "gateway", "serviceAccountName", "landingPage"}, + "service": {"name", "port", "namespace"}, + "routing": {"routes", "publicRoutes", "tls", "annotations"}, + "route": {"pathPrefix", "pathType"}, + "tls": {"enabled", "secretName"}, + "auth": {"enabled", "provider", "provisionClient", "enforceAtGateway", "forwardAccessToken", "denyRedirect", + "redirectURI", "clientSecretRef", "scopes", "groups", "issuerURL", "spaClient", "deviceFlowClient", + "keycloakConfig", "tokenExchange"}, + "landingPage": {"enabled", "displayName", "description", "icon", "iconLight", "iconDark", "category", "priority", "externalUrl", "healthCheck"}, + "healthCheck": {"enabled", "path", "port", "intervalSeconds", "timeoutSeconds"}, +} +HOSTNAME_RE = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$") +MINIMAL_SCOPES = {"openid", "profile", "email", "groups"} + +# README/doc heading heuristics: item id -> regexes (any match on a heading line). +HEADING_PATTERNS = { + "prerequisites": r"prerequisite|requirements|before you (install|begin)|what we need", + "limitations": r"known (limitation|issue|gap|limit)|limitations|not (yet )?(supported|established)|caveat", + "troubleshooting": r"troubleshoot|diagnos|common (failure|problem|error)|debugging|recovery", + "auth": r"\bauth|keycloak|oidc|\bsso\b|login|identity", + "values": r"values|configuration|configur|settings|parameters|reference", + "sizing": r"sizing|performance|resource (requirements|profiles|limits)|capacity|hardware|throughput|node (types?|groups?)", + "upgrade": r"upgrad|migrat|rollback|rebuild", +} + +# ----------------------------------------------------------------------------- helpers + + +def sh(cmd: list[str], cwd: Path | None = None, timeout: int = 300) -> tuple[int, str, str]: + try: + p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout) + return p.returncode, p.stdout, p.stderr + except FileNotFoundError: + return 127, "", f"{cmd[0]}: not found" + except subprocess.TimeoutExpired: + return 124, "", f"{' '.join(cmd)}: timed out after {timeout}s" + + +def have(tool: str) -> bool: + return shutil.which(tool) is not None + + +def read(p: Path, limit: int = 2_000_000) -> str: + try: + return p.read_text(errors="replace")[:limit] + except Exception: + return "" + + +def load_yaml(p: Path): + try: + return yaml.safe_load(read(p)) + except Exception as e: # noqa: BLE001 + return {"__error__": str(e)} + + +def load_all_yaml(text: str) -> list: + docs = [] + try: + for d in yaml.safe_load_all(text): + if isinstance(d, dict) and d.get("kind"): + docs.append(d) + except Exception as e: # noqa: BLE001 + docs.append({"kind": "__parse_error__", "error": str(e)}) + return docs + + +def rglob(root: Path, patterns: tuple[str, ...], skip=("/.git/", "/node_modules/", "/charts/", "/.tools/")) -> list[Path]: + out = [] + for pat in patterns: + for p in root.rglob(pat): + s = str(p) + if any(k in s for k in skip): + continue + out.append(p) + return sorted(set(out)) + + +def headings(md: str) -> list[str]: + return [ln.lstrip("#").strip() for ln in md.splitlines() if ln.startswith("#")] + + +def has_heading(mds: dict[str, str], key: str) -> list[str]: + rx = re.compile(HEADING_PATTERNS[key], re.I) + hits = [] + for name, text in mds.items(): + for h in headings(text): + if rx.search(h): + hits.append(f"{name}: '{h}'") + break + return hits + + +def deep_get(d, path: str, default=None): + cur = d + for part in path.split("."): + if not isinstance(cur, dict) or part not in cur: + return default + cur = cur[part] + return cur + + +# ----------------------------------------------------------------------------- data model + + +class Result: + def __init__(self): + self.checks: dict[str, dict] = {} + self.observations: list[dict] = [] + self.facts: dict = {} + + def set(self, cid: str, status: str, evidence: str, detail: list[str] | None = None): + self.checks[cid] = {"status": status, "evidence": evidence, "detail": detail or []} + + def observe(self, severity: str, title: str, detail: str = "", where: str = ""): + self.observations.append({"severity": severity, "title": title, "detail": detail, "where": where}) + + +# ----------------------------------------------------------------------------- repo prep + + +def prepare_repo(path: Path, ref: str | None) -> tuple[Path, dict]: + """Return (tree_to_audit, git_info). With --ref, export that ref into a temp dir.""" + info = {"path": str(path), "ref": ref, "is_git": (path / ".git").exists(), "tags": [], "head": None, + "default_branch_files": None, "remote": None} + git_dir = path + if info["is_git"]: + rc, out, _ = sh(["git", "-C", str(git_dir), "tag", "--list"]) + info["tags"] = [t for t in out.split() if t] + info["shallow"] = (path / ".git" / "shallow").exists() + rc, out, _ = sh(["git", "-C", str(git_dir), "rev-parse", "--short", ref or "HEAD"]) + info["head"] = out.strip() or None + rc, out, _ = sh(["git", "-C", str(git_dir), "remote", "get-url", "origin"]) + info["remote"] = out.strip() or None + rc, out, _ = sh(["git", "-C", str(git_dir), "log", "-1", "--format=%cs", ref or "HEAD"]) + info["last_commit_date"] = out.strip() or None + rc, out, _ = sh(["git", "-C", str(git_dir), "ls-tree", "-r", "--name-only", "HEAD"]) + info["default_branch_files"] = len(out.split()) if rc == 0 else None + if ref: + tmp = Path(tempfile.mkdtemp(prefix="pack-audit-")) + rc, out, err = sh(["bash", "-c", f"git -C {git_dir} archive {ref} | tar -x -C {tmp}"]) + if rc != 0: + sys.exit(f"could not export {ref}: {err}") + return tmp, info + return path, info + + +# ----------------------------------------------------------------------------- discovery + + +def find_charts(root: Path) -> list[Path]: + """Top-level charts: Chart.yaml files not nested under another chart's charts/ dir.""" + found = [] + for p in root.rglob("Chart.yaml"): + rel = p.relative_to(root).parts + if ".git" in rel or "node_modules" in rel: + continue + if "charts" in rel[:-2]: # inside a charts/ subdir (vendored dependency) + # keep only if it is the pack's own charts/ layout at the repo root + idx = rel.index("charts") + if idx != 0: + continue + found.append(p.parent) + return sorted(found, key=lambda p: (len(p.relative_to(root).parts), str(p))) + + +def chart_has_nebariapp(chart: Path) -> dict: + tpl = chart / "templates" + hits = {"template": None, "library_dep": False, "values_key": None, "hostname_key": None} + for p in tpl.rglob("*.yaml") if tpl.exists() else []: + t = read(p) + if "kind: NebariApp" in t or "nebari-app.nebariApp" in t: + hits["template"] = str(p.relative_to(chart)) + break + for p in tpl.rglob("*.tpl") if tpl.exists() else []: + if "kind: NebariApp" in read(p): + hits["template"] = hits["template"] or str(p.relative_to(chart)) + cy = load_yaml(chart / "Chart.yaml") or {} + for d in cy.get("dependencies") or []: + if isinstance(d, dict) and d.get("name") == "nebari-app": + hits["library_dep"] = True + vals = load_yaml(chart / "values.yaml") or {} + if isinstance(vals, dict): + for k in vals: + if k.lower().replace("_", "") in ("nebariapp", "nebari") and isinstance(vals[k], dict) and "enabled" in vals[k]: + hits["values_key"] = k + hits["hostname_key"] = "hostname" if "hostname" in vals[k] else None + break + return hits + + +def candidate_values_files(root: Path, chart: Path) -> list[Path]: + cands = [] + for pat in ("values-*.yaml", "values.*.yaml", "ci/*values*.yaml", "ci/*.yaml"): + cands += list(chart.glob(pat)) + for d in ("examples", "values", "config/values", "dev", "ci"): + dd = root / d + if dd.is_dir(): + cands += [p for p in dd.glob("*.yaml") if "values" in p.name or p.parent.name in ("examples", "values")] + # Exclude ArgoCD apps and non-values files. + # Prefer the template's examples/ convention, and Nebari-flavoured names, + # over local/dev profiles so EX-01 is judged on the file installers are told to use. + def rank(p: Path): + parts = {q.name for q in p.parents} + return (0 if "examples" in parts else 1 if "values" in parts else 2, + 0 if "nebari" in p.name else 1, + 1 if "dev" in parts or "local" in p.name else 0, + str(p)) + out = [] + for p in sorted(set(cands), key=rank): + if p.name == "values.yaml" and p.parent == chart: + continue + t = read(p, 4000) + if "kind: Application" in t or "apiVersion:" in t.split("\n", 3)[0]: + continue + out.append(p) + return out + + +# ----------------------------------------------------------------------------- helm + + +class HelmRun: + def __init__(self, root: Path, chart: Path, nk: dict, user_values: list[Path], user_sets: list[str], workdir: Path): + self.root, self.chart, self.nk = root, chart, nk + self.user_values, self.user_sets = user_values, user_sets + self.workdir = workdir + self.log: list[str] = [] + self.dep_ok = None + self.lint = None + self.renders: dict[str, dict] = {} + self.package_ok = None + + def _base(self, release: str) -> list[str]: + return ["helm", "template", release, str(self.chart), "--namespace", "audit", "--include-crds"] + + def deps(self): + if not (self.chart / "Chart.yaml").exists(): + return + cy = load_yaml(self.chart / "Chart.yaml") or {} + if not cy.get("dependencies"): + self.dep_ok = True + return + rc, out, err = sh(["helm", "dependency", "build", str(self.chart)], timeout=600) + if rc != 0: + rc, out, err = sh(["helm", "dependency", "update", str(self.chart)], timeout=600) + self.dep_ok = rc == 0 + self.log.append(f"helm dependency {'ok' if self.dep_ok else 'FAILED: ' + err.strip()[-400:]}") + + def _try(self, label: str, values: list[Path], sets: list[str]) -> dict: + cmd = self._base("audit") + for v in values: + cmd += ["-f", str(v)] + for s in sets: + cmd += ["--set", s] + rc, out, err = sh(cmd, timeout=180) + return {"ok": rc == 0, "cmd": " ".join(cmd), "stdout": out if rc == 0 else "", "stderr": err.strip()[-1500:], + "values": [str(v) for v in values], "sets": sets, "label": label} + + def render(self, label: str, sets: list[str]): + """Render with user-provided values first; otherwise search candidate values files.""" + attempts = [] + if self.user_values or self.user_sets: + r = self._try(label, self.user_values, self.user_sets + sets) + attempts.append(r) + if not attempts or not attempts[-1]["ok"]: + r = self._try(label, [], sets) + attempts.append(r) + if not r["ok"]: + cands = candidate_values_files(self.root, self.chart) + combos = [[c] for c in cands] + [list(c) for c in itertools.permutations(cands, 2)] + for combo in combos[:60]: + r = self._try(label, combo, sets) + attempts.append(r) + if r["ok"]: + break + r["attempts"] = len(attempts) + r["first_error"] = attempts[0]["stderr"] + self.renders[label] = r + self.log.append(f"render {label}: {'ok' if r['ok'] else 'FAILED'} after {len(attempts)} attempt(s)" + + (f" using {[str(Path(v).resolve()).replace(str(self.root.resolve()) + '/', '') for v in r['values']]}" if r["values"] else "")) + return r + + def run_lint(self, values: list[str], sets: list[str]): + cmd = ["helm", "lint", str(self.chart)] + for v in values: + cmd += ["-f", v] + for s in sets: + cmd += ["--set", s] + rc, out, err = sh(cmd, timeout=180) + self.lint = {"ok": rc == 0, "output": (out + err).strip()[-2000:], "cmd": " ".join(cmd)} + self.log.append(f"helm lint: {'ok' if rc == 0 else 'FAILED'}") + + def run_package(self): + rc, out, err = sh(["helm", "package", str(self.chart), "-d", str(self.workdir)], timeout=180) + self.package_ok = rc == 0 + self.log.append(f"helm package: {'ok' if rc == 0 else 'FAILED: ' + err.strip()[-300:]}") + + +# ----------------------------------------------------------------------------- manifest analysis + + +def containers_of(doc: dict) -> list[tuple[str, dict, dict]]: + """Yield (workload-name, pod-spec, container) for workloads and jobs.""" + kind = doc.get("kind") + name = deep_get(doc, "metadata.name", "?") + if kind in WORKLOAD_KINDS or kind == "Job": + pod = deep_get(doc, "spec.template.spec", {}) or {} + elif kind == "CronJob": + pod = deep_get(doc, "spec.jobTemplate.spec.template.spec", {}) or {} + else: + return [] + out = [] + for c in (pod.get("containers") or []) + (pod.get("initContainers") or []): + out.append((f"{kind}/{name}", pod, c)) + return out + + +def analyze_manifests(docs: list[dict]) -> dict: + a = {"kinds": {}, "images": [], "unpinned": [], "mutable": [], "probes_missing": [], "probes_ok": 0, + "containers": 0, "nonroot_missing": [], "hardening_missing": [], "resources_missing": [], + "networkpolicy": [], "monitors": [], "ingress": [], "services": {}, "nebariapps": [], "secrets_literal": [], + "grafana_dashboards": [], "parse_errors": [], "workloads": [], "scrape_annotated": [], "otlp_env": []} + for d in docs: + kind = d.get("kind") + if kind == "__parse_error__": + a["parse_errors"].append(d.get("error")) + continue + a["kinds"][kind] = a["kinds"].get(kind, 0) + 1 + name = deep_get(d, "metadata.name", "?") + if kind == "Service": + ports = [p.get("port") for p in (deep_get(d, "spec.ports") or []) if isinstance(p, dict)] + a["services"][name] = ports + elif kind == "NebariApp": + a["nebariapps"].append(d) + elif kind == "NetworkPolicy": + a["networkpolicy"].append(name) + elif kind in ("ServiceMonitor", "PodMonitor"): + a["monitors"].append(f"{kind}/{name}") + elif kind == "Ingress": + a["ingress"].append(name) + elif kind == "ConfigMap" and (deep_get(d, "metadata.labels.grafana_dashboard") is not None): + a["grafana_dashboards"].append(name) + elif kind == "Secret": + data = {**(d.get("stringData") or {}), **(d.get("data") or {})} + for k, v in data.items(): + if isinstance(v, str) and v and re.search(r"pass|secret|token|key|dsn|url|credential", k, re.I): + a["secrets_literal"].append(f"Secret/{name}:{k}={v[:24]}{'…' if len(v) > 24 else ''}") + if kind in WORKLOAD_KINDS: + a["workloads"].append(f"{kind}/{name}") + ann = deep_get(d, "spec.template.metadata.annotations") or {} + if str(ann.get("prometheus.io/scrape", "")).lower() == "true": + a["scrape_annotated"].append(f"{kind}/{name}") + if kind == "Service" and str(deep_get(d, "metadata.annotations.prometheus.io/scrape", "")).lower() == "true": + a["scrape_annotated"].append(f"Service/{name}") + for wl, pod, c in containers_of(d): + is_init = c in (pod.get("initContainers") or []) + img = str(c.get("image", "")) + if re.search(r"/pause(:|@|$)|image-puller|placeholder", img + " " + str(c.get("name", "")), re.I): + continue # infrastructure filler containers (z2jh image pullers, pause) are not the application + if any(str(e.get("name", "")).startswith("OTEL_EXPORTER_OTLP") for e in (c.get("env") or []) if isinstance(e, dict)): + a["otlp_env"].append(f"{wl}/{c.get('name')}") + a["containers"] += 1 + a["images"].append(img) + if "@sha256:" not in img: + tag = img.rsplit(":", 1)[1] if ":" in img.rsplit("/", 1)[-1] else None + if tag is None: + a["unpinned"].append(f"{wl} {img}") + elif tag in MUTABLE_TAGS: + a["mutable"].append(f"{wl} {img}") + psc, csc = pod.get("securityContext") or {}, c.get("securityContext") or {} + nonroot = csc.get("runAsNonRoot", psc.get("runAsNonRoot")) + uid = csc.get("runAsUser", psc.get("runAsUser")) + if not (nonroot is True or (isinstance(uid, int) and uid > 0)): + a["nonroot_missing"].append(f"{wl}/{c.get('name')}") + if not (csc.get("readOnlyRootFilesystem") is True and csc.get("allowPrivilegeEscalation") is False + and (nonroot is True or (isinstance(uid, int) and uid > 0))): + a["hardening_missing"].append(f"{wl}/{c.get('name')}") + if not deep_get(c, "resources.requests"): + a["resources_missing"].append(f"{wl}/{c.get('name')}") + if d.get("kind") in WORKLOAD_KINDS and not is_init: + if c.get("livenessProbe") and c.get("readinessProbe"): + a["probes_ok"] += 1 + else: + missing = [p for p in ("livenessProbe", "readinessProbe") if not c.get(p)] + a["probes_missing"].append(f"{wl}/{c.get('name')} (no {', '.join(missing)})") + return a + + +def validate_nebariapp(na: dict, services: dict) -> list[str]: + issues = [] + spec = na.get("spec") or {} + + def unknown(obj, allowed, where): + if isinstance(obj, dict): + for k in obj: + if k not in allowed: + issues.append(f"unknown field {where}.{k} (not in the NebariApp CRD at operator v0.1.1)") + + unknown(spec, CRD["spec"], "spec") + host = spec.get("hostname") + if not host: + issues.append("spec.hostname missing") + elif not HOSTNAME_RE.match(str(host)): + issues.append(f"spec.hostname '{host}' violates the CRD hostname pattern") + svc = spec.get("service") or {} + unknown(svc, CRD["service"], "spec.service") + if not svc.get("name"): + issues.append("spec.service.name missing") + elif services and svc["name"] not in services: + issues.append(f"spec.service.name '{svc['name']}' does not match any rendered Service ({', '.join(sorted(services))})") + elif services and svc.get("port") not in (services.get(svc["name"]) or []): + issues.append(f"spec.service.port {svc.get('port')} is not a port on Service {svc['name']} ({services.get(svc['name'])})") + if not isinstance(svc.get("port"), int): + issues.append(f"spec.service.port must be an integer, got {svc.get('port')!r}") + routing = spec.get("routing") + if routing is not None: + unknown(routing, CRD["routing"], "spec.routing") + unknown(routing.get("tls") or {}, CRD["tls"], "spec.routing.tls") + for i, r in enumerate(routing.get("routes") or []): + unknown(r, CRD["route"], f"spec.routing.routes[{i}]") + else: + issues.append("spec.routing omitted: the operator creates no HTTPRoute and treats TLS as disabled (set routing.routes and routing.tls explicitly)") + auth = spec.get("auth") or {} + unknown(auth, CRD["auth"], "spec.auth") + if auth.get("forwardAccessToken") and auth.get("enforceAtGateway") is False: + issues.append("auth.forwardAccessToken requires enforceAtGateway: true") + scopes = set(auth.get("scopes") or ["openid", "profile", "email"]) + if auth.get("enabled") and auth.get("groups") and "groups" not in scopes: + issues.append("auth.groups is set but 'groups' is not in auth.scopes: the operator only attaches the Keycloak groups scope/mapper when it is requested, " + "so the token carries no groups claim and a group-gated SecurityPolicy denies every login (verified on a NIC cluster 2026-10-09); add groups to auth.scopes") + if spec.get("gateway") not in (None, "public", "internal"): + issues.append(f"spec.gateway must be public|internal, got {spec.get('gateway')!r}") + lp = spec.get("landingPage") or {} + unknown(lp, CRD["landingPage"], "spec.landingPage") + unknown(lp.get("healthCheck") or {}, CRD["healthCheck"], "spec.landingPage.healthCheck") + if lp.get("enabled") and not lp.get("displayName"): + issues.append("landingPage.enabled requires displayName") + if lp.get("displayName") and len(str(lp["displayName"])) > 64: + issues.append("landingPage.displayName exceeds 64 chars") + if lp.get("description") and len(str(lp["description"])) > 256: + issues.append("landingPage.description exceeds 256 chars") + return issues + + +# ----------------------------------------------------------------------------- CI analysis + + +def ci_files(root: Path) -> dict[str, str]: + out = {} + for p in list((root / ".github" / "workflows").glob("*.y*ml")) if (root / ".github" / "workflows").is_dir() else []: + out[str(p.relative_to(root))] = read(p) + for name in (".gitlab-ci.yml", ".gitlab-ci.yaml", "Jenkinsfile", ".circleci/config.yml", "azure-pipelines.yml"): + if (root / name).exists(): + out[name] = read(root / name) + for p in (root / ".gitlab").rglob("*.yml") if (root / ".gitlab").is_dir() else []: + out[str(p.relative_to(root))] = read(p) + return out + + +def ci_grep(ci: dict[str, str], pattern: str) -> list[str]: + rx = re.compile(pattern, re.I | re.S) + return [f for f, t in ci.items() if rx.search(t)] + + +# ----------------------------------------------------------------------------- the scan + + +def scan(args) -> dict: + checklist = yaml.safe_load(read(CHECKLIST)) + src = Path(args.repo).resolve() + root, git = prepare_repo(src, args.ref) + name = args.name or src.name + workdir = Path(tempfile.mkdtemp(prefix="pack-audit-work-")) + R = Result() + R.facts["name"] = name + R.facts["git"] = git + R.facts["tools"] = {t: have(t) for t in ("helm", "kubeconform", "kubectl", "check-jsonschema")} + R.facts["scanned_at"] = dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds") + + # ---- inventory + top = sorted(p.name for p in root.iterdir() if p.name != ".git") + R.facts["top_level"] = top + charts = find_charts(root) + kustomizations = rglob(root, ("kustomization.yaml", "kustomization.yml")) + plain_nebariapps = [p for p in rglob(root, ("*.yaml", "*.yml")) if "kind: NebariApp" in read(p, 20000) + and "templates" not in p.parts] + R.facts["charts"] = [str(c.relative_to(root)) or "." for c in charts] + R.facts["kustomizations"] = [str(p.relative_to(root)) for p in kustomizations] + R.facts["plain_nebariapp_manifests"] = [str(p.relative_to(root)) for p in plain_nebariapps] + + # primary chart = chart with a NebariApp template, else the shallowest + primary, nk = None, {} + app_charts = [c for c in charts if (load_yaml(c / "Chart.yaml") or {}).get("type") != "library"] or charts + for c in app_charts: + h = chart_has_nebariapp(c) + if h["template"] or h["library_dep"]: + primary, nk = c, h + break + if primary is None and app_charts: + # no NebariApp anywhere: audit the chart with the most templates (the application, not a helper) + primary = max(app_charts, key=lambda c: (len(list((c / "templates").rglob("*.yaml"))) if (c / "templates").exists() else 0, -len(c.parts), -len(c.name))) + nk = chart_has_nebariapp(primary) + R.facts["primary_chart"] = (str(primary.relative_to(root)) or ".") if primary else None + R.facts["nebariapp_wiring"] = nk + if len(charts) > 1: + R.observe("info", f"{len(charts)} top-level charts found; auditing '{R.facts['primary_chart']}' as the pack chart", + ", ".join(R.facts["charts"])) + + # ---- static files + readme = next((root / n for n in ("README.md", "README.rst", "README") if (root / n).exists()), None) + readme_txt = read(readme) if readme else "" + docs_md = {str(p.relative_to(root)): read(p) for p in rglob(root, ("*.md",)) if "docs/site" not in str(p) + and "node_modules" not in str(p)} + all_md = {"README.md": readme_txt, **docs_md} if readme else docs_md + chart_readme = read(primary / "README.md") if primary and (primary / "README.md").exists() else "" + if chart_readme: + all_md.setdefault(f"{R.facts['primary_chart']}/README.md", chart_readme) + codeowners = next((root / n for n in ("CODEOWNERS", ".github/CODEOWNERS", "docs/CODEOWNERS") if (root / n).exists()), None) + meta_p = root / "pack-metadata.yaml" + meta = load_yaml(meta_p) if meta_p.exists() else None + ci = ci_files(root) + R.facts["ci_files"] = sorted(ci) + license_ = any((root / n).exists() for n in ("LICENSE", "LICENSE.md", "LICENSE.txt", "LICENCE")) + security_md = (root / "SECURITY.md").exists() + changelog = next((root / n for n in ("CHANGELOG.md", "CHANGES.md", "HISTORY.md") if (root / n).exists()), None) + + # ---- OI-01 template layout + missing = [] + if not (charts or kustomizations or plain_nebariapps): + missing.append("no Helm chart / kustomization / manifests") + if not readme: + missing.append("README.md") + if not license_: + missing.append("LICENSE") + if not meta_p.exists(): + missing.append("pack-metadata.yaml") + # CI wiring is not an Experimental-gate requirement (IN-06/IN-07/RE-07 score it; an observation notes its absence). + if not (nk.get("template") or nk.get("library_dep") or plain_nebariapps): + missing.append("NebariApp resource") + layout_ok = primary is not None and (R.facts["primary_chart"] in (".", "chart") or (primary / "templates").exists()) + if not missing and layout_ok: + R.set("OI-01", "PASS", "chart + NebariApp + README + LICENSE + pack-metadata present" + ("" if ci else "; no CI workflow (noted)")) + elif len(missing) <= 2 and (charts or kustomizations): + R.set("OI-01", "PARTIAL", "deployable unit present but template files missing: " + ", ".join(missing)) + else: + R.set("OI-01", "FAIL", "missing: " + ", ".join(missing) if missing else "non-standard layout") + if not security_md: + R.observe("low", "No SECURITY.md (template ships one with a private vulnerability-reporting path)") + if not (root / ".editorconfig").exists(): + R.observe("info", "No .editorconfig (template convention)") + + # ---- OI-02 CODEOWNERS + if codeowners: + owners = [ln for ln in read(codeowners).splitlines() if ln.strip() and not ln.strip().startswith("#") and "@" in ln] + R.set("OI-02", "PASS" if owners else "FAIL", f"{codeowners.relative_to(root)}: {len(owners)} owner line(s)", owners[:5]) + else: + R.set("OI-02", "FAIL", "no CODEOWNERS file") + + # ---- OI-03 / OI-06 / PS-03 pack-metadata + if meta is None: + R.set("OI-03", "FAIL", "pack-metadata.yaml missing at repo root") + R.set("OI-06", "FAIL", "pack-metadata.yaml missing") + R.set("PS-03", "MANUAL", "pre-sales gate (unscored); pack-metadata.yaml missing") + R.set("NA-03", "FAIL", "pack-metadata.yaml missing (nebariapp_integration undeclared)") + else: + errs = [] + try: + import jsonschema # type: ignore + schema = json.loads(read(SCHEMA)) + v = jsonschema.Draft202012Validator(schema, format_checker=jsonschema.FormatChecker()) + errs = [f"{'/'.join(map(str, e.path)) or ''}: {e.message}" for e in v.iter_errors(meta)] + except ImportError: + errs = ["jsonschema module unavailable; schema not validated"] + declared = {k: meta.get(k) for k in ("level", "owner", "nebariapp_integration", "scope", "product_owner", "docs_site")} + R.facts["pack_metadata"] = declared + req_missing = [k for k in ("level", "owner") if not meta.get(k)] + ( + [] if deep_get(meta, "scope.standalone-supported") is not None else ["scope.standalone-supported"]) + if not errs and not req_missing: + R.set("OI-03", "PASS", f"valid; level={meta.get('level')} owner={meta.get('owner')} integration={meta.get('nebariapp_integration')}") + elif errs: + R.set("OI-03", "FAIL", "schema errors: " + "; ".join(errs[:4]), errs) + else: + R.set("OI-03", "PARTIAL", "valid but missing: " + ", ".join(req_missing)) + R.set("OI-06", "PASS" if meta.get("product_owner") else "FAIL", f"product_owner={meta.get('product_owner')!r}") + R.set("PS-03", "MANUAL", "pre-sales gate (unscored); demo_notes " + ("present" if "demo_notes" in meta else "absent")) + + # ---- helm work + helm = None + enabled_docs, disabled_docs, enabled_a, disabled_a = [], [], None, None + if primary and have("helm"): + helm = HelmRun(root, primary, nk, [Path(v).resolve() for v in (args.values or [])], list(args.set or []), workdir) + helm.deps() + vk = nk.get("values_key") + on_sets = [f"{vk}.enabled=true"] + ([f"{vk}.hostname={AUDIT_HOST}"] if nk.get("hostname_key") or vk else []) if vk else [] + off_sets = [f"{vk}.enabled=false"] if vk else [] + r_on = helm.render("nebariapp-enabled", on_sets) + r_off = helm.render("nebariapp-disabled", off_sets) + lint_values = r_on["values"] if r_on["ok"] else (r_off["values"] if r_off["ok"] else []) + helm.run_lint(lint_values, r_on["sets"] if r_on["ok"] else []) + helm.run_package() + enabled_docs = load_all_yaml(r_on["stdout"]) if r_on["ok"] else [] + disabled_docs = load_all_yaml(r_off["stdout"]) if r_off["ok"] else [] + enabled_a = analyze_manifests(enabled_docs) if enabled_docs else None + disabled_a = analyze_manifests(disabled_docs) if disabled_docs else None + R.facts["helm"] = {"log": helm.log, "dep_ok": helm.dep_ok, "lint_ok": helm.lint["ok"] if helm.lint else None, + "package_ok": helm.package_ok, + "renders": {k: {kk: vv for kk, vv in v.items() if kk != "stdout"} for k, v in helm.renders.items()}} + if (workdir / "rendered").exists() or True: + (workdir / "rendered").mkdir(exist_ok=True) + for k, v in helm.renders.items(): + if v["ok"]: + (workdir / "rendered" / f"{k}.yaml").write_text(v["stdout"]) + R.facts["rendered_dir"] = str(workdir / "rendered") + # schema validation: kubeconform binary, else the pure-Python kubernetes-validate package, else skipped + target = workdir / "rendered" / ("nebariapp-enabled.yaml" if r_on["ok"] else "nebariapp-disabled.yaml") + if target.exists() and have("kubeconform"): + rc, out, err = sh(["kubeconform", "-strict", "-ignore-missing-schemas", "-summary", str(target)], timeout=300) + R.facts["kubeconform"] = {"tool": "kubeconform", "ok": rc == 0, "output": (out + err).strip()[-1500:]} + elif target.exists(): + try: + import kubernetes_validate # type: ignore + errs = [] + for d in load_all_yaml(target.read_text()): + if d.get("kind") in ("NebariApp", "__parse_error__") or "." in str(d.get("apiVersion", "")).split("/")[0] and "k8s.io" not in str(d.get("apiVersion")): + continue # CRDs have no bundled schema + try: + kubernetes_validate.validate(d, "1.30", strict=True) + except kubernetes_validate.ValidationError as e: # noqa: PERF203 + errs.append(f"{d.get('kind')}/{deep_get(d, 'metadata.name')}: {str(e)[:160]}") + except kubernetes_validate.SchemaNotFoundError: + pass + R.facts["kubeconform"] = {"tool": "kubernetes-validate", "ok": not errs, "output": "\n".join(errs[-20:])} + except ImportError: + R.facts["kubeconform"] = {"tool": None, "ok": None, "output": "no kubeconform on PATH and kubernetes-validate not installed; schema validation skipped"} + elif primary: + R.observe("warn", "helm not on PATH: render-based checks were skipped") + + analysis = enabled_a or disabled_a + R.facts["analysis_mode"] = "nebariapp-enabled" if enabled_a else ("nebariapp-disabled" if disabled_a else None) + R.facts["analysis"] = None + if analysis: + R.facts["analysis"] = {k: v for k, v in analysis.items() if k != "nebariapps"} + R.facts["analysis"]["nebariapp_specs"] = [n.get("spec") for n in analysis["nebariapps"]] + + # ---- IN-03 lint / IN-04 template + lint_ci = ci_grep(ci, r"helm\s+lint") + if helm and helm.lint: + if helm.lint["ok"]: + R.set("IN-03", "PASS", "helm lint ok" + (f"; CI runs it in {', '.join(lint_ci)}" if lint_ci else "; not wired to CI (noted)")) + else: + R.set("IN-03", "FAIL", "helm lint fails: " + helm.lint["output"][-300:].replace("\n", " | ")) + else: + R.set("IN-03", "PARTIAL" if lint_ci else "FAIL", "could not run helm lint" + ("; CI references it" if lint_ci else "")) + if helm: + on, off = helm.renders["nebariapp-enabled"], helm.renders["nebariapp-disabled"] + na_on = bool(enabled_a and enabled_a["nebariapps"]) + na_off = bool(disabled_a and disabled_a["nebariapps"]) + det = [f"enabled: {'ok' if on['ok'] else 'FAIL ' + on['stderr'][-200:]}", f"disabled: {'ok' if off['ok'] else 'FAIL ' + off['stderr'][-200:]}", + f"NebariApp rendered when enabled: {na_on}; when disabled: {na_off}"] + if on["values"]: + det.append("rendered with values: " + ", ".join(str(Path(v).resolve()).replace(str(root.resolve()) + '/', '') for v in on["values"])) + if not nk.get("values_key"): + R.set("IN-04", "FAIL", "no nebariapp toggle in values.yaml (NebariApp cannot be enabled/disabled)", det) + elif on["ok"] and off["ok"] and na_on and not na_off: + R.set("IN-04", "PASS", "both modes render; NebariApp gated correctly", det) + elif on["ok"] and off["ok"]: + R.set("IN-04", "PARTIAL", "both render but NebariApp gating is wrong", det) + elif on["ok"] or off["ok"]: + R.set("IN-04", "PARTIAL", "only one mode renders", det) + else: + R.set("IN-04", "FAIL", "helm template fails in both modes", det) + if on["values"] or off["values"]: + R.observe("info", "Default values.yaml does not render on its own; a profile values file was needed", + ", ".join(sorted({str(Path(v).resolve()).replace(str(root.resolve()) + '/', '') for v in on["values"] + off["values"]}))) + else: + R.set("IN-04", "FAIL" if primary else "NA", "no helm render performed" if primary else "no Helm chart") + + # ---- IN-05 kubeconform in CI + scripts = {str(p.relative_to(root)): read(p) for d in ("scripts", "ci", "hack", "Makefile", "Taskfile.yml", "justfile") + for p in ((root / d).rglob("*") if (root / d).is_dir() else ([root / d] if (root / d).is_file() else [])) if p.is_file()} + def script_grep(pattern): + rx = re.compile(pattern, re.I | re.S) + return [f for f, t in scripts.items() if rx.search(t)] + kc = ci_grep(ci, r"kubeconform|kubeval|--dry-run=server|pluto|kube-score|datree") + kc_s = script_grep(r"kubeconform|kubeval|--dry-run=server|pluto|kube-score|datree") + kcf = R.facts.get("kubeconform") or {} + ci_note = ("; CI also runs it in " + ", ".join(kc)) if kc else ("; scripts run it: " + ", ".join(kc_s[:2]) if kc_s else "; no CI/script runs it (noted)") + if kcf.get("ok") is True: + R.set("IN-05", "PASS", f"{kcf['tool']}: {kcf['output'].splitlines()[-1] if kcf.get('output') else 'valid'}" + ci_note) + elif kcf.get("ok") is False: + R.set("IN-05", "FAIL", f"{kcf['tool']} reports invalid manifests" + ci_note, kcf.get("output", "").splitlines()[-8:]) + else: + R.set("IN-05", "MANUAL", "no kubeconform/kubernetes-validate available to the auditor; run kubeconform -strict over helm template output" + ci_note) + kcf = R.facts.get("kubeconform") + if kcf and kcf["ok"] is False: + R.observe("warn", f"{kcf['tool']} reports errors on the rendered manifests", kcf["output"][-600:]) + elif kcf and kcf["ok"] is None: + R.observe("info", "Schema validation skipped", kcf["output"]) + + # ---- IN-06 / IN-07 integration & standalone tests + integ = ci_grep(ci, r"nebari-operator/releases|install\.yaml.*nebari-operator|action-nebari-sandbox[\s\S]{0,2000}?(nebariapp|NebariApp)|wait[^\n]*nebariapp|nebariapp[^\n]*Ready") + integ_s = script_grep(r"nebari-operator|action-nebari-sandbox|nebariapp.*ready") + R.set("IN-06", "PASS" if integ else ("PARTIAL" if integ_s else "FAIL"), + ("NIC integration test in " + ", ".join(integ)) if integ else (f"operator integration only in local scripts, not CI: {integ_s[:3]}" if integ_s else "no CI job exercises the nebari-operator / NebariApp Ready condition")) + standalone_declared = str(deep_get(meta or {}, "scope.standalone-supported", "")).lower() in ("yes", "true") + off_ok = bool(helm and helm.renders["nebariapp-disabled"]["ok"]) + declared_no = str(deep_get(meta or {}, "scope.standalone-supported", "")).lower() in ("no", "false") + standalone = False if declared_no else (standalone_declared or off_ok) + R.facts["standalone_supported"] = standalone + st = ci_grep(ci, r"(kind-action|kind create|k3d|minikube).*?(helm install|kubectl apply)|helm install.*--wait") + if not standalone: + R.set("IN-07", "NA", "standalone-supported: no") + else: + R.set("IN-07", "PASS" if st else "FAIL", ("standalone kind install in " + ", ".join(st)) if st else "no CI job installs the chart on a throwaway cluster") + + # ---- NA-01 precheck, NA-03, NA-06 + na_specs = (enabled_a or {}).get("nebariapps", []) if enabled_a else [] + if na_specs: + issues = [] + for n in na_specs: + issues += validate_nebariapp(n, enabled_a["services"]) + spec = na_specs[0].get("spec") or {} + auth_on = bool(deep_get(spec, "auth.enabled")) + R.facts["nebariapp_auth_enabled"] = auth_on + R.facts["nebariapp_enforce_gateway"] = deep_get(spec, "auth.enforceAtGateway", True) + det = [f"hostname={spec.get('hostname')} service={deep_get(spec, 'service.name')}:{deep_get(spec, 'service.port')} " + f"gateway={spec.get('gateway', 'public')} auth={auth_on} enforceAtGateway={deep_get(spec, 'auth.enforceAtGateway', True)} " + f"landingPage={deep_get(spec, 'landingPage.enabled', False)}"] + issues + if issues: + R.set("NA-01", "MANUAL", f"NebariApp renders but pre-check found {len(issues)} issue(s); cluster verification still required", det) + for i in issues: + R.observe("high" if "unknown field" in i or "does not match" in i else "warn", "NebariApp spec issue", i, "NebariApp") + else: + R.set("NA-01", "MANUAL", "NebariApp renders cleanly and the service reference resolves; needs cluster verification", det) + routed = spec.get("routing") is not None + depth = "full" if (auth_on and routed) else "partial" + hard = [i for i in issues if "unknown field" in i or "does not match" in i or "omitted" in i or "missing" in i or "must be" in i or "not in auth.scopes" in i] + if not issues: + R.set("NA-07", "PASS", "NebariApp spec uses only CRD fields, routing is explicit, and service name/port resolve to a rendered Service") + elif hard: + R.set("NA-07", "FAIL", f"{len(hard)} blocking spec issue(s)", hard) + else: + R.set("NA-07", "PARTIAL", f"{len(issues)} advisory issue(s)", issues) + else: + R.set("NA-07", "FAIL", "no NebariApp rendered") + depth = "none" + why = "no NebariApp in the rendered output" if helm else "no NebariApp resource found" + R.set("NA-01", "FAIL", why + "; the pack cannot get a route, TLS, or OIDC from the operator") + R.facts["nebariapp_auth_enabled"] = False + R.facts["detected_integration"] = depth + if meta is not None: + declared_i = meta.get("nebariapp_integration", "na") + R.set("NA-03", "PASS" if declared_i == depth else "FAIL", f"declared={declared_i} detected={depth}") + if analysis and analysis["workloads"]: + if not analysis["probes_missing"]: + R.set("NA-06", "PASS", f"liveness+readiness on all {analysis['probes_ok']} workload container(s)") + elif analysis["probes_ok"]: + R.set("NA-06", "PARTIAL", f"{len(analysis['probes_missing'])} container(s) lack probes", analysis["probes_missing"]) + else: + R.set("NA-06", "FAIL", "no workload container has both probes", analysis["probes_missing"]) + elif analysis: + R.set("NA-06", "NA", "no long-running workloads rendered") + else: + R.set("NA-06", "FAIL", "no render available to inspect") + + # ---- auth-dependent manual/judgment items + auth_on = R.facts.get("nebariapp_auth_enabled", False) + if na_specs and auth_on: + enforce = R.facts.get("nebariapp_enforce_gateway", True) + R.set("NA-04", "MANUAL", f"auth.enabled=true enforceAtGateway={enforce}; verify a 302 to Keycloak for anonymous requests" + + ("" if enforce is not False else " (app-native OAuth: verify the app itself rejects anonymous calls)")) + groups = deep_get(na_specs[0], "spec.auth.groups") or [] + R.set("NA-05", "MANUAL", f"groups={groups or 'none (any realm user)'}; verify with an in-group and an out-of-group user") + scopes = set(deep_get(na_specs[0], "spec.auth.scopes") or ["openid", "profile", "email"]) + extra = scopes - MINIMAL_SCOPES + R.set("SEC-03", "PASS" if not extra else "PARTIAL", f"scopes={sorted(scopes)}" + (f"; non-minimal: {sorted(extra)} need a documented reason" if extra else "")) + elif na_specs: + R.set("NA-04", "NA", "auth.enabled=false in the default render") + R.set("NA-05", "NA", "auth.enabled=false in the default render") + R.set("SEC-03", "NA", "auth disabled") + else: + R.set("NA-04", "FAIL", "no NebariApp; no gateway auth") + R.set("NA-05", "FAIL", "no NebariApp; no gateway auth") + R.set("SEC-03", "NA", "no NebariApp auth block") + + # ---- documentation + deploy_cmd = re.search(r"helm (install|upgrade)|kubectl apply|argocd app|kind: Application", readme_txt) + if readme and len(readme_txt) > 400 and deploy_cmd: + R.set("DOC-01", "PASS", f"README ({len(readme_txt)} chars) with a deploy command ('{deploy_cmd.group(0)}')") + elif readme and len(readme_txt) > 400: + R.set("DOC-01", "PARTIAL", "README present but no helm install / kubectl apply / ArgoCD snippet") + elif readme: + R.set("DOC-01", "FAIL", f"README is a stub ({len(readme_txt)} chars)") + else: + R.set("DOC-01", "FAIL", "no README") + pre, lim = has_heading(all_md, "prerequisites"), has_heading(all_md, "limitations") + pre_readme = has_heading({"README.md": readme_txt}, "prerequisites") + lim_readme = has_heading({"README.md": readme_txt}, "limitations") + if pre_readme and lim_readme: + R.set("DOC-02", "PASS", "README has both sections", pre_readme + lim_readme) + elif pre and lim: + R.set("DOC-02", "PARTIAL", "both sections exist but not in the README itself", pre[:2] + lim[:2]) + elif pre or lim: + R.set("DOC-02", "PARTIAL", "only one of Prerequisites / Known Limitations found", pre[:2] + lim[:2]) + else: + R.set("DOC-02", "FAIL", "neither Prerequisites nor Known Limitations section found") + R.set("IN-02", "JUDGMENT", "auto pre-check: " + ("Prerequisites heading found" if pre else "no Prerequisites heading"), pre[:3]) + prose = [ln for ln in readme_txt.splitlines() if ln.strip() and not ln.lstrip().startswith(("<", "[![", "![", "#"))] + R.set("OI-05", "JUDGMENT", f"README first prose line: {prose[0][:140] if prose else '(none)'}") + auth_doc = has_heading(all_md, "auth") + if na_specs and auth_on or (meta and meta.get("nebariapp_integration") == "full"): + R.set("DOC-03", "JUDGMENT", "auto pre-check: " + (f"auth-related headings: {auth_doc[:3]}" if auth_doc else "no auth heading found")) + else: + R.set("DOC-03", "NA" if not na_specs and not auth_doc else "JUDGMENT", "no NebariApp auth" if not na_specs and not auth_doc else f"headings: {auth_doc[:3]}") + ts = has_heading(all_md, "troubleshooting") + R.set("DOC-04", "PASS" if ts else "FAIL", ("found: " + "; ".join(ts[:3])) if ts else "no Troubleshooting section") + vals_doc = has_heading(all_md, "values") + schema_json = primary and (primary / "values.schema.json").exists() + R.set("DOC-05", "JUDGMENT", "auto pre-check: " + (f"values/config headings: {vals_doc[:3]}" if vals_doc else "no values reference heading") + + ("; values.schema.json present" if schema_json else "")) + sz = has_heading(all_md, "sizing") + R.set("DOC-06", "PASS" if sz else "FAIL", ("found: " + "; ".join(sz[:3])) if sz else "no sizing/performance section") + up = has_heading(all_md, "upgrade") + R.set("DOC-07", "PASS" if up else "FAIL", ("found: " + "; ".join(up[:3])) if up else "no upgrade/rollback section") + if changelog: + entries = len(re.findall(r"^##+\s", read(changelog), re.M)) + R.set("DOC-08", "PASS" if entries else "PARTIAL", f"{changelog.name} with {entries} entries") + else: + R.set("DOC-08", "FAIL", "no CHANGELOG.md") + astro = [p for p in rglob(root, ("astro.config.*",), skip=("/.git/", "/node_modules/"))] + starlight = any("@nebari/starlight" in read(p) or "@nebari/starlight" in read(p.parent / "package.json") for p in astro) + if astro and starlight: + R.set("DOC-09", "PASS", f"Astro + @nebari/starlight at {astro[0].parent.relative_to(root)}") + elif astro: + R.set("DOC-09", "PARTIAL", "Astro docs site present but not on the shared @nebari/starlight theme") + else: + R.set("DOC-09", "FAIL", "no docs site (docs/site with Astro + @nebari/starlight)") + + # ---- examples + ex_dir = next((root / d for d in ("examples", "values", "config/values") if (root / d).is_dir()), None) + example_values = [p for p in (candidate_values_files(root, primary) if primary else [])] + R.facts["example_values"] = [str(p.relative_to(root)) for p in example_values] + rendered_with = set() + if helm: + for v in helm.renders.values(): + rendered_with.update(str(Path(x).resolve()).replace(str(root.resolve()) + '/', '') for x in v["values"]) + clean = [p for p in example_values if not PLACEHOLDER_RE.search(read(p))] + if helm and helm.renders["nebariapp-enabled"]["ok"] and not helm.renders["nebariapp-enabled"]["values"]: + R.set("EX-01", "PASS", "default values.yaml renders with only the hostname set") + elif clean and helm and helm.renders["nebariapp-enabled"]["ok"]: + used = [Path(v) for v in helm.renders["nebariapp-enabled"]["values"]] + local_only = [str(v) for v in used if re.search(r"pullPolicy:\s*Never|LOCAL[- ]ONLY|kind proof|do not use in", read(v), re.I)] + if local_only: + R.set("EX-01", "PARTIAL", "the only example that renders is a local/kind-only profile: " + ", ".join(sorted(rendered_with))) + else: + R.set("EX-01", "PASS", "renders with example values: " + ", ".join(sorted(rendered_with))) + elif example_values: + R.set("EX-01", "PARTIAL", f"{len(example_values)} example values file(s) but they carry placeholders or did not render", + [str(p.relative_to(root)) for p in example_values[:8]]) + else: + R.set("EX-01", "FAIL", "no example values file") + # A Nebari example is one named for it, or (PARTIAL) the file the NebariApp-enabled render needed. + enabled_used = {Path(v).resolve() for v in (helm.renders["nebariapp-enabled"]["values"] if helm and helm.renders.get("nebariapp-enabled") else [])} + neb_ex = [p for p in example_values if re.search(r"nebari|prod|site", p.name, re.I) or p.resolve() in enabled_used] + R.set("EX-02", "PASS" if any("nebari" in p.name for p in neb_ex) else ("PARTIAL" if neb_ex else "FAIL"), + ("Nebari example: " + ", ".join(p.name for p in neb_ex[:3])) if neb_ex else "no nebari-values.yaml example") + if standalone: + st_ex = [p for p in example_values if re.search(r"standalone|kind|local|bootstrap|test", p.name, re.I)] + R.set("EX-03", "PASS" if any("standalone" in p.name for p in st_ex) else ("PARTIAL" if st_ex else "FAIL"), + ("standalone example: " + ", ".join(p.name for p in st_ex[:3])) if st_ex else "no standalone-values.yaml example") + else: + R.set("EX-03", "NA", "standalone not supported") + argo = [p for p in rglob(root, ("*.yaml", "*.yml")) if "kind: Application" in read(p, 20000) and "argoproj.io" in read(p, 20000)] + if argo: + t = read(argo[0]) + m = re.search(r"repoURL:\s*(\S+)", t) + url = m.group(1) if m else "" + placeholder = not url or url.startswith("<") or PLACEHOLDER_RE.search(url) or "YOUR-" in url or "example" in url + R.set("EX-04", "PARTIAL" if placeholder else "PASS", f"{argo[0].relative_to(root)} repoURL={url or '(none)'}" + + (" (placeholder: fill in the published repo/chart)" if placeholder else "")) + else: + R.set("EX-04", "FAIL", "no ArgoCD Application example") + + # ---- telemetry + if analysis: + mons = analysis["monitors"] + scrape = analysis.get("scrape_annotated", []) + otlp = analysis.get("otlp_env", []) + just = any(re.search(r"servicemonitor|podmonitor|prometheus|/metrics|otlp|opentelemetry", t, re.I) for t in all_md.values()) + pod_scrape = [x for x in scrape if not x.startswith("Service/")] + svc_scrape = [x for x in scrape if x.startswith("Service/")] + if pod_scrape or otlp: + R.set("TEL-01", "PASS", "metrics discoverable by NIC's OTel collector: " + (f"prometheus.io/scrape on pods {pod_scrape[:4]}" if pod_scrape else "") + (f" OTLP export env on {otlp[:4]}" if otlp else "")) + elif svc_scrape: + R.set("TEL-01", "PARTIAL", f"prometheus.io/scrape only on {svc_scrape[:3]}; NIC's collector discovers by pod (role: pod), so add the annotation to the pod template") + else: + R.set("TEL-01", "JUDGMENT", (f"ServiceMonitor/PodMonitor rendered ({mons}) but NIC's collector discovers via prometheus.io/scrape pod annotations or OTLP, not monitor CRDs" if mons + else "no prometheus.io/scrape annotations, OTLP env, or monitor CRDs rendered") + + ("; docs mention metrics" if just else "; docs do not mention metrics")) + R.set("TEL-03", "PASS" if analysis["grafana_dashboards"] or rglob(root, ("*dashboard*.json",)) else "FAIL", + f"dashboards: {analysis['grafana_dashboards'] or [str(p.relative_to(root)) for p in rglob(root, ('*dashboard*.json',))][:3]}") + else: + R.set("TEL-01", "JUDGMENT", "no render; inspect templates for ServiceMonitor/PodMonitor") + R.set("TEL-03", "FAIL", "no render / no dashboard found") + logdoc = any(re.search(r"json\s*log|structured\s*log|log(ging)?\s*format|stdout", t, re.I) for t in all_md.values()) + R.set("TEL-02", "JUDGMENT", "docs mention structured/stdout logging" if logdoc else "docs do not mention log format") + + # ---- security + if analysis and analysis["containers"]: + nm = analysis["nonroot_missing"] + R.set("SEC-01", "PASS" if not nm else ("PARTIAL" if len(nm) < analysis["containers"] else "FAIL"), + f"{analysis['containers'] - len(nm)}/{analysis['containers']} containers declare non-root", nm[:12]) + hm = analysis["hardening_missing"] + R.set("SEC-05", "PASS" if not hm else ("PARTIAL" if len(hm) < analysis["containers"] else "FAIL"), + f"{analysis['containers'] - len(hm)}/{analysis['containers']} containers fully hardened", hm[:12]) + bad = analysis["unpinned"] + analysis["mutable"] + # Tags injected on the audit command line (--set image.tag=...) are not repo state: also read the chart defaults. + default_tags = re.findall(r"^[ \t]*tag:[ \t]*[\"']?([^\s\"'#]*)", read(primary / "values.yaml") if primary else "", re.M) + weak_defaults = [t for t in default_tags if t == "" or t in MUTABLE_TAGS or PLACEHOLDER_RE.search(t)] + injected = any(x.split("=")[0].endswith(("tag", ".digest")) for x in (args.set or [])) + if weak_defaults: + bad = bad + [f"values.yaml default tag {t or '(empty)'!r}" for t in weak_defaults] + R.set("SEC-04", "PASS" if not bad else ("PARTIAL" if len(bad) < len(analysis["images"]) + len(weak_defaults) else "FAIL"), + f"{len(analysis['images']) - len(analysis['unpinned']) - len(analysis['mutable'])}/{len(analysis['images'])} rendered images pinned" + + (f"; {len(weak_defaults)} weak default tag(s) in values.yaml" if weak_defaults else "") + + ("; note: a tag was injected via --set for this audit" if injected else ""), bad[:12]) + R.set("SEC-06", "PASS" if analysis["networkpolicy"] else "FAIL", + f"NetworkPolicy: {analysis['networkpolicy'][:4]}" if analysis["networkpolicy"] else "no NetworkPolicy rendered") + if analysis["resources_missing"]: + R.observe("warn", f"{len(analysis['resources_missing'])} container(s) without resource requests", ", ".join(analysis["resources_missing"][:8])) + if analysis["ingress"] and na_specs: + R.observe("warn", "Both an Ingress and a NebariApp render: two front doors for one app", ", ".join(analysis["ingress"])) + elif analysis["ingress"] and not na_specs and R.facts["analysis_mode"] == "nebariapp-enabled": + R.observe("high", "Pack fronts itself with an Ingress instead of a NebariApp; routing/TLS/OIDC bypass the operator", ", ".join(analysis["ingress"])) + elif analysis["ingress"] and not na_specs: + R.observe("warn", "Only the NebariApp-disabled profile rendered, and it uses an Ingress; the Nebari path could not be inspected", ", ".join(analysis["ingress"])) + else: + for cid in ("SEC-01", "SEC-04", "SEC-05", "SEC-06"): + R.set(cid, "FAIL", "no rendered manifests to inspect") + # hardcoded secrets: rendered Secret literals + values.yaml defaults + lit = (analysis or {}).get("secrets_literal", []) if analysis else [] + if enabled_a and disabled_a: + other = set(disabled_a["secrets_literal"] if analysis is enabled_a else enabled_a["secrets_literal"]) + other_keys = {x.split("=")[0] for x in other} + generated = [x for x in lit if x.split("=")[0] in other_keys and x not in other] + if generated: + lit = [x for x in lit if x not in generated] + R.observe("info", f"{len(generated)} Secret value(s) differ between renders: generated at render time (lookup/rand), not literal", + "Under ArgoCD, lookup is empty at render, so these regenerate on every sync unless guarded") + vals_txt = read(primary / "values.yaml") if primary else "" + vd = [] + for m in re.finditer(r"^[ \t]*([A-Za-z0-9_]*(password|secret|token|apikey|api_key|accesskey|secretkey)[A-Za-z0-9_]*):[ \t]*[\"']?([^\s\"'#{]+)", vals_txt, re.I | re.M): + val = m.group(3) + if val.lower() in ("", "~", "null", "false", "true", "{}", "[]") or re.match(r"^[A-Z][A-Z0-9_]+$", val): + continue # empty, or an env-var NAME rather than a value + # *SecretKey / *PasswordKey / *UserKey name the key inside a Secret (bitnami, airflow): the value is a key name, not a credential. + if re.search(r"(?i)(.+secretkey|passwordkey|user(name)?key|tokenkey)$", m.group(1)): + continue + if not m.group(1).lower().endswith(("name", "ref", "key", "file", "existingsecret", "secretname", "mountpath", "path", "secret")) or m.group(1).lower().endswith(("password", "secretkey", "accesskey", "token")): + vd.append(f"values.yaml {m.group(1)}={m.group(3)[:20]}") + if not lit and not vd: + R.set("SEC-02", "PASS", "no literal credentials in rendered Secrets or values.yaml defaults") + elif all("CHANGEME" in x or "changeme" in x or "placeholder" in x.lower() for x in lit + vd): + R.set("SEC-02", "PARTIAL", "placeholder credentials are baked into default Secrets (should fail the render or use existingSecret)", (lit + vd)[:10]) + else: + R.set("SEC-02", "FAIL", f"{len(lit) + len(vd)} literal credential value(s) in defaults", (lit + vd)[:10]) + + # ---- release engineering + rel = ci_grep(ci, r"pack-release\.yaml|helm package|chart-releaser|helm push|oci://.*push|gh release create|helm repo index") + pkg_ok = helm.package_ok if helm else None + if pkg_ok: + R.set("RE-01", "PASS", "helm package ok" + (f"; publish workflow in {', '.join(rel)}" if rel else "; no publish automation (noted)")) + elif pkg_ok is False: + R.set("RE-01", "FAIL", "helm package failed") + else: + R.set("RE-01", "FAIL" if primary else "NA", "helm package not run" if primary else "no chart") + vtags = [t for t in git["tags"] if re.match(r"^v?\d+\.\d+\.\d+", t)] + if git.get("shallow") and not vtags: + R.set("RE-02", "MANUAL", "tags unknown (shallow clone); check the repo's releases" + (f"; release workflow in {', '.join(rel)}" if rel else "")) + elif vtags: + R.set("RE-02", "PASS", f"release tags {vtags[:5]}" + (f"; workflow in {', '.join(rel)}" if rel else "")) + elif not git["is_git"]: + R.set("RE-02", "FAIL", "not a git repository; no release history at all") + else: + R.set("RE-02", "FAIL", "no release tags" + ("; a release workflow exists but has never cut one" if rel else "")) + cy = load_yaml(primary / "Chart.yaml") if primary else {} + cy = cy if isinstance(cy, dict) else {} + R.facts["chart_yaml"] = {k: cy.get(k) for k in ("name", "version", "appVersion", "description", "home", "maintainers", "dependencies")} + R.set("RE-03", "JUDGMENT", f"version={cy.get('version')} appVersion={cy.get('appVersion')!r}; confirm appVersion equals the wrapped app release") + ver = str(cy.get("version", "0")) + ga_ver = re.match(r"^v?([1-9]\d*)\.", ver) is not None + bad_tags = [t for t in git["tags"] if not re.match(r"^v\d+\.\d+\.\d+(-[0-9A-Za-z.]+)?$", t)] + if ga_ver and vtags and not bad_tags: + R.set("RE-04", "PASS", f"chart {ver}; tags {vtags[:3]}") + elif ga_ver or vtags: + R.set("RE-04", "PARTIAL", f"chart {ver}; tags {git['tags'][:5] or 'none'}" + ("; non-conforming tags: " + ", ".join(bad_tags[:3]) if bad_tags else "")) + else: + R.set("RE-04", "FAIL", f"chart {ver}, no release tags") + if ver.startswith("v"): + R.observe("warn", f"Chart version '{ver}' carries a v prefix; Helm SemVer should be bare (tags get the v)") + if not cy.get("appVersion"): + R.observe("warn", "Chart.yaml has no appVersion") + build = ci_grep(ci, r"pack-build-image|docker/build-push-action|docker build|buildx|kaniko|podman build") + dockerfiles = rglob(root, ("Dockerfile", "Containerfile")) + R.set("RE-05", "JUDGMENT", (f"image build workflow in {build}" if build else "no image build workflow") + + (f"; {len(dockerfiles)} Dockerfile(s)" if dockerfiles else "; no Dockerfile (wraps upstream images)")) + R.set("RE-06", "NA", "first-party only (nebari-dev helm-repository sync)") + upg = ci_grep(ci, r"helm upgrade(?!\s+--install)(?!\s+-i\b)") + upg_s = script_grep(r"helm upgrade(?!\s+--install)(?!\s+-i\b)") + R.set("RE-07", "PASS" if upg else ("PARTIAL" if upg_s else "FAIL"), + ("helm upgrade step in " + ", ".join(upg)) if upg else (f"helm upgrade only in scripts, not wired to CI: {upg_s[:3]}" if upg_s else "no install-then-upgrade smoke test in CI")) + + # ---- first-party / manual + R.set("OI-04", "NA", "first-party only; vendor packs register with the platform team's catalog instead") + R.set("IN-01", "MANUAL", "install on a NIC dev cluster from the README alone") + for cid in ("PS-01", "PS-02", "PS-04", "SO-01", "SO-02", "SO-03", "SO-04", "SO-05"): + R.set(cid, "MANUAL", "process gate; not a repository property") + + # ---- NA-02 judgment evidence + if primary and nk.get("values_key"): + vk = nk["values_key"] + block = re.search(rf"^{re.escape(vk)}:\n((?:[ \t]+.*\n?)+)", vals_txt, re.M) + comment_lines = len([ln for ln in (block.group(1).splitlines() if block else []) if ln.strip().startswith("#")]) + key_lines = len([ln for ln in (block.group(1).splitlines() if block else []) if re.match(r"\s+[A-Za-z]", ln) and ":" in ln]) + values_table = any(re.search(r"^\|[^\n]*(\bkey\b|\bvalue\b|\bparameter\b)[^\n]*(default|description)[^\n]*\|", t, re.I | re.M) for t in all_md.values()) + R.set("NA-02", "JUDGMENT", f"values.yaml '{vk}:' block has {key_lines} keys and {comment_lines} comment lines; a values reference table exists in docs: {values_table}") + else: + R.set("NA-02", "FAIL" if primary else "NA", "no nebariapp block in values.yaml") + + # ---- structural observations + if git["is_git"] and args.ref: + R.observe("high", f"Audited ref '{args.ref}'; the default branch holds only {git['default_branch_files']} file(s)", + "Merge the pack to main (or point the platform at the branch) before anyone consumes it") + if (root / "charts").is_dir() and primary and primary.relative_to(root).parts[:1] == ("charts",): + n = len([p for p in (root / "charts").iterdir() if p.is_dir()]) + R.observe("warn", f"Repo vendors {n} charts under charts/ instead of declaring upstream dependencies in one pack chart", + "Template pattern: one chart at chart/ with upstream charts as Chart.yaml dependencies") + if ".gitlab-ci.yml" in ci and not any(k.startswith(".github/") for k in ci): + R.observe("info", "CI is GitLab-only; the nebari-dev reusable workflows (pack-build-image, pack-release) are GitHub Actions", + "Fine for a vendor pack hosted on GitLab, but the equivalent lint/template/kubeconform/release steps must be reproduced") + if not ci: + R.observe("high", "No CI pipeline definition at all (no .github/workflows, no .gitlab-ci.yml)") + if any("GENERATED" in t or "Do not edit" in t for t in (readme_txt,)) or (root / "GENERATED.md").exists(): + R.observe("info", "Repository is generated from another source tree; fixes must land upstream in the generator", + str(root / "GENERATED.md") if (root / "GENERATED.md").exists() else "") + if readme_txt and PLACEHOLDER_RE.search(readme_txt) and "PROPOSAL" in readme_txt: + R.observe("info", "README is marked PROPOSAL") + + # ---- assemble + out = {"name": name, "source": str(src), "ref": args.ref, "facts": R.facts, "checks": R.checks, + "observations": R.observations, "checklist_version": CHECKLIST.name} + out["score"] = score(out, checklist) + return out + + +# ----------------------------------------------------------------------------- scoring + report + +VALUE = {"PASS": 1.0, "PARTIAL": 0.5, "FAIL": 0.0} +LEVEL_ORDER = ["E", "A", "B", "GA"] + + +def score(result: dict, checklist: dict) -> dict: + items = {i["id"]: i for i in checklist["items"]} + weights = {k: v["weight"] for k, v in checklist["levels"].items()} + tot = got = 0.0 + per_level = {lv: {"scorable": 0, "earned": 0.0, "pass": 0, "partial": 0, "fail": 0, "judgment": 0, "manual": 0, "na": 0} for lv in LEVEL_ORDER} + per_cat: dict[str, dict] = {} + for cid, item in items.items(): + if item.get("enabled", True) is False: + continue + st = result["checks"].get(cid, {}).get("status", "JUDGMENT") + lv = item["level"] + cat = item["category"] + pc = per_cat.setdefault(cat, {"scorable": 0, "earned": 0.0, "fail": [], "partial": []}) + pl = per_level[lv] + if st in VALUE: + w = weights[lv] + tot += w + got += w * VALUE[st] + pl["scorable"] += 1 + pl["earned"] += VALUE[st] + pc["scorable"] += 1 + pc["earned"] += VALUE[st] + pl[st.lower()] += 1 + if st == "FAIL": + pc["fail"].append(cid) + elif st == "PARTIAL": + pc["partial"].append(cid) + else: + pl[st.lower() if st.lower() in pl else "manual"] += 1 + achieved = None + blockers: dict[str, list[str]] = {lv: [] for lv in LEVEL_ORDER} + pending: dict[str, list[str]] = {lv: [] for lv in LEVEL_ORDER} + for lv in LEVEL_ORDER: + for cid, item in items.items(): + if item["level"] != lv or item.get("enabled", True) is False: + continue + st = result["checks"].get(cid, {}).get("status", "JUDGMENT") + if st in ("FAIL", "PARTIAL"): + blockers[lv].append(cid) + elif st in ("JUDGMENT", "MANUAL"): + pending[lv].append(cid) + cum_ok = True + for lv in LEVEL_ORDER: + if blockers[lv]: + cum_ok = False + if cum_ok: + achieved = lv + return {"overall_pct": round(100 * got / tot, 1) if tot else None, "weighted_points": round(got, 2), "weighted_total": round(tot, 2), + "runs_on_nebari": runs_on_nebari(result), + "per_level": per_level, "per_category": {k: {**v, "pct": round(100 * v["earned"] / v["scorable"], 0) if v["scorable"] else None} for k, v in per_cat.items()}, + "repo_level": achieved, "blockers": blockers, "pending": pending} + + +def runs_on_nebari(result: dict) -> dict: + """Three-state answer to 'will this run on Nebari?'. + + verified: installed on a cluster with the operator; NebariApp Ready (NA-01 PASS) + likely: NebariApp renders with only CRD fields, explicit routing, resolvable service (NA-07 PASS) + no: no NebariApp, or the spec fails the CRD pre-check + A verified-but-failed cluster run reports 'no' with the operator's reason. + """ + c = result["checks"] + na01 = c.get("NA-01", {}).get("status") + na07 = c.get("NA-07", {}).get("status") + v = result["facts"].get("verify") or {} + if na01 == "PASS": + return {"state": "verified", "reason": c["NA-01"]["evidence"][:200]} + if na01 == "FAIL" and v.get("ran"): + return {"state": "no", "reason": "cluster run failed: " + c["NA-01"]["evidence"][:200]} + if na07 == "PASS": + return {"state": "likely", "reason": "static checks pass (NA-07); not yet installed on a cluster with the operator"} + if na07 == "PARTIAL": + return {"state": "likely", "reason": "advisory NebariApp spec issues (NA-07); not yet verified on a cluster"} + return {"state": "no", "reason": c.get("NA-07", {}).get("evidence", "no NebariApp")[:200]} + + +def level_name(checklist, lv): + return checklist["levels"][lv]["name"] if lv else "none (Experimental gate not met)" + + +def render_report(result: dict, checklist: dict) -> str: + items = {i["id"]: i for i in checklist["items"]} + cats = checklist["categories"] + sc = result["score"] + f = result["facts"] + L = [] + L.append(f"# Software-pack audit: {result['name']}\n") + L.append(f"- Source: `{result['source']}`" + (f" @ `{result['ref']}`" if result.get("ref") else "") + (f" (commit {f['git'].get('head')})" if f["git"].get("head") else "")) + L.append(f"- Scanned: {f['scanned_at']} | tools: " + ", ".join(k for k, v in f["tools"].items() if v) + + (f" | schema validation: {f['kubeconform'].get('tool') or 'skipped'}" if f.get("kubeconform") else "")) + L.append(f"- Primary chart: `{f.get('primary_chart')}` version {deep_get(f, 'chart_yaml.version')} appVersion {deep_get(f, 'chart_yaml.appVersion')!r}") + L.append(f"- NebariApp integration detected: **{f.get('detected_integration')}** | declared: {deep_get(f, 'pack_metadata.nebariapp_integration', '(no pack-metadata.yaml)')} | manifests inspected from the {f.get('analysis_mode') or 'no'} render") + L.append("") + L.append("## Readiness score\n") + ron = sc.get("runs_on_nebari") or runs_on_nebari(result) + L.append(f"**Overall: {sc['overall_pct']}%** (weighted {sc['weighted_points']}/{sc['weighted_total']}) | " + f"**Repo-readiness level: {level_name(checklist, sc['repo_level'])}** | " + f"**Runs on Nebari: {ron['state']}** ({ron['reason']})\n") + L.append("| Level | Scorable | Pass | Partial | Fail | Pending judgment | Manual | Gate |") + L.append("|---|---|---|---|---|---|---|---|") + cum = True + for lv in LEVEL_ORDER: + p = sc["per_level"][lv] + gate = "open" if cum and not sc["blockers"][lv] else ("blocked" if sc["blockers"][lv] else "blocked upstream") + if sc["blockers"][lv]: + cum = False + L.append(f"| {level_name(checklist, lv)} | {p['scorable']} | {p['pass']} | {p['partial']} | {p['fail']} | {p['judgment']} | {p['manual']} | {gate} |") + L.append("") + L.append("| Category | Score | Failing | Partial |") + L.append("|---|---|---|---|") + for cid, c in sc["per_category"].items(): + if c["scorable"]: + L.append(f"| {cats[cid]} | {c['pct']:.0f}% | {', '.join(c['fail']) or '-'} | {', '.join(c['partial']) or '-'} |") + L.append("") + L.append("## Blockers by level\n") + for lv in LEVEL_ORDER: + b = sc["blockers"][lv] + pend = sc["pending"][lv] + if b or pend: + L.append(f"**{level_name(checklist, lv)}**") + for cid in b: + ch = result["checks"][cid] + L.append(f"- [{ch['status']}] {cid} {items[cid]['title']}: {ch['evidence']}") + for cid in pend: + ch = result["checks"][cid] + L.append(f"- [{ch['status']}] {cid} {items[cid]['title']}: {ch['evidence']}") + L.append("") + L.append("## Observations (outside the checklist)\n") + if result["observations"]: + for o in sorted(result["observations"], key=lambda o: {"high": 0, "warn": 1, "low": 2, "info": 3}.get(o["severity"], 4)): + L.append(f"- **{o['severity']}** {o['title']}" + (f": {o['detail']}" if o["detail"] else "")) + else: + L.append("- none") + L.append("") + L.append("## Full checklist\n") + L.append("| ID | Lvl | Status | Item | Evidence | Source |") + L.append("|---|---|---|---|---|---|") + for cid, item in items.items(): + if item.get("enabled", True) is False: + continue + ch = result["checks"].get(cid, {"status": "JUDGMENT", "evidence": ""}) + ev = ch["evidence"].replace("|", "\\|").replace("\n", " ") + src = (f"[rule]({item['source']})" if item.get("source") else "") + (f" · [example]({item['example']})" if item.get("example") else "") + L.append(f"| {cid} | {item['level']} | {ch['status']} | {item['title']} | {ev} | {src} |") + excluded = [cid for cid, it in items.items() if it.get("enabled", True) is False] + if excluded: + L.append("\nExcluded from scoring (disabled in checklist.yaml): " + ", ".join(f"{cid} {items[cid]['title']} [{result['checks'].get(cid, {}).get('status', '-')}]" for cid in excluded)) + L.append("") + L.append("## Evidence details\n") + for cid, ch in result["checks"].items(): + if ch.get("detail"): + L.append(f"- **{cid}**") + for d in ch["detail"][:12]: + L.append(f" - {str(d).replace(chr(10), ' ')}") + if f.get("helm"): + L.append("\n### helm log\n") + for ln in f["helm"]["log"]: + L.append(f"- {ln}") + for k, v in f["helm"]["renders"].items(): + if not v["ok"]: + L.append(f"- {k} first error: `{v['first_error'][-300:]}`") + if f.get("analysis"): + a = f["analysis"] + L.append("\n### rendered inventory\n") + L.append(f"- kinds: {a['kinds']}") + L.append(f"- images: {sorted(set(a['images']))[:20]}") + L.append(f"- services: {a['services']}") + for s in a.get("nebariapp_specs") or []: + L.append(f"- NebariApp spec: `{json.dumps(s)[:900]}`") + return "\n".join(L) + "\n" + + +def render_summary(results: list[dict], checklist: dict) -> str: + items = {i["id"]: i for i in checklist["items"]} + cats = checklist["categories"] + L = ["# Software-pack readiness summary\n", + f"Generated {dt.datetime.now(dt.timezone.utc).isoformat(timespec='seconds')} from {len(results)} audit(s). " + "Scores weight items by the level at which they block promotion (E=4, A=3, B=2, GA=1); " + "the repo-readiness level is the highest level with no FAIL/PARTIAL at or below it. " + "MANUAL (cluster/person) and NA items are excluded.\n"] + L.append("| Pack | Score | Repo-readiness level | Runs on Nebari | Integration | E blockers | A blockers | B blockers | GA blockers | Pending judgment |") + L.append("|---|---|---|---|---|---|---|---|---|---|") + for r in sorted(results, key=lambda r: -(r["score"]["overall_pct"] or 0)): + sc = r["score"] + b = {lv: len(sc["blockers"][lv]) for lv in LEVEL_ORDER} + pend = sum(len([c for c in sc["pending"][lv] if r["checks"].get(c, {}).get("status") == "JUDGMENT"]) for lv in LEVEL_ORDER) + L.append(f"| {r['name']} | {sc['overall_pct']}% | {level_name(checklist, sc['repo_level'])} | {(sc.get('runs_on_nebari') or runs_on_nebari(r))['state']} | {r['facts'].get('detected_integration')} | " + f"{b['E']} | {b['A']} | {b['B']} | {b['GA']} | {pend} |") + L.append("") + L.append("## Category scores\n") + L.append("| Category | " + " | ".join(r["name"] for r in results) + " |") + L.append("|---|" + "---|" * len(results)) + for cid, cname in cats.items(): + row = [] + for r in results: + c = r["score"]["per_category"].get(cid) + row.append(f"{c['pct']:.0f}%" if c and c.get("pct") is not None else "-") + if any(x != "-" for x in row): + L.append(f"| {cname} | " + " | ".join(row) + " |") + L.append("") + L.append("## Shared gaps (failing or partial in every pack)\n") + common = [cid for cid in items if items[cid].get("enabled", True) is not False and all(r["checks"].get(cid, {}).get("status") in ("FAIL", "PARTIAL") for r in results)] + for cid in common: + L.append(f"- {cid} [{items[cid]['level']}] {items[cid]['title']}") + if not common: + L.append("- none") + L.append("") + L.append("## Experimental-gate blockers per pack\n") + for r in results: + L.append(f"**{r['name']}**") + for lv in ("E", "A"): + for cid in r["score"]["blockers"][lv]: + ch = r["checks"][cid] + L.append(f"- [{lv}] [{ch['status']}] {cid} {items[cid]['title']}: {ch['evidence'][:160]}") + L.append("") + L.append("## High-severity observations\n") + for r in results: + hi = [o for o in r["observations"] if o["severity"] in ("high", "warn")] + if hi: + L.append(f"**{r['name']}**") + for o in hi: + L.append(f"- {o['severity']}: {o['title']}" + (f" ({o['detail'][:140]})" if o["detail"] else "")) + L.append("") + return "\n".join(L) + "\n" + + +# ----------------------------------------------------------------------------- verify (kind + operator) + + +def verify(args, checklist: dict) -> dict: + """Install the pack on a local kind cluster that runs the Nebari stack and record NA-01 / NA-04. + + The cluster comes from the template's dev/Makefile `cluster` target (MetalLB, Envoy Gateway, + cert-manager, Keycloak, nebari-operator). Nothing here touches the pack repository. + """ + res = json.loads(read(Path(args.results))) + f = res["facts"] + name = res["name"] + host = args.hostname or f"{re.sub(r'[^a-z0-9-]', '-', name.lower())[:40]}.nebari.local" + ns = args.namespace or f"{re.sub(r'[^a-z0-9-]', '-', name.lower())[:40]}-verify" + release = "verify" + chart = Path(res["source"]) / (f.get("primary_chart") or ".") + if res.get("ref"): + chart = None # branch content lives in a temp export; the caller must pass --chart + if args.chart: + chart = Path(args.chart) + if chart is None or not (chart / "Chart.yaml").exists(): + sys.exit("verify: pass --chart pointing at the pack chart (branch audits export to a temp dir)") + vk = (f.get("nebariapp_wiring") or {}).get("values_key") + if not vk: + sys.exit("verify: this pack has no nebariapp toggle in values.yaml; nothing to verify (runs_on_nebari=no)") + ren = ((f.get("helm") or {}).get("renders") or {}).get("nebariapp-enabled") or {} + values = [v for v in ren.get("values", [])] + list(args.values or []) + sets = [x for x in ren.get("sets", []) if not x.startswith(f"{vk}.hostname=")] + list(args.set or []) + sets = [x for x in sets if not x.startswith(f"{vk}.enabled=")] + [f"{vk}.enabled=true", f"{vk}.hostname={host}"] + dev_dir = Path(args.template_dev or (HERE.parent.parent / "dev")) + cluster = args.cluster + plan = [] + + def step(desc, cmd, timeout=600, check=True): + plan.append((desc, cmd)) + if args.dry_run: + print(f"[dry-run] {desc}\n $ {' '.join(cmd)}") + return 0, "", "" + print(f"==> {desc}") + rc, out, err = sh(cmd, timeout=timeout) + if rc != 0 and check: + print(err[-800:] or out[-800:]) + return rc, out, err + + for tool in ("docker", "kind", "kubectl", "helm", "make"): + if not have(tool) and not args.dry_run: + sys.exit(f"verify: {tool} is required on PATH") + rc, out, _ = step("list kind clusters", ["kind", "get", "clusters"], check=False) + if cluster not in out.split() or args.dry_run: + if not (dev_dir / "Makefile").exists() and not args.dry_run: + sys.exit(f"verify: template dev/Makefile not found at {dev_dir}; pass --template-dev") + step("create the Nebari dev cluster (MetalLB, Envoy Gateway, cert-manager, Keycloak, operator); first run ~5-10 min", + ["make", "-C", str(dev_dir), "cluster", f"CLUSTER_NAME={cluster}"], timeout=1800) + step("use the kind context", ["kubectl", "config", "use-context", f"kind-{cluster}"]) + step("create namespace", ["kubectl", "create", "namespace", ns, "--dry-run=client", "-o", "yaml"], check=False) + if not args.dry_run: + sh(["bash", "-c", f"kubectl create namespace {ns} --dry-run=client -o yaml | kubectl apply -f -"]) + step("opt the namespace in", ["kubectl", "label", "namespace", ns, "nebari.dev/managed=true", "--overwrite"]) + step("fetch chart dependencies", ["helm", "dependency", "build", str(chart)], check=False) + cmd = ["helm", "upgrade", "--install", release, str(chart), "-n", ns, "--wait", "--timeout", args.timeout] + for v in values: + cmd += ["-f", v] + for x in sets: + cmd += ["--set", x] + rc, out, err = step("install the pack with the NebariApp enabled", cmd, timeout=1500) + result = {"ran": not args.dry_run, "cluster": cluster, "namespace": ns, "hostname": host, "install_ok": rc == 0, + "install_error": err[-600:] if rc else "", "conditions": [], "nebariapps": [], "redirect": None} + if not args.dry_run and rc != 0: + result["conditions_note"] = "helm install failed" + rc2, out2, _ = step("wait for the NebariApp Ready condition", + ["kubectl", "wait", "-n", ns, "nebariapp", "--all", "--for=condition=Ready", f"--timeout={args.timeout}"], timeout=900, check=False) + rc3, out3, _ = step("collect NebariApp status", ["kubectl", "get", "nebariapp", "-n", ns, "-o", "json"], check=False) + if not args.dry_run and rc3 == 0: + try: + items = json.loads(out3).get("items", []) + for it in items: + conds = {c["type"]: {"status": c.get("status"), "reason": c.get("reason"), "message": (c.get("message") or "")[:160]} + for c in (it.get("status", {}).get("conditions") or [])} + result["nebariapps"].append({"name": it["metadata"]["name"], "conditions": conds}) + except Exception as e: # noqa: BLE001 + result["conditions_note"] = f"could not parse NebariApp status: {e}" + rc4, ip, _ = step("find the gateway IP", + ["kubectl", "get", "svc", "-n", "envoy-gateway-system", "-l", "gateway.envoyproxy.io/owning-gateway-name=nebari-gateway", + "-o", "jsonpath={.items[0].status.loadBalancer.ingress[0].ip}"], check=False) + ip = ip.strip() + if ip or args.dry_run: + rc5, out5, _ = step("request the hostname through the gateway (expect 302 to Keycloak when auth is enabled, 200 otherwise)", + ["curl", "-k", "-sS", "-o", "/dev/null", "--max-time", "20", "--resolve", f"{host}:443:{ip or ''}", + "-w", "%{http_code} %{redirect_url}", f"https://{host}/"], timeout=60, check=False) + if not args.dry_run and rc5 == 0: + code, _, loc = out5.strip().partition(" ") + result["redirect"] = {"code": code, "location": loc} + if not args.keep and not args.dry_run: + step("uninstall the release", ["helm", "uninstall", release, "-n", ns, "--wait"], check=False) + step("delete the namespace", ["kubectl", "delete", "namespace", ns, "--wait=false"], check=False) + if args.dry_run: + print(f"[dry-run] would write NA-01 / NA-04 into {args.results} and rescore") + return res + # ---- write back + ready = bool(result["nebariapps"]) and all(n["conditions"].get("Ready", {}).get("status") == "True" for n in result["nebariapps"]) + detail = [f"{n['name']}: " + ", ".join(f"{k}={v['status']}" + (f" ({v['reason']})" if v['status'] != 'True' and v.get('reason') else "") for k, v in n["conditions"].items()) for n in result["nebariapps"]] + if ready: + res["checks"]["NA-01"] = {"status": "PASS", "evidence": f"verified on kind cluster '{cluster}' with nebari-operator: NebariApp Ready; conditions " + "; ".join(detail)[:300], "detail": detail} + else: + why = result["install_error"][-200:] if not result["install_ok"] else ("; ".join(detail) or result.get("conditions_note") or "no NebariApp reached Ready") + res["checks"]["NA-01"] = {"status": "FAIL", "evidence": f"cluster run on '{cluster}' did not reach Ready: {why}"[:400], "detail": detail} + auth_on = bool(res["facts"].get("nebariapp_auth_enabled")) + r = result["redirect"] + if r and auth_on: + ok = r["code"].startswith("3") and ("keycloak" in r["location"] or "/realms/" in r["location"] or "openid" in r["location"]) + res["checks"]["NA-04"] = {"status": "PASS" if ok else "FAIL", "evidence": f"anonymous GET https://{host}/ -> {r['code']} {r['location'][:120]}", "detail": []} + elif r: + res["checks"]["NA-04"] = {"status": "NA", "evidence": f"auth disabled; anonymous GET -> {r['code']}", "detail": []} + res["facts"]["verify"] = result + res["score"] = score(res, checklist) + Path(args.results).write_text(json.dumps(res, indent=2, default=str)) + outp = Path(args.results).with_suffix(".md") + md = render_report(res, checklist) + if outp.exists() and "\n# Software-pack audit:" in outp.read_text(): + narrative = outp.read_text().split("\n# Software-pack audit:", 1)[0].rstrip() + md = (narrative + "\n\n" + md) if narrative else md + outp.write_text(md) + print(f"{name}: runs_on_nebari={res['score']['runs_on_nebari']['state']} NA-01={res['checks']['NA-01']['status']} NA-04={res['checks'].get('NA-04', {}).get('status')} -> {outp}") + return res + + +# ----------------------------------------------------------------------------- export (SARIF / JUnit / CSV) + +SARIF_LEVEL = {"FAIL": "error", "PARTIAL": "warning", "JUDGMENT": "note", "MANUAL": "note"} + + +def export(res: dict, checklist: dict, fmt: str) -> str: + items = {i["id"]: i for i in checklist["items"] if i.get("enabled", True) is not False} + cats = checklist["categories"] + name = res["name"] + if fmt == "csv": + import csv, io + buf = io.StringIO(); w = csv.writer(buf) + w.writerow(["pack", "id", "level", "category", "status", "title", "evidence", "source", "example"]) + for cid, it in items.items(): + ch = res["checks"].get(cid, {"status": "JUDGMENT", "evidence": ""}) + w.writerow([name, cid, it["level"], cats[it["category"]], ch["status"], it["title"], ch["evidence"], it.get("source", ""), it.get("example", "")]) + return buf.getvalue() + if fmt == "junit": + import xml.etree.ElementTree as ET + suites = ET.Element("testsuites", name=f"pack-audit {name}") + for lv in LEVEL_ORDER: + lv_items = [(cid, it) for cid, it in items.items() if it["level"] == lv] + suite = ET.SubElement(suites, "testsuite", name=f"{name} {checklist['levels'][lv]['name']}", tests=str(len(lv_items))) + fails = skips = 0 + for cid, it in lv_items: + ch = res["checks"].get(cid, {"status": "JUDGMENT", "evidence": ""}) + tc = ET.SubElement(suite, "testcase", classname=f"{name}.{cats[it['category']]}", name=f"{cid} {it['title']}") + if ch["status"] in ("FAIL", "PARTIAL"): + fails += 1 + fe = ET.SubElement(tc, "failure", message=f"{ch['status']}: {ch['evidence'][:200]}", type=ch["status"]) + fe.text = ch["evidence"] + ("\n" + "\n".join(map(str, ch.get("detail") or [])) if ch.get("detail") else "") + (f"\nrule: {it.get('source', '')}" if it.get("source") else "") + elif ch["status"] in ("MANUAL", "NA", "JUDGMENT"): + skips += 1 + ET.SubElement(tc, "skipped", message=f"{ch['status']}: {ch['evidence'][:200]}") + else: + ET.SubElement(tc, "system-out").text = ch["evidence"] + suite.set("failures", str(fails)); suite.set("skipped", str(skips)) + ET.indent(suites) + return '\n' + ET.tostring(suites, encoding="unicode") + if fmt == "sarif": + rules, results = [], [] + for cid, it in items.items(): + rules.append({"id": cid, "name": cid.replace("-", ""), "shortDescription": {"text": it["title"]}, + "fullDescription": {"text": (it.get("expect") or it["title"]).strip()}, + "helpUri": it.get("source", ""), + "properties": {"level": it["level"], "category": cats[it["category"]], "example": it.get("example", "")}}) + ch = res["checks"].get(cid) + if not ch or ch["status"] not in SARIF_LEVEL: + continue + msg = f"[{ch['status']}] {it['title']}: {ch['evidence']}" + r = {"ruleId": cid, "level": SARIF_LEVEL[ch["status"]], "message": {"text": msg}, + "locations": [{"physicalLocation": {"artifactLocation": {"uri": (res["facts"].get("primary_chart") or ".") + "/Chart.yaml", "uriBaseId": "SRCROOT"}}}], + "properties": {"status": ch["status"], "maturityLevel": it["level"], "detail": [str(x) for x in (ch.get("detail") or [])][:12]}} + results.append(r) + sc = res["score"] + return json.dumps({"$schema": "https://json.schemastore.org/sarif-2.1.0.json", "version": "2.1.0", "runs": [{ + "tool": {"driver": {"name": "pack-audit", "informationUri": "https://github.com/nebari-dev/software-pack-template/tree/main/tools/pack-audit", + "version": "0.1.0", "rules": rules}}, + "originalUriBaseIds": {"SRCROOT": {"uri": "file://" + res["source"].rstrip("/") + "/"}}, + "properties": {"pack": name, "overall_pct": sc["overall_pct"], "repo_level": sc["repo_level"], "runs_on_nebari": sc.get("runs_on_nebari")}, + "results": results}]}, indent=2) + sys.exit(f"unknown format {fmt}") + + +# ----------------------------------------------------------------------------- judge (provider-neutral LLM pass) + +JUDGE_STATUSES = ("PASS", "PARTIAL", "FAIL", "NA") +TEMPLATE_REFERENCE_DOCS = {"release-readiness-checklist.md", "nebariapp-crd-reference.md", "nebariapp-template-adoption.md"} +LOG_LINE_RE = r"stdout|stderr|loki|log[ _-]?(format|level|output|file|dir|path|json)|structured log|json log|logging(?! in)|\blogs? (to|are|go)\b" +LEVEL_ALIASES = {"e": "E", "experimental": "E", "a": "A", "alpha": "A", "b": "B", "beta": "B", "ga": "GA"} +JUDGE_SECTION_PATTERNS = { + "IN-02": HEADING_PATTERNS["prerequisites"], + "NA-02": HEADING_PATTERNS["values"], + "DOC-03": HEADING_PATTERNS["auth"], + "DOC-05": HEADING_PATTERNS["values"], + "TEL-01": r"metric|telemetry|observab|prometheus|otel|opentelemetry|monitoring|scrap", + "TEL-02": r"\blog|logging|loki", + "RE-03": r"version|upstream|release|changelog|compatib", + "RE-05": r"image|registry|build|bundle|offline|air.?gap|publish", +} + +JUDGE_SYSTEM = """You are auditing a Nebari software-pack repository against the nebari-dev +software-pack-template release-readiness checklist. A deterministic scanner has +already decided everything it can; you decide only the items listed below, from +the excerpts given. You have no other access to the repository. + +Statuses: +- PASS: the expectation is met in substance, not merely by a heading with the right name. +- PARTIAL: the material exists but is incomplete, scattered, or wrong for the Nebari + deployment path (NebariApp + nebari-operator, Envoy Gateway, Keycloak OIDC). +- FAIL: missing. +- NA: cannot apply to this pack; the evidence must say why. + +Rules: +- Evidence is one sentence that cites a file path, heading, or value key from the excerpts. +- Do not invent content. If the excerpts do not show it, it is not there: answer FAIL or + PARTIAL and say what is missing. +- Where the scanner already set FAIL or NA, keep that status unless the excerpts contain + evidence the scanner missed, and name it. +- Telemetry: the platform's OpenTelemetry collector discovers metrics through + prometheus.io/scrape pod annotations or OTLP push, not ServiceMonitor CRDs, and ships + every container's stdout to Loki. Plain-text stdout logging passes TEL-02. +- Auth: app-native OIDC with enforceAtGateway false is legitimate when the docs explain + how the app consumes the operator-provisioned client secret. +- Images: PASS only when the repository shows how custom images are built and where they + are published; a pack that only wraps upstream images with no custom image is NA. + +Respond with a JSON array only, no prose and no code fences: +[{"id": "", "status": "PASS|PARTIAL|FAIL|NA", "evidence": ""}] +Include every item id you were given exactly once.""" + + +def md_files(root: Path, limit_files: int = 12) -> dict[str, str]: + """README plus docs/**/*.md, relative name -> text (capped).""" + out: dict[str, str] = {} + for p in [root / "README.md"] + (rglob(root / "docs", ("*.md",)) if (root / "docs").exists() else []): + if p.name in TEMPLATE_REFERENCE_DOCS: # copies of the template's own reference docs say nothing about this pack + continue + if p.exists() and p.is_file(): + out[str(p.relative_to(root))] = read(p, 200_000) + if len(out) >= limit_files: + break + return out + + +def md_sections(text: str, pattern: str, limit: int = 2500, max_sections: int = 2) -> list[str]: + """Sections (heading + body up to the next heading of the same or higher depth) whose heading matches.""" + rx = re.compile(pattern, re.I) + lines = text.splitlines() + out: list[str] = [] + i = 0 + while i < len(lines) and len(out) < max_sections: + m = re.match(r"^(#+)\s*(.*)", lines[i]) + if m and rx.search(m.group(2)): + depth = len(m.group(1)) + j = i + 1 + while j < len(lines): + m2 = re.match(r"^(#+)\s", lines[j]) + if m2 and len(m2.group(1)) <= depth: + break + j += 1 + sec = "\n".join(lines[i:j]).strip() + out.append(sec[:limit] + ("\n[... truncated]" if len(sec) > limit else "")) + i = j + else: + i += 1 + return out + + +def values_block(vals_txt: str, limit: int = 4000) -> str: + """The pack's top-level nebariapp/nebariApp/nebari block from values.yaml, comments included.""" + for m in re.finditer(r"^([A-Za-z_][\w-]*):[ \t]*\n", vals_txt, re.M): + key = m.group(1) + if key.lower().replace("_", "") not in ("nebariapp", "nebari"): + continue + start = m.start() + nxt = re.search(r"^[A-Za-z_][\w-]*:", vals_txt[m.end():], re.M) + end = m.end() + nxt.start() if nxt else len(vals_txt) + block = vals_txt[start:end].rstrip() + return block[:limit] + ("\n# [... truncated]" if len(block) > limit else "") + return "" + + +def grep_lines(text: str, pattern: str, max_lines: int = 30, name: str = "") -> str: + rx = re.compile(pattern, re.I) + hits = [f"{name}:{n}: {ln.strip()}" if name else f"{n}: {ln.strip()}" for n, ln in enumerate(text.splitlines(), 1) if rx.search(ln)] + return "\n".join(hits[:max_lines]) + (f"\n[... {len(hits) - max_lines} more]" if len(hits) > max_lines else "") + + +def judge_context(res: dict, repo: Path, cid: str, mds: dict[str, str]) -> list[tuple[str, str]]: + """(title, excerpt) pairs the judge needs for one checklist item. Sizes are capped so one audit stays one request.""" + f = res["facts"] + a = f.get("analysis") or {} + readme = mds.get("README.md", "") + chart_dir = repo / (f.get("primary_chart") or "chart") + vals_txt = read(chart_dir / "values.yaml", 200_000) + chart_txt = read(chart_dir / "Chart.yaml", 20_000) + parts: list[tuple[str, str]] = [] + + def sections(pattern: str, limit: int = 2500, per_file: int = 2, max_total: int = 4) -> list[str]: + got: list[str] = [] + for name, text in mds.items(): + for s in md_sections(text, pattern, limit, per_file): + got.append(f"--- {name} ---\n{s}") + if len(got) >= max_total: + return got + return got + + pat = JUDGE_SECTION_PATTERNS.get(cid, "") + if cid == "OI-05": + parts.append(("README.md (first 3000 chars)", readme[:3000])) + parts.append(("pack-metadata.yaml description", str(deep_get(f, "pack_metadata.description", "") or ""))) + elif cid == "IN-02": + secs = sections(pat) + parts.append(("Sections whose heading mentions prerequisites/requirements", "\n\n".join(secs))) + if not secs: + parts.append(("README.md (first 2500 chars; no prerequisites heading found)", readme[:2500])) + elif cid == "NA-02": + parts.append((f"{chart_dir.name}/values.yaml NebariApp block", values_block(vals_txt))) + parts.append(("Sections whose heading mentions values/configuration", "\n\n".join(sections(pat, 2000, 1, 3)))) + elif cid == "DOC-03": + parts.append(("Scanner facts", f"nebariapp_auth_enabled={f.get('nebariapp_auth_enabled')} enforceAtGateway={f.get('nebariapp_enforce_gateway')}")) + parts.append(("Sections whose heading mentions auth/keycloak/oidc/sso", "\n\n".join(sections(pat, 3000, 2, 4)))) + elif cid == "DOC-05": + parts.append(("Sections whose heading mentions values/configuration/settings", "\n\n".join(sections(pat, 2500, 2, 4)))) + parts.append((f"{chart_dir.name}/values.yaml (first 2500 chars)", vals_txt[:2500])) + elif cid == "TEL-01": + parts.append(("Rendered manifests (scanner facts)", + f"workloads={a.get('workloads')}\nscrape_annotated={a.get('scrape_annotated')}\notlp_env={a.get('otlp_env')}\n" + f"monitors={a.get('monitors')}\nservice ports={a.get('services')}")) + parts.append(("Sections whose heading mentions metrics/telemetry/observability", "\n\n".join(sections(pat, 2000, 2, 4)))) + parts.append(("values.yaml lines mentioning metrics/prometheus/otel", grep_lines(vals_txt, r"metric|prometheus|otel|otlp|scrape|telemetry", 25))) + elif cid == "TEL-02": + parts.append(("Sections whose heading mentions logs/logging", "\n\n".join(sections(pat, 2000, 2, 3)))) + log_lines = "\n".join(x for x in [grep_lines(t, LOG_LINE_RE, 12, n) for n, t in mds.items()] + [grep_lines(vals_txt, LOG_LINE_RE, 15, "values.yaml")] if x) + parts.append(("Lines mentioning log output across README, docs, and values.yaml", + log_lines or "(none: nothing in README, docs, or values.yaml mentions log files, log directories, log format, or a logging sidecar)")) + elif cid == "RE-03": + parts.append((f"{chart_dir.name}/Chart.yaml", chart_txt)) + lock = read(chart_dir / "Chart.lock", 5000) + if lock: + parts.append((f"{chart_dir.name}/Chart.lock", lock)) + parts.append(("Rendered image references (scanner facts)", "\n".join(sorted(set(a.get("images") or []))))) + parts.append(("README/docs lines mentioning versions", "\n".join(x for x in [grep_lines(t, r"\bv?\d+\.\d+(\.\d+)?\b.*(version|release|upstream|appVersion)|(version|release|upstream|appVersion).*\bv?\d+\.\d+", 10, n) for n, t in mds.items()] if x))) + elif cid == "RE-05": + parts.append(("Scanner facts", f"ci_files={list((f.get('ci_files') or {}).keys()) if isinstance(f.get('ci_files'), dict) else f.get('ci_files')}\n" + f"dockerfiles={[str(p.relative_to(repo)) for p in rglob(repo, ('Dockerfile', '*.Dockerfile', 'Containerfile'))][:10]}\n" + f"images={sorted(set(a.get('images') or []))}")) + parts.append(("Sections whose heading mentions images/registry/build/bundle", "\n\n".join(sections(pat, 2000, 1, 3)))) + else: + parts.append(("README.md (first 2500 chars)", readme[:2500])) + return [(t, b) for t, b in parts if b and b.strip()] + + +def build_judge_prompt(res: dict, checklist: dict, repo: Path, pending_only: bool = False) -> tuple[str, str, list[str]]: + """Return (system, user, item_ids). One prompt covers every judgment item so an audit is one request.""" + items = [i for i in checklist["items"] if i.get("check") == "judgment" and i.get("enabled", True) is not False] + if pending_only: + items = [i for i in items if res["checks"].get(i["id"], {}).get("status", "JUDGMENT") == "JUDGMENT"] + mds = md_files(repo) + levels = checklist["levels"] + L = [f"# Pack: {res['name']}", f"Source: {res['source']}" + (f" @ {res['ref']}" if res.get("ref") else ""), + f"Primary chart: {res['facts'].get('primary_chart')} NebariApp integration detected: {res['facts'].get('detected_integration')}", ""] + heads = [] + for name, text in mds.items(): + hs = headings(text) + if hs: + heads.append(f"{name}: " + " | ".join(hs[:25]) + (" | ..." if len(hs) > 25 else "")) + if heads: + L += ["## Document headings (for orientation)", *heads, ""] + L.append("## Items to decide\n") + for it in items: + cid = it["id"] + ch = res["checks"].get(cid, {"status": "JUDGMENT", "evidence": "", "detail": []}) + L.append(f"### {cid}: {it['title']}") + L.append(f"Maturity level: {levels[it['level']]['name']}. Expectation: {(it.get('expect') or '').strip()}") + L.append(f"Scanner pre-check: {ch['status']}: {ch.get('evidence', '')}") + for d in (ch.get("detail") or [])[:4]: + L.append(f" - {d}") + for title, body in judge_context(res, repo, cid, mds): + L.append(f"\n#### {title}\n```\n{body.strip()}\n```") + L.append("") + L.append("## Answer\nReturn the JSON array now, one entry per item id above: " + ", ".join(i["id"] for i in items) + ".") + return JUDGE_SYSTEM, "\n".join(L), [i["id"] for i in items] + + +def parse_judge_answer(text: str) -> list[dict]: + """Tolerant JSON extraction: whole text, then a fenced block, then the outermost [...] span.""" + t = text.strip() + for cand in (t, (re.search(r"```(?:json)?\s*(.*?)```", t, re.S) or [None, None])[1], t[t.find("["): t.rfind("]") + 1] if "[" in t and "]" in t else None): + if not cand: + continue + try: + data = json.loads(cand) + except json.JSONDecodeError: + continue + if isinstance(data, dict): + data = data.get("items") or data.get("results") or data.get("verdicts") or [data] + if isinstance(data, list): + return [d for d in data if isinstance(d, dict)] + raise ValueError("no JSON array of verdicts found in the judge answer") + + +def apply_judgments(res: dict, verdicts: list[dict], allowed: list[str], decided_by: str) -> tuple[list[tuple[str, str, str]], list[dict]]: + applied, rejected = [], [] + seen = set() + for v in verdicts: + cid = str(v.get("id", "")).strip().upper() + st = str(v.get("status", "")).strip().upper() + ev = " ".join(str(v.get("evidence", "")).split()) + if cid not in allowed or cid in seen or st not in JUDGE_STATUSES or not ev: + rejected.append(v) + continue + seen.add(cid) + old = res["checks"].get(cid, {"status": "JUDGMENT", "evidence": "", "detail": []}) + res["checks"][cid] = {"status": st, "evidence": f"judge({decided_by}): {ev}"[:600], + "detail": [f"scanner: {old['status']}: {old.get('evidence', '')}"[:300]] + [str(d) for d in (old.get("detail") or [])][:3], + "decided_by": decided_by} + applied.append((cid, old["status"], st)) + j = res.setdefault("judge", {}) + j.update({"decided_by": decided_by, "at": dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds"), + "applied": [{"id": c, "from": a, "to": b} for c, a, b in applied], + "missing": sorted(set(allowed) - seen), "rejected": len(rejected)}) + return applied, rejected + + +def llm_chat(base_url: str, api_key: str, model: str, system: str, user: str, max_tokens: int = 4000, timeout: int = 240) -> tuple[str, dict]: + """One OpenAI-style chat completion over plain HTTP. Works with any endpoint that serves /chat/completions.""" + import urllib.error + import urllib.request + url = base_url.rstrip("/") + "/chat/completions" + body = {"model": model, "messages": [{"role": "system", "content": system}, {"role": "user", "content": user}], + "temperature": 0, "max_tokens": max_tokens} + headers = {"Content-Type": "application/json", "Accept": "application/json", "User-Agent": "pack-audit/0.1"} + if api_key: + headers["Authorization"] = f"Bearer {api_key}" + req = urllib.request.Request(url, data=json.dumps(body).encode(), headers=headers, method="POST") + try: + with urllib.request.urlopen(req, timeout=timeout) as r: + data = json.loads(r.read().decode()) + except urllib.error.HTTPError as e: + sys.exit(f"judge endpoint {url} returned HTTP {e.code}: {e.read().decode(errors='replace')[:500]}") + except (urllib.error.URLError, TimeoutError) as e: + sys.exit(f"judge endpoint {url} unreachable: {e}") + try: + content = data["choices"][0]["message"]["content"] + except (KeyError, IndexError, TypeError): + sys.exit(f"judge endpoint returned no choices: {json.dumps(data)[:500]}") + if isinstance(content, list): # some servers return content parts + content = "".join(p.get("text", "") for p in content if isinstance(p, dict)) + return content, data.get("usage") or {} + + +def write_results(res: dict, checklist: dict, results_path: Path, md_path: Path | None = None) -> Path: + """Rescore, rewrite the JSON, and regenerate the markdown while keeping any narrative above the generated report.""" + res["score"] = score(res, checklist) + results_path.write_text(json.dumps(res, indent=2, default=str)) + md = render_report(res, checklist) + outp = md_path or results_path.with_suffix(".md") + if outp.exists() and "\n# Software-pack audit:" in outp.read_text(): + narrative = outp.read_text().split("\n# Software-pack audit:", 1)[0].rstrip() + if narrative: + md = narrative + "\n\n" + md + outp.write_text(md) + return outp + + +def judge(args, checklist: dict) -> None: + results_path = Path(args.results) + res = json.loads(read(results_path)) + repo = Path(args.repo) if args.repo else Path(res["source"]) + if not repo.is_dir(): + sys.exit(f"pack source {repo} is not a directory; pass --repo (branch audits export to a temporary directory that is gone)") + system, user, ids = build_judge_prompt(res, checklist, repo, pending_only=args.pending_only) + if not ids: + print(f"{res['name']}: nothing to judge") + return + approx_tokens = (len(system) + len(user)) // 4 + if args.prompt_out: + Path(args.prompt_out).write_text(system + "\n\n" + user + "\n") + print(f"{res['name']}: judge prompt for {len(ids)} item(s) -> {args.prompt_out} ({len(system) + len(user)} chars, ~{approx_tokens} tokens)") + if not args.ingest and not args.endpoint: + return + answer, usage, decided_by = None, {}, args.decided_by + if args.ingest: + answer = read(Path(args.ingest)) + decided_by = decided_by or "ingest" + elif args.endpoint: + api_key = os.environ.get("PACK_AUDIT_LLM_API_KEY", "") + model = args.model or os.environ.get("PACK_AUDIT_LLM_MODEL", "") + if not model: + sys.exit("judge: --model or PACK_AUDIT_LLM_MODEL is required with --endpoint") + if args.dry_run: + print(f"{res['name']}: would POST ~{approx_tokens} tokens to {args.endpoint.rstrip('/')}/chat/completions model={model} (key {'set' if api_key else 'absent'})") + return + answer, usage = llm_chat(args.endpoint, api_key, model, system, user, max_tokens=args.max_tokens) + decided_by = decided_by or model + else: + sys.exit("judge: give --prompt-out (to run the model elsewhere), --ingest , or --endpoint ") + try: + verdicts = parse_judge_answer(answer) + except ValueError as e: + Path(str(results_path.with_suffix("")) + ".judge-answer.txt").write_text(answer or "") + sys.exit(f"judge: {e}; raw answer kept next to the results JSON") + applied, rejected = apply_judgments(res, verdicts, ids, decided_by) + if usage: + res["judge"]["usage"] = usage + res["judge"]["prompt_chars"] = len(system) + len(user) + outp = write_results(res, checklist, results_path) + sc = res["score"] + for cid, a, b in applied: + print(f" {cid}: {a} -> {b}") + if res["judge"]["missing"]: + print(f" not answered: {', '.join(res['judge']['missing'])}") + if rejected: + print(f" rejected {len(rejected)} malformed verdict(s)") + tok = f" tokens in/out: {usage.get('prompt_tokens', '?')}/{usage.get('completion_tokens', '?')}" if usage else "" + print(f"{res['name']}: {sc['overall_pct']}% repo-level={sc['repo_level']} judged {len(applied)}/{len(ids)} by {decided_by}{tok} -> {outp}") + + +# ----------------------------------------------------------------------------- gate + + +def gate(res: dict, checklist: dict, min_level: str, allow_pending: bool = True) -> tuple[bool, list[str]]: + """Exit-code friendly check: is the repo-readiness level at least min_level?""" + want = LEVEL_ALIASES.get(min_level.strip().lower()) + if not want: + sys.exit(f"gate: unknown level {min_level!r}; use one of E, A, B, GA (or experimental, alpha, beta, ga)") + items = {i["id"]: i for i in checklist["items"]} + sc = res["score"] + achieved = sc["repo_level"] + ok = achieved is not None and LEVEL_ORDER.index(achieved) >= LEVEL_ORDER.index(want) + lines = [f"{res['name']}: repo-readiness level {level_name(checklist, achieved)}; required {level_name(checklist, want)}; score {sc['overall_pct']}%"] + for lv in LEVEL_ORDER[: LEVEL_ORDER.index(want) + 1]: + for cid in sc["blockers"][lv]: + ch = res["checks"][cid] + lines.append(f" BLOCKER {lv} {cid} [{ch['status']}] {items[cid]['title']}: {ch['evidence'][:160]}") + pend = [c for c in sc["pending"][lv] if res["checks"].get(c, {}).get("status", "JUDGMENT") == "JUDGMENT"] + if pend: + lines.append(f" pending judgment at {lv}: {', '.join(pend)}" + ("" if allow_pending else " (counted as blocking)")) + if not allow_pending: + ok = False + lines.append("GATE: " + ("PASS" if ok else "FAIL")) + return ok, lines + + +# ----------------------------------------------------------------------------- CLI + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + sub = ap.add_subparsers(dest="cmd", required=True) + s = sub.add_parser("scan") + s.add_argument("repo") + s.add_argument("--ref") + s.add_argument("--name") + s.add_argument("--values", action="append", help="values file(s) to render with (overrides auto-search)") + s.add_argument("--set", action="append", help="extra --set for helm") + s.add_argument("--out", default=str(HERE / "reports")) + s.add_argument("--carry-from", help="previous results JSON: carry over judgment verdicts and 'override:' statuses") + r = sub.add_parser("report") + r.add_argument("results") + r.add_argument("--out") + m = sub.add_parser("summary", help="cross-pack SUMMARY.md from several results JSON files") + m.add_argument("results", nargs="+") + m.add_argument("--out") + v = sub.add_parser("verify", help="install on a local kind cluster running the Nebari stack; records NA-01/NA-04 (needs docker, kind, kubectl, helm, make)") + v.add_argument("results") + v.add_argument("--chart", help="chart directory (required for --ref audits, otherwise taken from the results)") + v.add_argument("--cluster", default="pack-audit-verify") + v.add_argument("--template-dev", help="path to software-pack-template/dev (default: ../../dev relative to audit.py)") + v.add_argument("--hostname", help="default .nebari.local") + v.add_argument("--namespace") + v.add_argument("--values", action="append") + v.add_argument("--set", action="append") + v.add_argument("--timeout", default="5m") + v.add_argument("--keep", action="store_true", help="leave the release and namespace installed") + v.add_argument("--dry-run", action="store_true", help="print the commands without running them") + e = sub.add_parser("export", help="export a results JSON as sarif, junit, or csv") + e.add_argument("results") + e.add_argument("--format", choices=["sarif", "junit", "csv"], required=True) + e.add_argument("--out") + j = sub.add_parser("judge", help="decide the judgment items with a language model: emit a prompt, ingest an answer, or call an OpenAI-compatible endpoint") + j.add_argument("results") + j.add_argument("--repo", help="pack source directory (default: the results' source path)") + j.add_argument("--prompt-out", help="write the full prompt here for an external runner") + j.add_argument("--ingest", help="answer file produced by an external runner (JSON array, fences tolerated)") + j.add_argument("--endpoint", default=os.environ.get("PACK_AUDIT_LLM_BASE_URL"), help="OpenAI-compatible base URL, e.g. https://openrouter.ai/api/v1 (env PACK_AUDIT_LLM_BASE_URL)") + j.add_argument("--model", help="model name for --endpoint (env PACK_AUDIT_LLM_MODEL)") + j.add_argument("--decided-by", help="label recorded on each verdict (default: the model name, or 'ingest')") + j.add_argument("--pending-only", action="store_true", help="only items still JUDGMENT; default re-decides every judgment-type item, keeping scanner FAIL/NA unless evidence says otherwise") + j.add_argument("--max-tokens", type=int, default=4000) + j.add_argument("--dry-run", action="store_true", help="with --endpoint: print what would be sent, send nothing") + g = sub.add_parser("gate", help="exit 1 unless the repo-readiness level is at least --min-level") + g.add_argument("results") + g.add_argument("--min-level", required=True, help="E|A|B|GA or experimental|alpha|beta|ga") + g.add_argument("--fail-on-pending", action="store_true", help="also fail while judgment items at or below the level are undecided") + args = ap.parse_args() + checklist = yaml.safe_load(read(CHECKLIST)) + if args.cmd == "scan": + res = scan(args) + if args.carry_from: + old = json.loads(read(Path(args.carry_from))) + items = {i["id"]: i for i in checklist["items"]} + carried = 0 + for cid, och in old.get("checks", {}).items(): + is_judgment = items.get(cid, {}).get("check") == "judgment" + if (is_judgment and och["status"] != "JUDGMENT") or "override:" in och.get("evidence", ""): + res["checks"][cid] = {"status": och["status"], "evidence": och["evidence"] if och["evidence"].startswith("carried:") else "carried: " + och["evidence"], "detail": och.get("detail", [])} + carried += 1 + res["carried_from"] = args.carry_from + res["score"] = score(res, checklist) + print(f"carried {carried} agent verdict(s) from {args.carry_from}") + out = Path(args.out) + out.mkdir(parents=True, exist_ok=True) + (out / f"{res['name']}.json").write_text(json.dumps(res, indent=2, default=str)) + (out / f"{res['name']}.md").write_text(render_report(res, checklist)) + sc = res["score"] + print(f"{res['name']}: {sc['overall_pct']}% repo-level={sc['repo_level']} " + f"pending-judgment={sum(len(v) for v in sc['pending'].values())} -> {out / (res['name'] + '.md')}") + elif args.cmd == "verify": + verify(args, checklist) + elif args.cmd == "export": + res = json.loads(read(Path(args.results))) + res["score"] = score(res, checklist) + text = export(res, checklist, args.format) + ext = {"sarif": ".sarif", "junit": ".junit.xml", "csv": ".csv"}[args.format] + outp = Path(args.out) if args.out else Path(args.results).with_suffix(ext) + outp.write_text(text) + print(f"{res['name']}: {args.format} -> {outp}") + elif args.cmd == "judge": + judge(args, checklist) + elif args.cmd == "gate": + res = json.loads(read(Path(args.results))) + res["score"] = score(res, checklist) + ok, lines = gate(res, checklist, args.min_level, allow_pending=not args.fail_on_pending) + print("\n".join(lines)) + sys.exit(0 if ok else 1) + elif args.cmd == "summary": + results = [] + for f in args.results: + res = json.loads(read(Path(f))) + res["score"] = score(res, checklist) + results.append(res) + outp = Path(args.out) if args.out else Path(args.results[0]).parent / "SUMMARY.md" + outp.write_text(render_summary(results, checklist)) + print(f"summary of {len(results)} pack(s) -> {outp}") + else: + res = json.loads(read(Path(args.results))) + res["score"] = score(res, checklist) + Path(args.results).write_text(json.dumps(res, indent=2, default=str)) + md = render_report(res, checklist) + outp = Path(args.out) if args.out else Path(args.results).with_suffix(".md") + if outp.exists() and "\n# Software-pack audit:" in outp.read_text(): + narrative = outp.read_text().split("\n# Software-pack audit:", 1)[0].rstrip() + if narrative: + md = narrative + "\n\n" + md + outp.write_text(md) + print(f"{res['name']}: {res['score']['overall_pct']}% repo-level={res['score']['repo_level']} -> {outp}") + + +if __name__ == "__main__": + main() diff --git a/tools/pack-audit/checklist.yaml b/tools/pack-audit/checklist.yaml new file mode 100644 index 0000000..52b87a9 --- /dev/null +++ b/tools/pack-audit/checklist.yaml @@ -0,0 +1,569 @@ +# Software-pack readiness rubric. +# +# Derived item-for-item from the upstream release-readiness checklist in +# nebari-dev/software-pack-template (docs/release-readiness-checklist.md, vendored +# in reference/). Each item carries: +# +# level E | A | B | GA The maturity level at which the item becomes a +# promotion blocker (Experimental, Alpha, Beta, GA). +# check auto audit.py decides the status from the repository contents. +# judgment audit.py gathers evidence; the auditor agent decides. +# manual needs a cluster, a person, or an org system; never scored. +# applies all | first-party | standalone | auth +# first-party items only apply to packs living in the nebari-dev org; +# for vendor packs they are reported N/A with the adapted expectation. +# +# Scoring (see audit.py): +# weight by level E=4 A=3 B=2 GA=1 +# PASS=1 PARTIAL=0.5 FAIL=0 MANUAL and NA are excluded from the denominator. +# The PS (pre-sales) and SO (sign-off) categories are never scored: every item is +# `manual`. They stay in the report so the promotion PR has the full list. +# Items with `enabled: false` are excluded from scoring and gates entirely and +# listed at the end of the report. Decision 2026-10-09: "runs in CI" is never +# the requirement by itself; what matters is that the check exists and passes. +# Lint, schema validation and packaging are verified by the auditor directly; +# the install/integration/upgrade tests count fully from CI and partially from +# a checked-in script. +# The achieved level is the highest level L such that every scorable item at +# level <= L is PASS or NA. + +levels: + E: { name: Experimental, weight: 4 } + A: { name: Alpha, weight: 3 } + B: { name: Beta, weight: 2 } + GA: { name: GA, weight: 1 } + +categories: + OI: Ownership & Identity + IN: Installation + NA: NebariApp Integration + DOC: Documentation + EX: Examples + TEL: Telemetry + SEC: Security + RE: Release Engineering + PS: Pre-sales Verification + SO: Sign-off + +items: + # ---------------------------------------------------------------- Ownership + - id: OI-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#ownership--identity + example: https://github.com/nebari-dev/software-pack-template/blob/main/README.md#repository-structure + level: E + category: OI + check: auto + title: Repo follows the software-pack template layout + expect: > + A deployable unit (Helm chart, Kustomize overlay, or plain manifests) that + carries a NebariApp resource, plus README, LICENSE, CI workflows, and + pack-metadata.yaml at the repo root. A Helm chart lives at chart/ or at the + repo root. + - id: OI-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#ownership--identity + example: https://github.com/nebari-dev/harbor-pack/blob/main/CODEOWNERS + level: E + category: OI + check: auto + title: CODEOWNERS names at least one accountable engineer + expect: CODEOWNERS (root or .github/) with at least one non-comment owner line. + - id: OI-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#pack-metadata-file + example: https://github.com/nebari-dev/software-pack-dashboard/blob/main/schema/pack-metadata.example.yaml + level: E + category: OI + check: auto + title: pack-metadata.yaml exists, validates against the dashboard schema, and declares level + owner + scope + expect: > + pack-metadata.yaml at the repo root that validates against + reference/pack-metadata.schema.json and sets level, owner, nebariapp_integration + and scope.standalone-supported. + - id: OI-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#ownership--identity + example: https://github.com/nebari-dev/software-pack-dashboard/blob/main/tracked-packs.yaml + level: E + category: OI + check: manual + applies: first-party + title: Pack is listed in nebari-dev/software-pack-dashboard tracked-packs.yaml + expect: > + First-party packs only. Vendor packs instead need an entry in the platform + team's pack catalog / ArgoCD app-of-apps; record where. + - id: OI-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#ownership--identity + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md + level: E + category: OI + check: judgment + title: README explains what the pack does and who it is for + expect: A reader new to the product can tell what it deploys and which users it serves within the first screen. + - id: OI-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#ownership--identity + example: https://github.com/nebari-dev/langfuse-pack/blob/main/pack-metadata.yaml + level: B + category: OI + check: auto + title: product_owner is populated in pack-metadata.yaml + expect: product_owner is a non-null string. + + # ------------------------------------------------------------- Installation + - id: IN-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#quick-start-on-nebari + level: A + category: IN + check: manual + title: Installs cleanly on a fresh current-release NIC dev cluster following only the README + expect: Verified on a cluster; the auditor records whether the README alone is sufficient. + - id: IN-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#prerequisites + level: A + category: IN + check: judgment + title: Prerequisites documented (NIC version, sizing, namespace labels, external dependencies) + expect: A Prerequisites section naming platform version, cluster sizing, the nebari.dev/managed namespace label, storage, GPU, and external services. + - id: IN-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/harbor-pack/blob/main/.github/workflows/lint.yaml + level: B + category: IN + check: auto + title: helm lint passes + expect: helm lint exits 0 (CI wiring is noted, not required). + - id: IN-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/software-pack-template/blob/main/.github/workflows/lint.yaml + level: B + category: IN + check: auto + title: helm template renders with NebariApp enabled and disabled + expect: Both renders succeed; the NebariApp appears only when enabled. + - id: IN-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/harbor-pack/blob/main/.github/workflows/lint.yaml + level: B + category: IN + check: auto + title: Rendered manifests pass Kubernetes schema validation + expect: > + The auditor runs kubeconform -strict (or the pure-Python kubernetes-validate) + over the rendered manifests; they must validate. A CI step that does the same + is noted, not required (decision 2026-10-09). + - id: IN-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/mlflow-pack/blob/main/.github/workflows/test-integration.yaml + level: GA + category: IN + check: auto + title: Integration test against the full NIC stack (CI or repeatable script) + expect: A CI job or a checked-in script that installs nebari-operator (or the Nebari sandbox action) and waits for the NebariApp Ready condition; script-only is PARTIAL. + - id: IN-07 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#installation + example: https://github.com/nebari-dev/harbor-pack/blob/main/.github/workflows/test.yaml + level: GA + category: IN + check: auto + applies: standalone + title: Standalone install test (nebariapp.enabled=false) in CI or a repeatable script + expect: A CI job that installs on kind without the operator and health-checks the service. + + # ------------------------------------------------------ NebariApp integration + - id: NA-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/software-pack-template/blob/main/README.md#verifying-the-deployment + level: A + category: NA + check: manual + title: NebariApp reaches Ready with RoutingReady, TLSReady, AuthReady healthy + expect: > + Cluster verification. audit.py pre-checks that the NebariApp renders, uses + only CRD fields, and that spec.service.name/port resolve to a rendered Service. + - id: NA-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/harbor-pack/blob/main/chart/values.yaml + level: A + category: NA + check: judgment + title: All NebariApp fields the pack exposes are documented in its values reference + expect: Every nebariapp.* value is commented in values.yaml or listed in the README/docs values table. + - id: NA-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/software-pack-dashboard/blob/main/schema/pack-metadata.example.yaml + level: A + category: NA + check: auto + title: nebariapp_integration in pack-metadata.yaml matches the actual integration depth + expect: full when a NebariApp with routing+auth renders; partial when routing only; none when no NebariApp. + - id: NA-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/auth-flow.md + level: B + category: NA + check: manual + applies: auth + title: Auth-protected routes reject unauthenticated requests + expect: Cluster verification. Pre-check that auth.enabled and enforceAtGateway are set or the app-native OAuth path is documented. + - id: NA-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/auth-flow.md + level: B + category: NA + check: manual + applies: auth + title: Auth-protected routes admit authenticated users with the right group membership + expect: Cluster verification. + - id: NA-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#nebariapp-integration + example: https://github.com/nebari-dev/software-pack-template/blob/main/examples/basic-nginx/chart/templates/deployment.yaml + level: B + category: NA + check: auto + title: Health and readiness probes configured + expect: Every long-running workload container has liveness and readiness probes. + + - id: NA-07 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/nebariapp-crd-reference.md + example: https://github.com/nebari-dev/harbor-pack/blob/main/chart/templates/nebariapp.yaml + level: A + category: NA + check: auto + title: NebariApp spec validates against the CRD and resolves to a rendered Service (auditor-added pre-check) + expect: > + Not in the upstream checklist; added because it is the cheapest predictor of + NA-01. Only CRD fields are used, spec.routing is present (omitting it means no + HTTPRoute on operator v0.1.1), spec.service.name/port match a rendered Service, + and the hostname matches the CRD pattern. + + # ------------------------------------------------------------ Documentation + - id: DOC-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md + level: E + category: DOC + check: auto + title: README exists with what the pack does, who it is for, and a deploy command + expect: README.md with a runnable helm install / kubectl apply / ArgoCD snippet. + - id: DOC-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#known-limitations + level: A + category: DOC + check: auto + title: README includes Prerequisites and Known Limitations sections + expect: Both headings present (README or linked docs), with substantive content. + - id: DOC-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#authentication + level: B + category: DOC + check: judgment + applies: auth + title: Authentication setup is documented + expect: How Keycloak/OIDC is wired (gateway-enforced or app-native), which groups gate access, and what the app does with the identity. + - id: DOC-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#troubleshooting + level: B + category: DOC + check: auto + title: Troubleshooting section covers common failure modes + expect: A Troubleshooting heading with concrete symptoms and fixes. + - id: DOC-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/harbor-pack/blob/main/README.md#values-reference-pack-specific + level: B + category: DOC + check: judgment + title: Upstream chart values users must customize are documented + expect: A values reference that calls out the handful of keys every install must set. + - id: DOC-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/langfuse-pack/blob/main/README.md#datastores + level: GA + category: DOC + check: auto + title: Performance and sizing guidance documented + expect: A sizing/resources section with concrete numbers. + - id: DOC-07 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/langfuse-pack/blob/main/docs/src/content/docs/deployment.md + level: GA + category: DOC + check: auto + title: Upgrade path documented + expect: An upgrade/rollback section. + - id: DOC-08 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#documentation + example: https://github.com/nebari-dev/mlflow-pack/blob/main/CHANGELOG.md + level: GA + category: DOC + check: auto + title: CHANGELOG.md exists with notable changes + expect: CHANGELOG.md at the root with at least one versioned entry. + - id: DOC-09 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/pack-docs-requirements.md + example: https://github.com/nebari-dev/langfuse-pack/blob/main/docs/astro.config.mjs + level: GA + category: DOC + check: auto + title: Docs site compliant with pack docs requirements (Astro + @nebari/starlight) + expect: docs/site (or docs/astro) with astro.config using @nebari/starlight and base //. + + # ----------------------------------------------------------------- Examples + - id: EX-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#examples + example: https://github.com/nebari-dev/harbor-pack/blob/main/examples/nebari-values.yaml + level: A + category: EX + check: auto + title: At least one example values file deploys without modification (other than hostname) + expect: An example/values file that renders with only a hostname set; placeholders like <...> or REPLACE_ make it PARTIAL. + - id: EX-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#examples + example: https://github.com/nebari-dev/harbor-pack/blob/main/examples/nebari-values.yaml + level: B + category: EX + check: auto + title: Example values file for a full Nebari deployment (nebari-values.yaml) + expect: examples/nebari-values.yaml (or equivalent) with nebariapp enabled and auth configured. + - id: EX-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#examples + example: https://github.com/nebari-dev/harbor-pack/blob/main/examples/standalone-values.yaml + level: B + category: EX + check: auto + applies: standalone + title: Example values file for standalone deployment (standalone-values.yaml) + expect: examples/standalone-values.yaml (or equivalent) with nebariapp disabled. + - id: EX-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#examples + example: https://github.com/nebari-dev/harbor-pack/blob/main/examples/argocd-application.yaml + level: B + category: EX + check: auto + title: ArgoCD Application example that references the published chart + expect: An argoproj.io Application manifest whose repoURL is real (not a placeholder). + + # ---------------------------------------------------------------- Telemetry + - id: TEL-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#telemetry + example: https://github.com/nebari-dev/lgtm-pack/blob/main/chart/values.yaml + level: B + category: TEL + check: judgment + title: Metrics discoverable by the platform collector, or documented justification for not exposing metrics + expect: > + Upstream wording says ServiceMonitor/PodMonitor, but on NIC the OpenTelemetry + collector DaemonSet discovers targets through `prometheus.io/scrape` / + `prometheus.io/port` pod annotations (role: pod) or accepts OTLP push; monitor + CRDs are not read. PASS when workload pods carry the scrape annotations or + export OTLP; a ServiceMonitor alone is PARTIAL; otherwise a documented reason. + - id: TEL-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#telemetry + example: https://github.com/nebari-dev/lgtm-pack/blob/main/README.md#opentelemetry-collector-wiring + level: B + category: TEL + check: judgment + title: Application logs go to stdout/stderr (structured format preferred) + expect: > + The platform (LGTM pack promtail / OTel collector) ships every container's + stdout to Loki with no pack-side config, so PASS when the app logs to + stdout/stderr. Upstream says "JSON preferred", not required: note plain-text + logging in the evidence but do not deduct. FAIL only when logs go to files or + a sidecar the platform cannot read. + - id: TEL-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#telemetry + example: https://github.com/nebari-dev/lgtm-pack/blob/main/README.md#contributing-dashboards-from-other-software-packs + level: GA + category: TEL + check: auto + title: Example Grafana dashboard for the LGTM stack + expect: > + A ConfigMap in the pack's own namespace labelled `grafana_dashboard: "1"` + (optionally annotated `grafana_folder`); the LGTM pack's sidecar watches all + namespaces and loads it with no coupling. + + # ----------------------------------------------------------------- Security + - id: SEC-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/langfuse-pack/blob/main/chart/values.yaml + level: B + category: SEC + check: auto + title: Containers do not run as root (or justified) + expect: runAsNonRoot true (or runAsUser > 0) on every container; exceptions documented. + - id: SEC-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/harbor-pack/blob/main/chart/templates/stable-secrets-job.yaml + level: B + category: SEC + check: auto + title: No secrets hardcoded in templates or default values + expect: No literal credential values in values.yaml or rendered Secrets; use existingSecret / lookup / required. + - id: SEC-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/nebariapp-crd-reference.md#specauth + level: B + category: SEC + check: auto + applies: auth + title: OIDC scopes minimally scoped + expect: scopes within openid, profile, email, groups; anything else justified. + - id: SEC-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/software-pack-template/blob/main/.github/workflows/release.yaml + level: B + category: SEC + check: auto + title: Container images pinned to a specific tag or digest (never latest) + expect: No image without a tag, and no mutable tags (latest, prod, main, dev, stable). + - id: SEC-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/langfuse-pack/blob/main/chart/values.yaml + level: GA + category: SEC + check: auto + title: securityContext sets readOnlyRootFilesystem, runAsNonRoot, allowPrivilegeEscalation false + expect: All three on every container where possible. + - id: SEC-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#security + example: https://github.com/nebari-dev/data-science-pack/blob/main/templates/hub-nebi-networkpolicy.yaml + level: GA + category: SEC + check: auto + title: NetworkPolicy restricts unnecessary pod-to-pod traffic + expect: At least one NetworkPolicy in the render. + + # ------------------------------------------------------ Release engineering + - id: RE-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#release-engineering + example: https://github.com/nebari-dev/software-pack-template/blob/main/.github/workflows/release.yaml + level: B + category: RE + check: auto + title: Chart packages cleanly (helm package) + expect: > + helm package succeeds. Where the packaged chart is published (nebari-dev + helm-repository for first-party, any reachable OCI/Helm registry for vendors) + is noted but not required. + - id: RE-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#version-tagging-convention + example: https://github.com/nebari-dev/harbor-pack/blob/main/.github/workflows/release.yaml + level: B + category: RE + check: auto + title: At least one pre-1.0 release exists (tag or published chart) + expect: At least one vX.Y.Z tag (or documented published release). Release automation is noted, not required. + - id: RE-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#release-engineering + example: https://github.com/nebari-dev/harbor-pack/blob/main/chart/Chart.yaml + level: B + category: RE + check: judgment + title: appVersion in Chart.yaml matches the upstream application version + expect: appVersion set and equal to the wrapped application's version. + - id: RE-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#version-tagging-convention + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#version-tagging-convention + level: GA + category: RE + check: auto + title: Chart version 1.0.0+ and tags follow the v-prefixed EffVer convention + expect: Chart version >= 1.0.0 and tags like v1.0.0. + - id: RE-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#release-engineering + example: https://github.com/nebari-dev/software-pack-template/blob/main/.github/workflows/build-images.yaml + level: GA + category: RE + check: judgment + title: Custom container images are published and accessible + expect: A build-images workflow (or documented bundle delivery) and a registry path the platform can pull from. + - id: RE-06 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#release-engineering + example: https://github.com/nebari-dev/software-pack-template/blob/main/README.md#release-releaseyaml + level: GA + category: RE + check: manual + applies: first-party + title: Chart syncs to nebari-dev/helm-repository and publishes to quay.io/nebari/charts + expect: First-party only. + - id: RE-07 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#release-engineering + example: https://github.com/nebari-dev/harbor-pack/blob/main/.github/workflows/test.yaml + level: GA + category: RE + check: auto + title: Upgrade smoke test (helm install then helm upgrade) in CI or a repeatable script + expect: A CI step that runs helm upgrade after helm install. + + # ---------------------------------------------------------------- Pre-sales + - id: PS-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#pre-sales-verification + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: A + category: PS + check: manual + title: Pre-sales engineer has run the demo end-to-end + - id: PS-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#pre-sales-verification + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: B + category: PS + check: manual + title: Pre-sales engineer confirms the pack can be demoed without engineering on the call + - id: PS-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#pre-sales-verification + example: https://github.com/nebari-dev/software-pack-dashboard/blob/main/schema/pack-metadata.example.yaml + level: B + category: PS + check: manual + title: demo_notes in pack-metadata.yaml reflects current demo gotchas (or is empty) + expect: > + Pre-sales category is reported but never scored (decision 2026-10-09: it is a + process gate owned by pre-sales, not a repository property). The scanner + still notes whether the field exists. + - id: PS-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#pre-sales-verification + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: GA + category: PS + check: manual + title: Pre-sales engineer verified the demo flow on the GA release commit + + # ----------------------------------------------------------------- Sign-off + - id: SO-01 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#sign-off + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: A + category: SO + check: manual + title: Pack owner approves the promotion PR + - id: SO-02 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#sign-off + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: A + category: SO + check: manual + title: Pre-sales rep approves the promotion PR + - id: SO-03 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#sign-off + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: B + category: SO + check: manual + title: Tech lead approves the promotion PR + - id: SO-04 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#sign-off + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: GA + category: SO + check: manual + title: Product owner has documented and verified acceptance criteria + - id: SO-05 + source: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#sign-off + example: https://github.com/nebari-dev/software-pack-template/blob/main/docs/release-readiness-checklist.md#promotion-process + level: GA + category: SO + check: manual + title: Product owner approves the promotion PR diff --git a/tools/pack-audit/reference/pack-metadata.schema.json b/tools/pack-audit/reference/pack-metadata.schema.json new file mode 100644 index 0000000..39a5fc8 --- /dev/null +++ b/tools/pack-audit/reference/pack-metadata.schema.json @@ -0,0 +1,123 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/nebari-dev/pack-dashboard/blob/main/schema/pack-metadata.schema.json", + "title": "Nebari pack metadata", + "description": "Schema for the pack-metadata.yaml file placed at the root of each first-party Nebari pack repository.", + "type": "object", + "required": [ + "name", + "display_name", + "level", + "owner", + "deprecated" + ], + "additionalProperties": false, + "properties": { + "name": { + "type": "string", + "minLength": 1, + "description": "Must equal the GitHub repo name." + }, + "display_name": { + "type": "string", + "minLength": 1 + }, + "description": { + "type": "string", + "minLength": 1, + "maxLength": 200 + }, + "level": { + "type": "string", + "enum": ["experimental", "alpha", "beta", "ga"] + }, + "owner": { + "type": "string", + "minLength": 1, + "pattern": "^[^@].*$", + "description": "GitHub username, no @ prefix." + }, + "deprecated": { + "type": "boolean" + }, + "docs_site": { + "type": "boolean", + "description": "If true, this pack has a docs site routed at packs.nebari.dev//. Default false." + }, + "sunset_date": { + "type": ["string", "null"], + "format": "date", + "pattern": "^\\d{4}-\\d{2}-\\d{2}$" + }, + "product_owner": { + "type": ["string", "null"] + }, + "nebariapp_integration": { + "type": "string", + "enum": ["none", "partial", "full", "na"], + "default": "na" + }, + "scope": { + "type": "object", + "additionalProperties": false, + "properties": { + "standalone-supported": { + "type": ["boolean", "string"], + "enum": [true, false, "yes", "no"] + } + } + }, + "last_promoted_at": { + "type": "string", + "pattern": "^\\d{4}-\\d{2}-\\d{2}$" + }, + "last_promoted_pr": { + "type": "integer", + "minimum": 1 + }, + "demo_notes": { + "type": "string", + "maxLength": 500 + }, + "links": { + "type": "object", + "additionalProperties": false, + "properties": { + "docs": { + "type": "string", + "format": "uri" + }, + "demo": { + "type": "string", + "format": "uri" + } + } + } + }, + "allOf": [ + { + "if": { + "properties": { "deprecated": { "const": true } }, + "required": ["deprecated"] + }, + "then": { + "required": ["sunset_date"], + "properties": { + "sunset_date": { "type": "string" } + } + } + }, + { + "if": { + "properties": { "level": { "const": "ga" } }, + "required": ["level"] + }, + "then": { + "required": ["product_owner"], + "properties": { + "product_owner": { "type": "string", "minLength": 1 } + } + } + } + ] +} diff --git a/tools/pack-audit/reference/template-conventions.md b/tools/pack-audit/reference/template-conventions.md new file mode 100644 index 0000000..5b8fc24 --- /dev/null +++ b/tools/pack-audit/reference/template-conventions.md @@ -0,0 +1,134 @@ +# What a healthy Nebari software pack looks like + +Distilled on 2026-10-09 from `nebari-dev/software-pack-template` (the contract) +and the first-party packs that follow it (llm-serving, mlflow, superset, harbor, +langfuse, data-science, lgtm). Use this as the mental model when judging a +pack; the scored rubric is `checklist.yaml`. + +## The one non-negotiable + +A pack is a Kubernetes deployment that carries a `NebariApp` +(`reconcilers.nebari.dev/v1`). The nebari-operator reads it and creates the +HTTPRoute on the shared Envoy Gateway, the cert-manager Certificate, and (if +`auth.enabled`) the Keycloak client plus the Envoy SecurityPolicy. Anything +that fronts itself with its own Ingress, gateway, or identity provider is not +integrated, however good the chart is. + +Minimum useful spec (operator v0.1.1): + +```yaml +spec: + hostname: app.nebari.example.com # required, DNS-label pattern + service: {name: , port: } # must match a rendered Service + routing: # REQUIRED in practice: omitted => no HTTPRoute + routes: [{pathPrefix: /, pathType: PathPrefix}] + tls: {enabled: true} + auth: + enabled: true + provider: keycloak + provisionClient: true + enforceAtGateway: true # false only for app-native OAuth (Harbor, Grafana) + scopes: [openid, profile, email, groups] # include groups whenever auth.groups is set + groups: [] # optional allow-list; needs the groups scope above + gateway: public # or internal + landingPage: {enabled: true, displayName: ...} +``` + +The namespace must carry `nebari.dev/managed=true` (ArgoCD +`managedNamespaceMetadata` is the usual way). + +## Repository shape + +``` +/ + chart/ # or Chart.yaml at the root; one pack chart + Chart.yaml # apiVersion v2, semver version, appVersion = upstream app + values.yaml # nebariapp: {enabled, hostname, service, routing, auth, ...} + values.schema.json # optional but common + templates/ + nebariapp.yaml # gated by .Values.nebariapp.enabled + _helpers.tpl, NOTES.txt + examples/ + argocd-application.yaml # real repoURL, valuesObject with nebariapp + nebari-values.yaml # full Nebari deployment + standalone-values.yaml # nebariapp.enabled=false (if standalone-supported) + dev/Makefile # kind cluster with operator+gateway+keycloak + docs/ # Astro site on @nebari/starlight, base // + .github/workflows/ + lint.yaml # helm lint; helm template both modes; kubeconform; check-jsonschema pack-metadata.yaml + test.yaml # kind install standalone, port-forward health check + test-integration.yaml # operator installed, NebariApp Ready, SecurityPolicy present + release.yaml # pack-release.yaml@v1 (first-party) or equivalent publish + build-images.yaml # pack-build-image.yaml@v1 when there are first-party images + docs.yml, docs-preview-cleanup.yml + pack-metadata.yaml # name, display_name, level, owner, deprecated, nebariapp_integration, scope + CODEOWNERS + README.md # what / who / deploy command / prerequisites / known limitations / troubleshooting / link to docs + SECURITY.md, LICENSE, CHANGELOG.md, .editorconfig, .gitignore +``` + +Two accepted ways to render the NebariApp: + +- the `nebari-app` library chart (`oci://quay.io/nebari/charts`, `>=0.1.1`) via + `include "nebari-app.nebariApp"` with `tplCtx`, which lets `values.yaml` + carry `{{ }}` expressions ending in `| toJson`; +- a hand-written `templates/nebariapp.yaml` (harbor, langfuse, and most packs built outside the org). + +Both are fine. What matters is the `enabled` gate, explicit `routing`, and a +`service` reference that resolves. + +## Wrapping upstream software + +Most packs wrap an upstream chart: add it to `Chart.yaml` `dependencies`, +override its values under its key, and point the NebariApp at its Service. +Vendoring a copy of the upstream chart under `charts/` is tolerated but is a +maintenance liability and a sign the pack has not been shaped yet. + +## Values conventions + +- `nebariapp.enabled: false` by default so the chart installs standalone. +- `nebariapp.hostname` is required when enabled (`required` / `fail`). +- Secrets: `existingSecret` names, `required` for must-set values, or + `lookup`-based generation guarded for GitOps. No literal defaults. +- Images: immutable tag or digest; first-party images get `sha-` tags + pinned by the release workflow. +- Every `nebariapp.*` key is commented in `values.yaml` or tabled in the docs. + +## CI conventions + +- Lint on every push/PR: `helm lint`, `helm template` with the NebariApp on + and off, `kubeconform -strict -ignore-missing-schemas`, + `check-jsonschema` on `pack-metadata.yaml`. +- Standalone test on kind with `--wait` and an HTTP health check. +- Integration test with the operator (pinned release or + `action-nebari-sandbox`) waiting for `NebariApp` `Ready`. +- Release on `Chart.yaml` version bump or `v*` tag; chart packaged and + published; images pinned to the release sha. + +## Maturity, in practice + +- **Experimental**: repo shaped like the template, owner known, metadata + present, README says what it is. +- **Alpha**: installs on a NIC dev cluster from the README, NebariApp goes + Ready, prerequisites and known limitations written down, one example + values file that works with only a hostname change. +- **Beta**: lint/template/kubeconform in CI, probes, non-root, pinned images, + no literal secrets, auth and troubleshooting docs, Nebari + standalone + examples, ArgoCD example, a published 0.x release, `product_owner` set. +- **GA**: integration test against the NIC stack, hardened securityContext, + NetworkPolicy, sizing and upgrade docs, CHANGELOG, docs site, 1.0.0, + install-then-upgrade smoke test. + +## Things that look like integration but are not + +- An `Ingress` (ALB, Traefik, nginx) alongside or instead of the NebariApp. +- A bundled Keycloak, Ory, Istio, or Envoy: the platform already provides + identity and the gateway. +- `routing` omitted from the NebariApp. +- `auth.enforceAtGateway: false` without docs on how the app uses the + operator-provisioned OIDC client Secret. +- `auth.groups` set without `groups` in `auth.scopes`: the operator attaches + the Keycloak groups scope only when requested, so the token has no groups + claim and the group gate rejects everyone (seen live on a NIC cluster). +- Placeholders (``, `REPLACE_`, `CHANGEME`) that `helm install` + accepts silently. diff --git a/tools/pack-audit/summary_html.py b/tools/pack-audit/summary_html.py new file mode 100644 index 0000000..2dc958c --- /dev/null +++ b/tools/pack-audit/summary_html.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Render a self-contained HTML readiness page from reports/*.json and the +auditor narratives at the top of reports/*.md. + + python3 -I summary_html.py reports/a.json reports/b.json ... --out reports/readiness.html +""" +from __future__ import annotations + +import argparse +import html +import json +import re +from pathlib import Path + +import yaml + +HERE = Path(__file__).resolve().parent +LEVELS = ["E", "A", "B", "GA"] +LEVEL_NAME = {"E": "Experimental", "A": "Alpha", "B": "Beta", "GA": "GA"} + + +def md_inline(s: str) -> str: + s = html.escape(s) + s = re.sub(r"`([^`]+)`", r"\1", s) + s = re.sub(r"\*\*([^*]+)\*\*", r"\1", s) + s = re.sub(r"(?\1", s) + return s + + +def md_block(md: str) -> str: + """Tiny markdown subset: ##/### headings, -/1. lists (one level of nesting), paragraphs.""" + out, stack = [], [] # stack of open list tags + def close(to=0): + while len(stack) > to: + out.append(f"") + for raw in md.splitlines(): + if not raw.strip(): + continue + m = re.match(r"^(#{1,4})\s+(.*)", raw) + if m: + close() + lvl = min(len(m.group(1)) + 1, 4) + out.append(f"{md_inline(m.group(2))}") + continue + m = re.match(r"^(\s*)([-*]|\d+\.)\s+(.*)", raw) + if m: + depth = 1 + (len(m.group(1)) // 2 if m.group(1) else 0) + tag = "ol" if m.group(2)[0].isdigit() else "ul" + if depth > len(stack): + out.append(f"<{tag}>"); stack.append(tag) + elif depth < len(stack): + close(depth) + out.append(f"
  • {md_inline(m.group(3))}
    • ") + continue + close() + out.append(f"

    {md_inline(raw.strip())}

    ") + close() + return "\n".join(out) + + +def narrative(md_path: Path) -> dict[str, str]: + """Split the agent-written narrative (above the generated '# Software-pack audit:' line) into sections.""" + text = md_path.read_text() if md_path.exists() else "" + head = text.split("\n# Software-pack audit:", 1)[0] + sections: dict[str, str] = {} + cur, buf = "intro", [] + for ln in head.splitlines(): + m = re.match(r"^##\s+(.*)", ln) + if m: + sections[cur] = "\n".join(buf).strip(); cur, buf = m.group(1).strip().lower(), [] + elif ln.startswith("# "): + continue + else: + buf.append(ln) + sections[cur] = "\n".join(buf).strip() + return sections + + +def pick(sections: dict[str, str], *keys: str) -> str: + for k, v in sections.items(): + if any(key in k for key in keys): + return v + return "" + + +def badge(status: str) -> str: + return f'{status}' + + +def build(results: list[dict], checklist: dict, narratives: dict[str, dict[str, str]], title: str, lede: str) -> str: + items = {i["id"]: i for i in checklist["items"]} + cats = checklist["categories"] + results = sorted(results, key=lambda r: -(r["score"]["overall_pct"] or 0)) + baseline = [r for r in results if r["name"].startswith("baseline")] + packs = [r for r in results if not r["name"].startswith("baseline")] + + def level_cell(r): + lv = r["score"]["repo_level"] + return LEVEL_NAME[lv] if lv else "below Experimental" + + def ron(r): + x = r["score"].get("runs_on_nebari") or {"state": "unknown", "reason": ""} + return f"{x['state']}" + + rows = [] + for r in packs + baseline: + sc = r["score"] + b = {lv: len(sc["blockers"][lv]) for lv in LEVELS} + rows.append( + f"{html.escape(r['name'])}" + + ("baseline, first-party" if r in baseline else "") + "" + f"
    {sc['overall_pct']}%" + f"{level_cell(r)}{ron(r)}{html.escape(str(r['facts'].get('detected_integration')))}" + + "".join(f"{b[lv]}" for lv in LEVELS) + "") + + cat_head = "".join(f"{html.escape(r['name'])}" for r in packs + baseline) + cat_rows = [] + for cid, cname in cats.items(): + cells = [] + any_ = False + for r in packs + baseline: + c = r["score"]["per_category"].get(cid) + if c and c.get("pct") is not None: + any_ = True + cells.append(f"{c['pct']:.0f}%") + else: + cells.append("n/a") + if any_: + cat_rows.append(f"{cname}{''.join(cells)}") + + shared = [cid for cid in items if packs and all(r["checks"].get(cid, {}).get("status") in ("FAIL", "PARTIAL") for r in packs)] + shared_html = "".join(f"
  • {cid} {LEVEL_NAME[items[cid]['level']]} {html.escape(items[cid]['title'])}" + + (f" rule" if items[cid].get("source") else "") + "
    • " for cid in shared) + + pack_sections = [] + for r in packs + baseline: + sc, f = r["score"], r["facts"] + n = narratives.get(r["name"], {}) + verdict = pick(n, "verdict") + fixes = pick(n, "top fixes", "what still", "blocks") + risks = pick(n, "risk") + verify = pick(n, "cluster verification", "needs") + fp = pick(n, "false positive", "scanner") + gate_rows = [] + for lv in LEVELS: + p = sc["per_level"][lv] + blk = sc["blockers"][lv] + gate_rows.append(f"{LEVEL_NAME[lv]}{p['pass']}{p['partial']}" + f"{p['fail']}{p['manual']}{'' if not blk else ', '.join(blk)}") + check_rows = [] + for cid, it in items.items(): + if it.get("enabled", True) is False: + continue + ch = r["checks"].get(cid, {"status": "JUDGMENT", "evidence": ""}) + src = (f"rule" if it.get("source") else "") + \ + (f" · example" if it.get("example") else "") + check_rows.append(f"{cid}{it['level']}{badge(ch['status'])}" + f"{html.escape(it['title'])}{html.escape(ch['evidence'])}{src}") + src = f"{html.escape(r['source'])}" + (f" @ {html.escape(r['ref'])}" if r.get("ref") else "") + pack_sections.append(f""" +
    +
    +
    +

    {'Baseline, first-party pack' if r in baseline else 'Audited pack'}

    +

    {html.escape(r['name'])}

    +

    {src}
    chart {html.escape(str(f.get('primary_chart')))} {html.escape(str((f.get('chart_yaml') or {}).get('version')))} + · NebariApp integration {html.escape(str(f.get('detected_integration')))} · scanned {html.escape(f['scanned_at'][:10])}

    +
    +
    {sc['overall_pct']}%{level_cell(r)}Runs on Nebari: {ron(r)}
    +
    +
    +
    +

    Verdict

    {md_block(verdict) or '

    No narrative yet.

    '} +

    Gates

    +
    {''.join(gate_rows)}
    LevelPassPartialFailManualBlocking items
    +
    +
    +

    Top fixes

    {md_block(fixes) or '

    See the checklist below.

    '} +
    +
    +
    Risks outside the checklist{md_block(risks) or '

    none recorded

    '}
    +
    Needs cluster verification{md_block(verify) or '

    none recorded

    '}
    + {f'
    Scanner corrections{md_block(fp)}
    ' if fp else ''} +
    Full checklist ({len(items)} items) +
    {''.join(check_rows)}
    IDLvlStatusItemEvidenceSource
    +
    +
    """) + + return f"""{html.escape(title)} + + +
    +

    Nebari software-pack audit

    +

    {html.escape(title)}

    +

    {html.escape(lede)} + Items are weighted by the level at which they block promotion (E=4, A=3, B=2, GA=1). The repo-readiness level is the highest + level with no failing or partial item at or below it. Items that need a cluster or a person are listed, not scored.

    + +

    Overview

    +
    + + + {''.join(rows)}
    PackScoreRepo-readiness levelRuns on NebariNebariAppE blockersA blockersB blockersGA blockers
    +

    Runs on Nebari: verified = installed on a cluster with the operator and NebariApp reached Ready · likely = NebariApp renders with valid CRD fields, explicit routing and a resolvable Service (static only) · no = no NebariApp or spec fails + NebariApp: full = routing + gateway auth render · partial = NebariApp without auth · none = no NebariApp

    + +

    Category scores

    +
    {cat_head}{''.join(cat_rows)}
    Category
    + +

    Gaps shared by every audited pack

    +
      {shared_html or '
    • none
      • '}
    + + {''.join(pack_sections)} + +

    Generated by tools/pack-audit (audit.py + the software-pack-auditor agent). Audit mode is read-only on the pack repositories.

    +
    +""" + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("results", nargs="+") + ap.add_argument("--out", required=True) + ap.add_argument("--title", default="Pack Readiness") + ap.add_argument("--lede", default="Software packs scored against the nebari-dev software-pack-template release-readiness checklist (Experimental, Alpha, Beta, GA).") + args = ap.parse_args() + checklist = yaml.safe_load((HERE / "checklist.yaml").read_text()) + results, narratives = [], {} + for f in args.results: + p = Path(f) + r = json.loads(p.read_text()) + results.append(r) + narratives[r["name"]] = narrative(p.with_suffix(".md")) + Path(args.out).write_text(build(results, checklist, narratives, args.title, args.lede)) + print(f"wrote {args.out} ({Path(args.out).stat().st_size // 1024} KB)") + + +if __name__ == "__main__": + main()