Langfuse Dashboard Forge

Schema-aware dashboards as code, with a reviewable plan before writes.

A runnable Python project and an installable agent skill. Define native Langfuse widgets and their layout in YAML; discover the target deployment's schema; validate, plan, apply, and verify through the public API. No database writes, application fork, UI automation, or custom chart renderer.

Version 0.1.0 · source review 16 September 2026 · independent of Langfuse.

What is included

• Deployment-local OpenAPI 3.0/3.1 parsing using jsonschema and referencing; local $ref, allOf, oneOf, nullable values, request/response validation, and a machine-readable transport catalog.
• Per-view semantic discovery using Langfuse's getMetricsSchema MCP tool, or an explicitly supplied result file. Fields and aggregations are not guessed from a hand-written global list.
• A thin manifest using Langfuse's native widget fields, plus stable ownership keys, widget reuse, presets, and checked 12-column grid placements.
• Remote-read-only plans, explicit plan-ID approval, schema/drift checks, bounded aggregate-query preflight, and read-back verification.
• Idempotent reconciliation, explicit pruning, guarded compensating rollback, private recovery journals, and portable exports for cloning dashboards.
• A sibling-compatible SKILL.md, examples, tests, a credential-free HTTP demo, an opt-in live smoke script, and implementation/research references.

Not included: arbitrary metadata dimensions or metrics, scatter plots, legacy traces widget creation, adoption of existing resources by name, database mutation, browser screenshots, or a universal OpenAPI client generator. Metadata filters remain supported. One manifest manages one dashboard and its owned widget set; repeat the workflow for additional dashboards.

Two ready-to-deploy manifests are included in dashboards/: overview.yaml tracks local usage, cost, tokens, and latency; reliability.yaml tracks errors, slow root operations, and numeric evaluation scores. Each has its own stable dashboard key and five owned widgets under the local-langfuse namespace.

Install from this ZIP/checkout

Use Python 3.11 or newer. macOS, Linux, and WSL are supported for deployment locking. The core package does not require Node, Docker, or Langfuse's SDK.

cd langfuse-dashboard-forge
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[test,mcp]'
lfdash --help

This package is supplied as source, not assumed to be published on PyPI. The optional mcp extra uses the official SDK's v1 Streamable HTTP client; it is bounded below v2 because that API differs. Live acceptance exercised that SDK transport against Langfuse 4.36.1. Without the extra, install .[test] and supply a raw getMetricsSchema response with --metrics-schema FILE.

The scripts/lfdash.py launcher is an alternative to the installed entry point:

/path/to/venv/bin/python /path/to/langfuse-dashboard-forge/scripts/lfdash.py --help

It preserves the caller's working directory. Global options precede the verb.

Try the entire workflow without credentials

From the project root after installation:

python -m pytest
python scripts/demo.py

The demo starts a loopback REST test double, invokes the real CLI, discovers its schema, creates five widgets and a dashboard, verifies their definitions, re-applies with zero writes, exports, then rolls back. It uses a temporary workspace and dummy credentials. It is not a real Langfuse server, query engine, MCP transport, or UI. The fixture source and limitations are documented in tests/fixtures/PROVENANCE.md.

To inspect parsing and validation directly, without an API connection:

lfdash --workspace /tmp/lfdash-offline discover --offline \
  --openapi-file tests/fixtures/openapi.yaml \
  --metrics-schema tests/fixtures/metrics-schema.json
lfdash --workspace /tmp/lfdash-offline catalog --view observations
lfdash --workspace /tmp/lfdash-offline schema --out /tmp/lfdash-schema.json
lfdash --workspace /tmp/lfdash-offline validate examples/platform.yaml

Offline bundles deliberately cannot plan or apply to a real project.

Use your Langfuse deployment

Keep credentials and deployment state in a project workspace, not in an agent-indexed skill folder. init refuses to overwrite existing files.

mkdir -p ~/projects/langfuse-dashboards
cd ~/projects/langfuse-dashboards
lfdash init
cp .env.example .env
chmod 600 .env

Edit .env locally:

LANGFUSE_BASE_URL=https://your-langfuse-deployment.example
LANGFUSE_PUBLIC_KEY=pk-lf-your-project-public-key
LANGFUSE_SECRET_KEY=sk-lf-your-project-secret-key

Use the actual region/deployment URL and a project API key pair. Do not paste keys into chat. LANGFUSE_HOST is also accepted, but conflicting values are an error. An explicitly named .env overrides shell variables; interpolation and shell evaluation are disabled. --env - uses only the shell. An optional LANGFUSE_PROJECT_ID adds local plan binding; it is not independently checked against the key's project by this tool.

1. Discover, then inspect the real contract

lfdash --env .env doctor
lfdash --env .env discover
lfdash catalog --view observations
lfdash schema --out .lfdash/transport-schema.json

doctor is local only. discover reads /api/openapi.yaml; it tries the same host's /generated/api/openapi.yml only when the first path returns 404. It never silently switches to public Cloud or repository main. Schemas cannot redirect requests to a different host. The schema's servers field is not a routing authority.

By default, semantic discovery initializes /api/public/mcp, lists tools, and calls only getMetricsSchema. It never calls dashboard-writing MCP tools.

Where MCP transport/access is unavailable, save the raw JSON tool result from the same deployment and explicitly use:

lfdash --env .env discover --metrics-schema /path/to/getMetricsSchema.json

The file must contain the result object with a views mapping, not the outer JSON-RPC envelope. Its provenance is labeled supplied-file, not live MCP. apply still requires successful bounded Metrics API query execution. Do not substitute the bundled reduced fixture for a real semantic catalog.

2. Define and validate

Edit the generated dashboard.yaml. Its five-widget starter distinguishes logical root observations from generation costs, includes a latency histogram, and includes a model-summary pivot. The production environment filter is an example: align it with your actual instrumentation.

lfdash validate dashboard.yaml
lfdash --env .env check-queries dashboard.yaml

A small complete definition:

apiVersion: langfuse-dashboard-forge/v1
namespace: agent-platform

dashboard:
  key: overview
  name: Agent Platform Overview
  description: Model usage and performance.
  filters:
    - column: environment
      type: string
      operator: '='
      value: production

widgets:
  cost-by-model:
    name: Generation cost by model
    view: observations
    dimensions:
      - field: providedModelName
    metrics:
      - measure: totalCost
        agg: sum
    filters:
      - column: type
        type: string
        operator: '='
        value: GENERATION
    chartType: HORIZONTAL_BAR
    chartConfig:
      row_limit: 10
      show_value_labels: true

placements:
  - key: model-cost
    widget: cost-by-model
    x: 0
    y: 0
    width: 12
    height: 6

Do not change namespace/dashboard/widget keys merely to change display names. Keys define ownership. width and height are public API grid units; do not use the database's x_size/y_size field names.

assets/manifest.schema.json supports editor validation. Runtime validation adds source-reviewed chart constraints and the actual discovered vocabulary. The schema alone is not the full validator.

3. Plan and review exact changes

lfdash --env .env plan dashboard.yaml --out plan.json

No remote writes occur. Inspect the full plan.json, particularly the target, operation payloads, before images, warnings, and planId. The summary only shows counts. By default, removing a widget or tile from YAML does not remove it remotely: pruning must be explicit. Manually added tiles are retained.

4. Apply the reviewed plan

lfdash --env .env apply plan.json --approve FULL_PLAN_ID_FROM_REVIEW
lfdash --env .env verify dashboard.yaml

An apply requires the matching discovery bundle and plan, the exact deployment and public-key fingerprint, a plan younger than 24 hours, unchanged relevant remote state, and unchanged OpenAPI. Before a regular mutating apply, it executes bounded aggregate queries (default last 24 hours; at most seven days). Each write is journaled and read back; a final pass verifies the desired definitions. A repeated unchanged plan produces no writes and skips unnecessary preflight.

The approval hash is not a signature or a substitute for organizational permissions. Restrict who can edit/approve plans and serialize deployments in CI. API keys stay in the invoking process, not the plan.

5. Iterate or prune

lfdash --env .env plan dashboard.yaml --out next-plan.json
# Review next-plan.json, then apply with its exact planId.

lfdash --env .env plan dashboard.yaml --prune --out prune-plan.json
# Review every deletion before approving this separate plan.

The tool blocks changing widgets that are shared with other dashboards. It never adopts objects by matching names. A new namespace means new ownership, not automatic deletion or adoption of old resources.

6. Recover or export

lfdash --env .env rollback-plan .lfdash/runs/RUN.json --out rollback-plan.json
# Review inverse operations, then apply with that plan's exact planId.

lfdash --env .env export DASHBOARD_ID \
  --namespace cloned-platform --key overview --out clone.yaml

Rollback is a guarded, per-run compensating operation, not a transaction. Recreated deleted widgets receive new IDs; restored dashboard references are rewired. Unknown write outcomes and intervening edits block unsafe rollback. For an uncertain POST outcome, re-plan forward first; ownership markers find resources the server may already have created.

export writes this project's manifest for cloning. It does not import the Langfuse UI's separate $langfuseDashboard/$langfuseWidget portable envelope. Legacy trace-view widgets are rejected rather than silently converted.

Skill installation

Place the entire source folder beside the official Langfuse skill:

.agents/skills/
  langfuse/                    # optional official sibling, separately installed
    SKILL.md
  langfuse-dashboard-forge/    # this whole folder
    SKILL.md
    scripts/
    src/
    references/

Keep virtual environments, test outputs, .env, and .lfdash outside the indexed skill directory. The skill references the official sibling but does not vendor, install, overwrite, or change its permissions. The optional agents/openai.yaml adds display metadata for compatible clients.

Verification status and boundaries

See evidence/ and references/testing.md for executed tests and exact limits. Authenticated acceptance against a local Langfuse 4.36.1 deployment covered OpenAPI and MCP discovery, ten bounded Metrics API queries, deployment of both five-widget dashboards, read-back, zero-drift verification, repeat no-op applies, and browser rendering. Credentials remained in an ignored sibling environment file and are not part of this repository. Deterministic tests also exercise reduced source-derived contracts and an independent HTTP test double; the fixture is explicitly not a byte-for-byte upstream OpenAPI snapshot.

There is no cross-resource REST transaction or ETag compare-and-swap. Local file locks do not serialize two CI workers or an operator editing in the UI. The tool detects many races before and after writes, but cannot eliminate the last check-then-write window. Use a single writer per dashboard.

The semantic schema can expose fields that persisted widgets hide. Transport, semantic, and query validation improve safety but cannot prove every dashboard will render. A rejected widget creation stops and journals the run; it does not justify bypassing the API. Verify actual chart rendering and data usefulness on your deployment before adopting this in production.

Project map

Path	Responsibility
src/.../openapi.py	OpenAPI normalization, operation discovery, transport validation
src/.../discovery.py	Deployment schema and semantic discovery, provenance bundle
src/.../capabilities.py	Per-view semantic validation and explicit chart-rule adapter
src/.../manifest.py	Manifest schema, compilation, ownership, layout checks
src/.../planner.py	Read-only inventory and reconciliation plans
src/.../deploy.py	Preconditions, journals, apply, verification, rollback planning
src/.../client.py	Narrow HTTP adapter, auth, bounded reads, pagination
src/.../metrics.py	Runtime query construction and bounded preflight
src/.../exporting.py	Native dashboard to cloneable manifest
scripts/demo.py	Credential-free end-to-end loopback demo
scripts/live_smoke.py	Explicitly authorized isolated live smoke test
references/	Workflow, safety, design, research, source provenance, testing

MIT-licensed project code. Third-party source references are documented in THIRD_PARTY_NOTICES.md.