Skip to content

AGENTS.md

Latest commit

 

History

History
86 lines (70 loc) · 4.85 KB

AGENTS.md

File metadata and controls

86 lines (70 loc) · 4.85 KB
Raw

Agent Guidelines for Apache DataFusion Comet

Read the Contributor Guide before making changes. Relevant entry points:

  • Development Guide: build, test, and common pitfalls (including why -pl must not be used and the JVM/native build order).
  • Spark SQL Tests: the only supported way to modify files under dev/diffs/. Never hand-edit a diff file. Clone Spark, apply the existing diff, modify the Spark source, then regenerate the diff as documented there.
  • Adding a New Expression / Adding a New Operator.
  • Debugging Guide.
  • Documentation Style Guide: the terms to use for how Comet runs a query, and the conventions for docs pages. Read it before editing docs/source, README.md, or the .doc(...) strings in config classes.
  • Backporting to Release Branches: a fix that goes to one release branch also goes to every newer one. Cherry-pick with -x, and never add an unrelated change to a backport pull request.

When opening a pull request, use the PR template and fill in every section.

Use git push for normal updates to a PR branch. If a rebase or amend requires a force push, use git push --force-with-lease, never --force or -f, to reduce the risk of overwriting another maintainer's commits. If the lease check rejects the push, inspect and integrate the remote changes before retrying; do not bypass it with --force. See Submitting a Pull Request.

Checking a change against CI

A green pull request does not mean a change is safe to queue. The pull request tier runs the Comet suites, Spark's own SQL suite (catalyst and sql_core) and the Iceberg 1.11 suite against the default Spark profile only. For a pull request from a fork it runs on the fork, where CI (fork) reports it as advisory, and Required Checks here passes without a build. Everything else first reports in the merge queue (Spark SQL sql_hive), where a failure evicts the pull request and blocks everyone else's merges, or in the nightly run (the other Spark and Iceberg versions), after the change has already landed.

So when a change touches code those suites cover but the pull request tier does not run, such as Hive table support, a Spark shim for a version other than 4.1, or an Iceberg code path that differs across versions, get a verdict first. Either run the suite locally:

dev/local-ci.sh spark 3.5 sql_core-1  # a nightly-tier version, named explicitly
dev/local-ci.sh iceberg 1.10 shard-2  # one shard, or `iceberg` for every target

or apply the matching run-* label so CI runs it instead. Do not do either for Spark SQL 4.1 (catalyst and sql_core) or Iceberg 1.11, which every pull request already runs. Which changes warrant which suite, the label names, and the script's caveats are in Continuous Integration.

Pick one rather than both. A full local job is hours of compute and tens of GB of disk, so prefer the shard that covers the change, and prefer the label when the change is broad. Two things matter when running it unattended: preparing sweeps the whole local Maven repository, which is a shared cache, and SKIP_PREPARE=1 skips the Comet install so it must not be used after changing Comet.

Skills

Repository-specific agent skills live under .ai/skills/. Each subdirectory is a single skill with a SKILL.md (YAML frontmatter + body). Check that directory for an applicable skill before starting a task; new skills go in .ai/skills/<skill-name>/SKILL.md.

For compatibility with agents that look in vendor-specific locations, .claude/skills is a symlink to .ai/skills and CLAUDE.md is a symlink to this file. Add new content here rather than to the symlinks.