Mine bug-fix commits out of open-source repositories — across smart contracts, blockchain nodes, wallets, operating systems and cryptography.
Why · Install · Run · Coverage · Pipeline · Schema · Classifying without a model · Verification · Contributing
A uniform harness for mining bug-fix commits out of open-source repositories, across security domains.
One framework, five registries, one output schema:
domains/<domain>/repos.yml → mirror → walk → signals → diffs → classify → CSV
which repositories git every deterministic patches a model
(the only place a clone commit annotations from the reads the
repo is ever named) in every (no model) mirror diff
branch
Everything up to classify is deterministic, offline after the clone, resumable,
and free. classify is the only stage that needs a model.
Because the commit worth reading usually does not say so — and now there is a number for it. Of the 672 security fixes found across 5,607 smart-contract commits:
| carrying a CVE / GHSA / RUSTSEC id | 8 (1.2%) |
| firing no deterministic signal at all | 637 (95%) |
A keyword crawl finds about one in twenty. The other nineteen are titled
feat: independent collateral, fix: minor issues, fix: Math.min.
A keyword crawl decides what to look at from the words the author happened to
choose, which is exactly the assumption a silent fix breaks. The commit that
says cleanup, refactor validation, or prevent unwanted wrapping is the one
that quietly closed an integer overflow. Teams fix security bugs quietly all the
time — before a disclosure window, to avoid handing out an exploit, or because
nobody realised the bug was exploitable.
So this harness reads all of it and lets the classifier be the only judge of meaning. The only things filtered before the model are structural facts about the commit, never a guess at what it means:
| dropped | why |
|---|---|
| merge commits | carry no diff of their own |
| commits touching no source file | cannot contain a code defect |
commits above --max-files (60) |
vendored drops, generated output, formatting sweeps — not one defect, and the diff does not fit a prompt |
A refactor is not dropped. Neither is a commit whose message says nothing. Those are the population this project exists to search.
uv venv && uv pip install -e .
oss-bugfixes validate # every registry parses
oss-bugfixes domains # what is covered# the deterministic half — no model, no API key, no cost
oss-bugfixes mirror --domain crypto --tier 1 --workers 8
oss-bugfixes walk --domain crypto --tier 1
oss-bugfixes signals --domain crypto
oss-bugfixes diffs --domain crypto --min-signals 1
# the model half
oss-bugfixes classify --domain crypto --engine llm:openai --model qwen3-coder \
--base-url https://ollama.com/v1
# or all of it, with the no-model baseline
oss-bugfixes run --domain crypto --repo argon2 --engine heuristic
oss-bugfixes status --domain cryptoEvery stage is separately re-runnable and skips what the state file says is
already done, because a 180-repo sweep takes hours and will be interrupted.
One repository failing never takes down the run; status tells you what to retry.
| domain | repos | scope rule | past-vulnerability corpus |
|---|---|---|---|
smartcontract |
155 | deployed on-chain code and the contract libraries it inherits — Solidity, Solana Rust and Move | 4,470 audit findings · 220,912 commits walked, 125,110 classified |
blockchain |
47 | node software, where a defect can halt or split a chain | 2,225 client fixes |
wallet |
178 | anything whose defect costs funds, keys or signing authority | 34,526 wallet fixes |
os |
34 | code running at a privilege the applications above cannot revoke | — |
crypto |
42 | implementations where a defect breaks a guarantee, not just a process | — |
456 repositories. domains/<d>/repos.yml is the coverage claim — no crawler,
script or notebook in this repository hard-codes a repository name, so adding one
is a registry edit and nothing else.
domains/wallet/
repos.yml the registry — the only place a repository is named
data/
vulns.parquet the past-vulnerability corpus (see docs/corpora.md)
vulns.preview.csv a slice GitHub renders as a table
bugfixes.csv what this framework produced (created on first run)
docs/ scope notes, limitations, analysis
No code lives in a domain folder. No repository name lives outside one.
vulns.parquet is the predecessor corpora, migrated in their own schema:
expert-judged audit findings and curated client/wallet fixes, carrying severity
ratings, STRIDE/CWE labels and contest provenance this framework does not
produce.
bugfixes.csv is what this framework walked and classified, in the schema in
docs/schema.md.
They are kept apart on purpose. A finding says what an auditor caught; a commit says what shipped. Folding one into the other would make the provenance of the corpus unreadable, and would quietly claim this pipeline had examined rows it never saw.
Blobless mirrors make diffs expensive, and one fetch makes them cheap again.
A --filter=blob:none clone leaves file contents on the server and fetches them
one at a time on first access — a network round trip per commit. On
P-H-C/phc-winner-argon2 (496 commits), fresh clone each time:
| blobless, diff the whole history lazily | 214.7 s |
blobless + --hydrate, then diff |
0.9 s + 2.0 s |
A hundredfold, bought with 0.9 s and 6 MB. So walk defaults to --name-only
(file paths come from trees, which a blobless clone already has), line counts and
backport detection are opt-in behind --hydrate, and the diffs stage hydrates
a mirror by itself once it needs more than a couple of hundred patches.
A fix verb alone is not a signal. fix typo matches every fix-verb regex
ever written. The signal pass only counts a fix verb when it co-occurs with a
recognisable impact class, and scores keywords over unique tokens rather than
occurrences — counting repetitions is how release-note boilerplate once scored
itself to 1.0 and produced 95 bogus Critical findings.
A diff's text contains sentences the author wrote about the code — the one channel this corpus has decided not to trust. So the prompt carries code only: comment-only lines dropped, trailing comments cut, every structural marker kept.
That is a correctness measure before it is an optimisation. Left in, this happened:
- // We use 100 because the calldata length is 4 + 32 * 2.
+ // We use 68 because the calldata length is 4 + 32 * 2.
let callStatus := call(gas(), token, 0, callData, 68, 0, 0)All 24 changed lines of that commit were comments and the code always read
68. The model reported "corrects the calldata length constants … fixing a
functional bug" — it had read a comment about a constant as the constant. A
model that cannot see the comment cannot make that mistake.
It is also the cheapest speed-up available: 22% smaller prompts across 5,607 commits, and 816 of them (14.6%) change no code at all, so the structural engine settles those in-process and they never reach an endpoint.
The deterministic pass annotates; it does not gate. A row that fires no signal is still handed to the classifier. The signals exist to order a classification budget, to tell the model what is already known, and to measure how much a keyword crawl would have missed — the disagreement between the baseline and the model is the most informative number this project produces.
--min-signals N scopes a run without pretending the rest were examined and
rejected: skipped rows are marked drop:below_min_signals, not labelled negative.
These are not style preferences; each one is a bug this lineage already shipped.
- A skipped check is
<NA>, neverFalse. Backport detection that timed out once reported "0 backported", indistinguishable from a repo that backports nothing. Every tri-state column stays tri-state end to end. - A failed classification is undecided, not negative. A timeout or an
unparseable reply records the error; it never becomes
is_bugfix=false. - A truncated diff says so, in the prompt. A model handed a silently cut patch reasons confidently about code it never saw.
- An optional column never costs a repository. Backport detection failing loses the column and nothing else.
- Human labels win. A row whose
classifierishumanis not overwritten by a model unless you type--overwrite-human.
| engine | needs | for |
|---|---|---|
structural |
nothing | a deterministic pre-filter at 107 commits/sec — see below |
lean |
the Lean toolchain | the same classifier with unbounded proofs, 56 commits/sec |
heuristic |
nothing | CI, baselines, and measuring what the model adds |
llm:claude |
the claude CLI |
interactive work |
llm:openai |
any OpenAI-compatible endpoint | whole-corpus sweeps (Ollama Cloud, vLLM, LM Studio) |
llm:ollama |
a local ollama | offline work |
Replies are cached by (model, prompt), so a rerun after a crash costs nothing
for the part that finished. Adding an engine is one register() call.
ossbugfix/detect is a classifier with no model in it: a
total diff parser, ten structural edit detectors, and a three-valued verdict.
The honesty rules above are not conventions here. The classifier's decision has a finite domain — three booleans read off the diff, times subsets of ten signatures, 8,192 inputs — so the rules are discharged over all of it, three independent ways:
$ uv run python -m ossbugfix.detect.verify
[exec] ran the real decide() over all 8,192 inputs: ok
[z3] 8 properties proved unsat-of-negation
[link] SMT model vs real decide(), all 8,192 points: agree
VERIFIED
A positive verdict is also unconstructable without a witness:
Verdict.asserted() takes the witnessing signatures as an argument, and
__post_init__ rejects any verdict asserting more than its evidence supports —
so no such object exists anywhere in the process, not merely as a return value.
The [link] step is the one that matters. A solver proves properties of the
model you hand it, and the first time it ran it reported a disagreement between
the SMT encoding and the code: Z3 had "proved" all eight properties of a model
that did not describe the program. docs/verification.md
draws the line between proved, tested, and merely believed.
Measured on 5,607 smart-contract commits:
| throughput | 107 commits/sec (vs ~0.25 for a hosted model) |
| provably refuted, needs no model ever | 14% |
| abstained and handed on | 78% |
It is a filter, not a replacement. Hand-written token rules over-fire by
roughly an order of magnitude against a model's base rate, and three rounds of
tightening did not close that — which is exactly why the literature on this
problem (VFFinder, Fixseeker, GraphSPD) builds AST and code-property graphs and
learns, instead of matching tokens.
docs/deterministic-classification.md
has the numbers, the prior art and the GNN roadmap.
A second implementation lives in lean/, where parser totality and
monotonicity are theorems over the unbounded input space rather than
properties under test. The two agree on 5,607 / 5,607 commits and
tests/test_engine_parity.py keeps them that way.
docs/pipeline.md— each stage, its failure modes, what to do about themdocs/schema.md— every column, and what<NA>means in itdocs/adding-a-domain.md— adding a domain or a repositorydocs/corpora.md— where the migrated corpora came fromdocs/deterministic-classification.md— classifying without a model: prior art, the measurements, and what it does not dodocs/verification.md— what is proved, what is tested, and where the line islean/README.md— the Lean package itself
This supersedes three separate efforts, whose collection/ directories were two
near-copies of the same pipeline:
NyxFoundation/ethereum-vuln-dataset→domains/blockchainNyxFoundation/wallet-vuln-dataset→domains/walletNyxFoundation/vulnerability-reports→domains/smartcontract
The framework is the wallet lineage generalised. That corpus walks whole histories and lets the model judge; the earlier Ethereum one started from keyword search and therefore could not see a silent fix by construction. Where the two disagreed, the wallet design won.
The highest-value contribution is a registry entry: one repository, one
repos.yml line, no code. See
docs/adding-a-domain.md and
CONTRIBUTING.md.
Corrections to the corpus are equally welcome. A row whose classifier is
human is never overwritten by a model, so a hand-checked verdict is
permanent.
MIT — the framework, the registries and the migrated corpora alike. See
LICENSE.
If this corpus or harness is useful in published work, cite the repository and the predecessor datasets it supersedes (listed under Lineage).