Repository navigation
[TRACKER] CI Test Failures Needing attention #2118
Description
This is a tracker issue for CI test failures, mostly from flaky tests, that need attention. We should strive to root cause and fix the issues, rather than skipping tests or adjusting thresholds without understanding why the test is failing.
- [CI] STATS_TEST silhouetteScore flaky #2406
- [CI] CLUSTER_TEST kmeans flaky #2100
- [CI] NEIGHBORS_ANN_NN_DESCENT_TEST flaky #2093
- [CI] NEIGHBORS_ALL_NEIGHBORS_TEST flaky #2137
- [CI] NEIGHBORS_MG_TEST flaky #2588
- [CI] SCANN Test flaky #2549
- [CI] Lucene: GPU rewrite path bypasses Lucene's exact-search fallback for selective filters #2522
- [CI] TestCuVSAcceleratedHNSWGaps.testVectorSearchWithAlternatingDocuments is flaky #2551
- [BUG] TestLucene99AcceleratedHNSWVectorsFormat.testMergeTwoSegsWithASingleDocPerSeg fails with vector data mismatch after forceMerge(1) #2550
- [CI] CUTILE_SMOKE_TEST GB300 failed #2602
- [CI] TestCagraToHnswSerializationAndSearch flaky #2735
- [CI] NNTest_fp32_fused GB300 error #2759
- [CI] NEIGHBORS_DYNAMIC_BATCHING_TEST flaky #2957
- [A] [CI] CLUSTER_TEST spectral flaky #2094
- [B] [BUG] CAGRA hang on h100 oldest-deps CI (cuvs test_cagra, then cuvs_bench test_run_command_creates_results) #2772
- [C] [CI] TestMultiSegmentGPUFilterConcurrency flaky #2736
- [D] No gh issue, but merged Load cuvs-lucene GPU indexes on first search for readers opened for a merge #2795
- [E] [BUG] kmeans::fit is not run-to-run reproducible; KmeansFitBatchedTestF.Result/4 flakes ~50% on GB10 and should not gate CI #2657
- [F] No gh issue, Fix CAGRA dataset descriptor race across threads sharing a handle #2807
- [G] [CI] NEIGHBORS_ANN_IVF_RABITQ_TEST flaky #2971
- [H] No gh issue, Fix flaky IVF-PQ Python tests #2797
Todo from #2783:
| WS | Issue(s) | Fix PR(s) | Status |
|---|---|---|---|
| A | #2094, #1594 (closed) | #2841 | ✅ Merged 10/16. Alternative fix #2794 closed. |
| B | #2772, #2732 | NVIDIA/raft#3186, #2933 | ✅ Merged 10/26. Fix moved into RAFT (@robertmaynard, 10/22); #2933 supersedes #2829 and depends on NVIDIA/raft#3186. |
| C | #2736 | #2582, #2799, #2967 | ✅ Merged 10/30. Fixed on main by #2582 (merged 10/22). #2799 (TestCuVSGaps topK cap) merged 10/29; C++ regression test #2967 merged 10/30. |
| D | none | #2795 | ✅ Merged 10/13. |
| E | #2657 | #2801, #2804, NVIDIA/raft#3176 | 🔴 Update 10/29: Waiting to see if E may be resolved by NVIDIA/raft#3176. #2801: changes requested (@viclafargue, 10/22): benchmark against NVIDIA/raft#3176 and consider making determinism optional. #2804: no reviews yet. |
| F | none | #2807 | 🟡 Needs C++ review (CAGRA dataset descriptor race across threads sharing a handle). CI green. |
| G | #2971 | #2798 | 🟡 Needs C++ review (IVF-RaBitQ build with empty lists). CI green. |
| H | #1445 (closed) | #2797 | 🟡 Needs Python review (test-only, test_ivf_pq.py). CI green. |
| J | #631, #1048, #1468 | #2805, #2830 | 🟡 #2805 (reject single-row CAGRA build) merged 10/21. #2830 (re-enable CagraRandomizedIT with a recall check): has merge conflicts; imotov (10/19) suggests fixing the empty-BitSet bug in Util.concatenate instead of bypassing it in CagraIndexImpl. |
Todo:
After release/26.10 reenable AnnNNDescentTestUI8_U32.
Activity
cjnolet commentedon Oct 21, 2026
@bdice can you provide a quick review/feedback on the status on this tracker PR based on the status of your fixes? Just making sure we're staying aligned on the current status.