prompt-testing

An AI prompt optimizer for writing better prompts and getting better AI results.

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

LLM Reasoning and Generation Benchmark. Evaluate LLMs in complex scenarios systematically.

Ollama Model Test - Figure out the best model for the task

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.

OMK — Evidence-backed evaluation and observability for prompts, RAG, skills, agents, and workflows. Native Codex, Claude Code, and DeepSeek Harness support.

Turn failed AI agent runs into replayable regression tests. Catch regressions before you ship.

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

Open-source prompt optimization tool — improve your LLM prompts with AI-powered suggestions and explanations.

Semantic diff for LLM prompts — compare prompt versions like git diff

Test, compare, and optimize your AI prompts in minutes

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

Detect prompt regressions before they reach production — per-category accuracy scoring, deterministic validation, and False Improvement detection. Pure Python, zero dependencies.

Open-source simulation harness for testing AI agents. Simulate LLM and tool calls to test edge cases, failure paths, and agent logic without live API calls.

Community Plugin for Genkit to use Promptfoo

Local-first desktop app for building LLM test suites and running them against many local or cloud models at once