Agent Pipeline

A small Node.js multi-agent pipeline for searching the web with Tavily, generating research notes with Groq, and turning those notes into a structured summary. The project deliberately uses direct API calls rather than an orchestration framework, so each stage is easy to inspect and replace.

What the project does

The pipeline accepts a topic and runs three Groq-backed stages:

1. Researcher - searches Tavily with advanced search and up to five results, then asks the configured Groq model to produce raw, fact-based bullet points with the actual source URLs.
2. Summarizer - converts the findings into a JSON object containing a title, three key points, and a takeaway.
3. Evaluator - checks whether the summary contains three distinct, high-quality points and returns APPROVED or REVISE.

The coordinator runs these stages in sequence and returns the complete result through one API endpoint. If the evaluator returns REVISE, the coordinator regenerates the summary from the same research notes and evaluates the new summary again, with a maximum of five total iterations.

Important: Tavily supplies the web-search results and Groq generates the research notes from those results. The project does not independently fact-check the returned claims or verify that every citation is authoritative.

Architecture

Client
  |
  | POST /run { "topic": "..." }
  v
Express server (server.js)
  |
  v
Coordinator (agents/coordinator.js)
  |
  +--> Researcher (agents/researcher.js)
  |       Tavily advanced search (max 5 results)
  |       Groq chat completion -> sourced research notes
  |
  +--> Summarizer (agents/summarizer.js)
  |       Groq chat completion -> structured JSON
  |
  +--> Evaluator (agents/evaluator.js)
          Groq chat completion -> APPROVED or REVISE

Each stage writes a JSONL event to logs/run-<timestamp>.log

Technology stack

• Node.js with ECMAScript modules
• Express 4 for the HTTP API
• @tavily/core for web search
• groq-sdk for Groq chat completions
• dotenv for loading local environment variables
• nodemon for development reloads
• No LangChain, CrewAI, Anthropic SDK, or other agent framework is used

Project structure

agent-pipeline/
├── agents/
│   ├── coordinator.js   # Runs the stages and applies the iteration limit
│   ├── evaluator.js     # Evaluates the generated summary
│   ├── researcher.js    # Generates raw research-style notes
│   └── summarizer.js    # Generates the required summary JSON
├── logs/                # Runtime JSONL logs (ignored by Git)
├── .env.example         # Environment variable template
├── index.js             # Command-line entry point
├── logger.js            # Console and file logging helper
├── package.json
├── package-lock.json
├── server.js            # Express API entry point
└── README.md

Prerequisites

• Node.js and npm installed and available on your PATH
• A Groq account and API key
• A Tavily account and API key
• A Groq model available to your account

Setup

From the agent-pipeline directory:

1. Install dependencies

npm install

2. Create the environment file

PowerShell:

Copy-Item .env.example .env

Edit .env and set a real key:

GROQ_API_KEY=gsk_your_actual_groq_key_here
GROQ_MODEL=openai/gpt-oss-20b
PORT=3000
TAVILY_API_KEY=tvly-your_tavily_api_key

GROQ_MODEL is passed to the researcher, summarizer, and evaluator requests. The researcher uses the configured model after retrieving Tavily search results. TAVILY_API_KEY authenticates the web-search request. Keep both keys in .env; never commit them.

3. Create the runtime log directory

The logger appends to a file in logs/. Ensure that directory exists before starting the service:

New-Item -ItemType Directory -Force logs

The directory contents are ignored by Git. Never commit .env or an API key.

Running the project

HTTP API mode

Start the server:

npm start

For development with automatic restarts:

npm run dev

The server listens on PORT, or port 3000 when PORT is not set.

Command-line mode

Run the pipeline without starting Express:

npm run cli -- "state of serverless in 2026"

If no topic is supplied, index.js uses the current state of AI agent frameworks.

API reference

GET /health

Returns a lightweight readiness response:

{
  "status": "ok"
}

POST /run

Request body:

{
  "topic": "state of serverless in 2026"
}

PowerShell example:

Invoke-RestMethod `
  -Uri "http://localhost:3000/run" `
  -Method Post `
  -ContentType "application/json" `
  -Body '{"topic":"state of serverless in 2026"}'

Successful responses have this shape:

{
  "topic": "state of serverless in 2026",
  "notes": "- Finding one...\n- Finding two...",
  "summary": {
    "title": "A generated title",
    "keyPoints": [
      "First key point",
      "Second key point",
      "Third key point"
    ],
    "takeaway": "A concise overall takeaway"
  },
  "iterations": 1,
  "verdict": "APPROVED"
}

The notes and summary values are generated output. The researcher first obtains up to five advanced-search results from Tavily and then gives those results to Groq. The API does not independently validate citations, source quality, or factual claims.

Error responses

• 400 - the request does not contain a truthy topic field:

  { "error": "topic is required" }

• 500 - a pipeline or Groq request failed:

  { "error": "pipeline failed", "detail": "..." }

Logging

Each stage logs an event to the console and to a file named logs/run-<timestamp>.log. Each line is a JSON object with:

{
  "timestamp": "2026-09-12T06:19:09.000Z",
  "stage": "researcher",
  "data": {}
}

The logs may contain the topic, research notes, and generated summary. Treat them as potentially sensitive and do not commit or publish them.

How to extend the project

Customize web research

The researcher is implemented in agents/researcher.js. It uses Tavily's advanced search with maxResults: 5, then includes the returned results in a Groq prompt that asks for factual bullet points with actual URLs and no summary or conclusion. To change the research behavior:

1. Update the Tavily search options, such as searchDepth or maxResults.
2. Update the researcher prompt while preserving the topic input and search-result context.
3. Preserve actual source URLs in the returned notes and continue instructing the model not to invent URLs.
4. Add input-length limits and request error handling before exposing the endpoint publicly.

The result should not be described as verified research unless the returned sources and claims are independently checked.

Implement true revision retries

The coordinator already regenerates and re-evaluates the summary when the evaluator returns REVISE. To make revision quality more targeted:

1. Change evaluate to return structured feedback, not just APPROVED or REVISE.
2. Pass the evaluator feedback into summarize on the next iteration.
3. Evaluate the revised summary.
4. Keep the existing maximum-iteration guard.

This would make the existing research → summarize → evaluate → revise workflow feedback-driven instead of simply generating another summary from the same notes.

Add another pipeline stage

Create a module in agents/, export a function that accepts the preceding stage's output, and call it from runPipeline in agents/coordinator.js. Keep provider calls and stage-specific prompts inside their stage modules so the coordinator remains an orchestration layer.

Change the Groq model

Update GROQ_MODEL in .env. This changes the model used by all three Groq-backed stages: researcher, summarizer, and evaluator.

Operational considerations

• Keep GROQ_API_KEY server-side and out of source control.
• Validate and limit topic length before forwarding user input to the model in a public deployment.
• Add authentication and rate limiting before exposing /run outside a trusted local environment.
• Consider request timeouts and retry policies for production Groq calls.
• Treat generated citations as unverified until checked against the original Tavily results and original sources.
• Avoid logging sensitive user input if the service handles private topics.

Current limitations

• No persistent database or job queue is used.
• Requests are processed synchronously.
• There is no authentication, rate limiting, schema validation, or CORS configuration.
• The evaluator returns only APPROVED or REVISE; it does not provide detailed revision instructions.
• The researcher uses Tavily search results but does not independently verify source quality or factual accuracy.
• There are currently no automated tests in this package.

License

This project is licensed under the MIT License. See the LICENSE file for details.