An AI agent that researches stocks and answers follow-up questions using live financial data, news, and SEC filings.

Live Demo: financial-research-agent.streamlit.app | Built with Claude Code

Documentation: start at docs/README.md — every question a reviewer might ask, mapped to the document that answers it.

A research agent that turns live market data, news, and SEC filings into investment briefs, and audits the quantitative and forward-looking claims in each brief's Executive Summary and Outlook against the sources it retrieved, using an LLM judge whose own accuracy is measured against blind human labels. It runs as a six-service stack (FastAPI, Celery, Redis, PostgreSQL, Streamlit, MCP server) on Kubernetes, with the grounding eval as a gated Argo Workflows DAG. On OCI it runs on single-node k3s on A10 GPU VMs, where vLLM serves a QLoRA fine-tune that was measured against the hosted models and ships disabled. Every rate this repository quotes names its judge version and the run or source it came from, with a Wilson confidence interval wherever the claim counts are on record.

• Where: two VM.GPU.A10.1 instances (1x A10 24 GB, Ubuntu 22.04): vm-a10-inst-1 (the demo target) and vm-a10-inst-2 (fallback), each a separate single-node k3s cluster, rebuilt from the runbook on 2026-09-23. They are reached only through ssh tunnels; the VCN security list admits port 22 only.
• What runs there: the six services; vLLM v0.10.2 serving the merged fine-tune (financial-lora) on the node's A10, behind the default-off USE_LOCAL_MODEL flag; and Argo Workflows running the gated grounding-eval DAG (the nightly CronWorkflow is suspended on k3s; runs are submitted with make vm-eval).
• One manifest set: a kustomize base with kind, k3s, and oke overlays; scripts/render_diff.py proves overlay changes never alter the kind render (docs/verification.md).
• OKE: Terraform for an OKE basic cluster, both node pools (including an A10 GPU pool), OCIR, an Object Storage bucket, and a Block Volume storage class is written and passes terraform validate, but has never been applied. The k3s VMs are the running target.
• How to deploy it: docs/deploy-runbook.md (single-node k3s path, and the OKE steps).

Every row links to docs/numbers-of-record.md, which carries the full records and the rules for quoting them. Judge-v2 rates are approximate per the judge's held-out calibration (75% recall / 60% precision on UNSUPPORTED).

Generate Brief -- enter a ticker and the app produces a structured investment brief:

1. Fetches stock data (yfinance), news (NewsAPI), and SEC filing summaries (EDGAR)
2. Runs two concurrent Pinecone RAG queries to ground the SEC Filing Highlights and Risk Factors sections in actual filing text (a retrieval defect that indexed exhibit text instead of Item 1A for most tickers was found and fixed 2026-09-04 — see docs/eval-methodology.md, "Retrieval defect")
3. Generates four middle sections in parallel using Claude Haiku
4. Streams the Executive Summary and Outlook from Claude Sonnet, which receives the pre-written sections as context
5. Caches the completed brief in Redis (exact key research:{TICKER}) and PostgreSQL

Ask a follow-up -- a LangGraph ReAct agent answers free-form questions, selecting whichever tools it needs (stock data, news, SEC filings, or RAG search).

Two execution paths, by design: the Streamlit UI imports the agent and runs the pipeline in-process (so the hosted demo needs no backend and can stream tokens directly), while the FastAPI app runs the same agent/ code behind REST endpoints for programmatic consumers (and is what ECS/Kubernetes deploy). Same pipeline, two entry points -- there is no Streamlit→FastAPI hop.

I built this to answer a question I couldn't find a good answer to: can an LLM agent produce investment briefs that are actually grounded in real sources -- and how would you even know?

The answer required building both the agent and the measurement layer to audit it.

I built an evaluation framework that audits the quantitative and forward-looking claims in each brief's Executive Summary and Outlook against the retrieved source context (the four pre-written sections are judge input, not audited directly). A Sonnet judge (temperature 0) labels each claim SUPPORTED, UNSUPPORTED, or INFERENCE.

Early results: 49% unsupported claim rate (judge v1, pre-retrieval-fix). Nearly half of what the agent said wasn't backed by anything it retrieved.

After iterating on prompt constraints and forcing generation to stay grounded in source material, the 2026-08-24 10-ticker re-measure found 0/84 unsupported — a dated record (judge v1, a lower bound; pre-retrieval-fix). The current grounding number of record is the 40-ticker hosted baseline j4cnp (2026-09-05/06): 12/392 = 3.06% unsupported (Wilson 95% CI 1.8–5.3%), judge v2 on the fixed retrieval pipeline — approximate per the judge's held-out calibration (75% recall / 60% precision on UNSUPPORTED, blind labels, n=50). See docs/numbers-of-record.md.

Judge-version note: every unsupported rate in this README names its judge prompt version. v1 rates are lower bounds (2026-09-04 human validation: v1 recall on UNSUPPORTED 1/9). v2 rates carry the held-out calibration — kappa 0.580, 75% recall / 60% precision on UNSUPPORTED against blind human labels (n=50, 2026-09-06) — and are approximate point estimates; A/B directions are unaffected when both arms share the judge (docs/eval-methodology.md).

The prompt engineering work -- not the retrieval architecture -- was what actually moved the needle.

I added optional cross-encoder reranking to the RAG pipeline and ran a controlled 4-arm eval across 10 tickers to measure whether it improved grounding. A dated record: Jun 2026, judge v1, pre-retrieval-fix, local grounding_check.py runs (pre-Argo, so no workflow run IDs); intervals from eval/stats.py:

Conclusion: reranking adds 4--5x retrieval latency with no reliable grounding benefit. It ships default-off. The measurement framework is the deliverable -- it's what demonstrates the feature isn't needed, rather than assuming it would help.

Can a small local model replace Claude Haiku on section generation at lower cost?

I fine-tuned Qwen2.5-1.5B-Instruct with QLoRA on 104 deterministic, Claude-free training pairs built from real SEC filings and financial data. When enabled, the fine-tuned model is routed 2 of 4 brief sections (Financial Health and Risk Factors); the other two stay on Haiku because deterministic targets couldn't be built for them -- an honest finding about the data, not a gap to paper over. The measured verdict (2026-09-05/06, 40-ticker in-cluster A/B, judge v2, same image and index): the fine-tune fails the 5% grounding gate -- 8.15% unsupported (lsnnc, 30/368, Wilson 95% CI 5.8–11.4%) vs a 3.06% hosted baseline (j4cnp, 12/392, CI 1.8–5.3%), Fisher p = 0.0023 -- and the failure concentrates in exactly the two sections it owns (19.82%, 22/111, CI 13.5–28.2%, vs 0.50%, 1/202, CI 0.1–2.8%, on attributed claims; same two runs), so it ships default-off. Judge-v2 rates are approximate per the held-out calibration (75% recall / 60% precision on UNSUPPORTED). The rest of this section is the experiment record.

No measurable full-brief cost reduction. Measured with the committed cost harness (scripts/cost_report.py; details in benchmarks.md), Sonnet synthesis dominates the bill: hosted $0.0316/brief vs hybrid $0.0321/brief (both on the pre-retrieval-fix pipeline; the current cost of record is $0.0366/brief, 2026-09-06 post-retrieval-fix) -- the saving on the two local sections is within run-to-run variance. Shipped default-off.

Aug 2026 re-measure (benchmarks.md): the grounding regression reproduced in direction -- grounding (supported share) 86.2% hosted (56/65, Wilson 95% CI 75.7–92.5%) vs 77.8% hybrid (56/72, CI 66.9–85.8%); judge v1, 9 tickers balanced, pre-retrieval-fix, local grounding_check.py run with no workflow run ID. The local model runs behind a pluggable OpenAI-compatible backend (LOCAL_MODEL_BACKEND=openai); on this AVX2-only dev CPU the prebuilt vLLM image SIGILLs (root-caused to its AVX-512 requirement), so locally the code path is exercised via Ollama's /v1 endpoint.

Sep 2026 GPU serving + in-cluster A/B: vLLM v0.10.2 served the merged fine-tune pinned to one A10 on an OCI VM -- plain Docker (2026-09-02), then in-cluster on single-node k3s (2026-09-03) -- and the gated eval DAG ran a 40-ticker A/B against it (2026-09-05/06, judge v2, same image and index both arms): 8.15% unsupported (lsnnc, 30/368, CI 5.8–11.4%) vs the hosted baseline 3.06% (j4cnp, 12/392, CI 1.8–5.3%), Fisher p = 0.0023 -- the local-model arm fails the 5% gate, the hosted baseline passes, and per-section attribution places the excess entirely in the two fine-tune-owned sections (19.82%, 22/111, CI 13.5–28.2% vs 0.50%, 1/202, CI 0.1–2.8%, p = 4.6e-10). An earlier 10-ticker pass agreed in direction but could not separate the arms (dated records in docs/eval-methodology.md). A measured negative result, and the reason USE_LOCAL_MODEL ships off.

Four-arm comparison (2026-09-23, a dated comparison set, not numbers of record; 40 tickers, judge v2, identical pinned sampling). To separate training from model size, the same harness scored four arms that differ only in who writes Financial Health and Risk Factors: each local model writes those two sections, while Haiku writes the other two and Sonnet the synthesis in every arm. Unsupported rates: the hosted baseline (kcf7s, 4/383 = 1.04%, CI 0.4–2.7%), the fine-tune (v924f, 25/385 = 6.49%, CI 4.4–9.4%), its untuned base Qwen2.5-1.5B-Instruct (4nfsm, 31/400 = 7.75%, CI 5.5–10.8%), and untuned Qwen2.5-7B-Instruct (cnkp2, 18/393 = 4.58%, CI 2.9–7.1%). The fine-tune matched its own base (p = 0.58); within Qwen2.5, 1.5B → 7B improved with borderline significance (p = 0.076) at 3.7x lower serving throughput on the same A10; and every open-weight arm trailed hosted (7B vs hosted p = 0.0039). Judge-v2 rates, approximate per the held-out calibration. Details: docs/eval-methodology.md.

I also re-implemented the same fine-tune with a hand-written PyTorch training loop (fine_tune_pytorch_loop.ipynb) -- custom Dataset, manual gradient accumulation and optimizer.step(), hand-written cosine LR, no Hugging Face Trainer. Benchmarked against the Trainer on identical data and config (adamw_torch, cosine schedule, grad-accum 8), the two loss curves track each other closely over 21 optimizer steps -- both start around 1.4--1.5 and trend down together, finishing at 0.50 (native) and 0.35 (Trainer). The curves cross repeatedly, so that final-step gap sits within the run-to-run noise at this scale (~7 optimizer steps/epoch, plus shuffle order and 4-bit-kernel non-determinism) rather than a systematic difference -- confirming the hand-written loop reproduces the Trainer's training dynamics at the gradient-accumulation and optimizer-step level.

Does breaking the single agent into a supervisor-orchestrated team improve grounding, or just add cost?

I refactored the brief pipeline into a supervisor graph: a planner decomposes the ticker into SEC retrieval sub-questions and coverage points, a research agent executes the plan against the existing RAG and model-routing code, an inline grounding-critic scores the drafted brief with the same LLM-as-judge used by the offline eval, and a supervisor sends the brief back for revision (bounded at 2 passes) until it clears a grounding threshold. It is flag-gated behind MULTI_AGENT_ENABLED (default off), with the original single-agent pipeline kept as the A/B control. The inline critic and the offline judge share one definition, so there is a single source of truth for grounding.

I ran both paths over the standard 10-ticker set, scored by the same temperature-0 Sonnet judge, with retrieval held at baseline (reranking off, top-3) on both sides so the only differences were the planner-driven queries and the critic loop. Critic threshold: 5% unsupported. A dated record: Jun 2026, judge v1, pre-retrieval-fix, local runs (pre-Argo, no workflow run IDs); intervals from eval/stats.py.

The multi-agent path also costs more per brief (an extra Haiku planner call and a Sonnet critic call on every brief). The cost ratio this experiment recorded came from an uncommitted harness, so it is kept only as a historical note in docs/PHASE0_AUDIT.md and not quoted here; the current re-runnable cost of record for the single-agent pipeline is $0.0366/brief (2026-09-06, post-retrieval-fix; scripts/cost_report.py).

Two findings, stated plainly:

1. The critic fired 0 revisions across all 10 real drafts. The base pipeline already drives unsupported claims to the floor (1/73 on the control arm above; a later re-measure on 2026-08-24 found 0/84, Wilson 95% CI 0.0–4.4%, judge v1, pre-retrieval-fix, local run with no workflow run ID) on the Executive Summary and Outlook sections this eval scores, so the critic looked at every first draft, found nothing to fix, and passed it. There was no headroom for the revision loop to recover.

2. The one single-agent unsupported claim did not reproduce. Across 73 single-agent claims exactly one was flagged (on WMT). Re-running WMT produced a different draft with zero unsupported claims, confirming the lone flag was temperature-0.2 generation variance, not a systematic weakness in the base synthesis prompt.

The orchestration adds cost and latency (the planner adds a Haiku call, and the inline critic adds a Sonnet judge to every brief) for a grounding benefit that, on this workload, was null.

Conclusion: it ships default-off. On this workload the multi-agent path showed no grounding benefit because the single-agent baseline was already at the grounding floor, leaving nothing for the critic to recover. That is a statement about this corpus, not a general claim about multi-agent orchestration. The revision loop itself is verified working (an injected bad draft with a fabricated price target is caught by the live judge, sent back, and cleaned on the second pass); it is dormant in practice, not broken. The harness is retained to re-measure if a harder corpus, thinner retrieval, a weaker base model, or longer briefs ever create real grounding headroom. If that happens, a three-way comparison (baseline / planner-only / planner+critic) is the documented next step to attribute any gain to the planner versus the critic. As with the reranking experiment, the deliverable is the measurement that shows when the feature is and is not worth its cost.

The full designed topology runs on a single-node kind cluster locally and on single-node k3s on the OCI A10 VMs (see k8s/README.md, docs/deploy-runbook.md, and the audit in docs/PHASE0_AUDIT.md). The diagram shows the kind layout; the k3s overlay adds vLLM on the node's A10:

flowchart LR
    subgraph kind["kind cluster (WSL2, single node)"]
        ST["Streamlit UI<br/>(runs the pipeline in-process,<br/>app.py unchanged)"]
        API["FastAPI<br/>:30080"]
        WK["Celery worker"]
        RD[("Redis<br/>exact-key cache research:TICKER<br/>+ Celery broker/backend")]
        PG[("PostgreSQL<br/>research_briefs (PVC)")]
        MCP["MCP server<br/>streamable-HTTP :30800"]
        subgraph argo["Argo Workflows"]
            WF["nightly grounding eval<br/>fan-out per ticker → aggregate<br/>→ FAIL below threshold"]
        end
    end
    EXT["Anthropic API · NewsAPI · SEC EDGAR<br/>· yfinance · Pinecone · LangSmith"]
    LOCAL["Local model (default-off)<br/>OpenAI-compatible endpoint:<br/>vLLM on an A10 (k3s) / Ollama (dev laptop)"]

    API -->|"cache + enqueue"| RD
    API -->|"briefs"| PG
    WK -->|"consume + cache"| RD
    ST -->|"cache"| RD
    API --> EXT
    WK --> EXT
    ST --> EXT
    MCP --> EXT
    WF --> EXT
    API -.->|"USE_LOCAL_MODEL=true"| LOCAL
    WK -.->|"USE_LOCAL_MODEL=true"| LOCAL

• Celery vs. Argo: request-time async stays on Celery; batch/eval runs on Argo (reasoning in the Celery vs. Argo section).
• Eval pods force BYPASS_CACHE=true and never touch the live cache.
• Feature flags are restated at their audited defaults in the ConfigMap; reranking, multi-agent, and the local model all ship off, each for a measured reason.

"Generate Brief"
       │
       ▼
Redis cache (research:TICKER) ──hit──► cached brief
       │ miss
       ▼
get_stock_data  (yfinance)
       │
       ▼
┌──────────────────┐  ┌─────────────────┐
│ get_company_news │  │ get_sec_filings  │  parallel
└──────────────────┘  └─────────────────┘
       │                       │
       └───────────┬───────────┘
                   │
       ┌───────────▼───────────┐
       │   Pinecone RAG (x2)   │  concurrent
       └───────────┬───────────┘
                   │
   ┌───────────────┼───────────────────┐
   ▼               ▼           ▼       ▼
Haiku           Haiku       Haiku   Haiku    4 parallel calls
Financial       Recent      SEC     Risk
Health          Dev.        High.   Factors
   └───────────────┴───────────┴───────┘
                   │
       ┌───────────▼───────────┐
       │  Sonnet: Exec Summary │  streams to browser
       │  + Outlook            │
       └───────────────────────┘

"Ask" (free-form question)
       │
       ▼
LangGraph ReAct agent (claude-sonnet-4-6)
  ├─ get_stock_data
  ├─ get_company_news
  ├─ get_sec_filings
  └─ query_sec_filing (Pinecone RAG)
       │
       ▼
     answer

The single-agent brief pipeline can be swapped for a supervisor-orchestrated graph. It's off by default -- the single-agent path stays the production default and the A/B control -- and produces the same brief schema and API response, so nothing downstream changes. Toggle the flag to compare the two paths.

"Generate Brief"  (MULTI_AGENT_ENABLED=true)
       │
       ▼
   ┌─────────┐  decomposes the ticker into a research plan: the SEC RAG
   │ Planner │  sub-questions that ground the filing-based sections + coverage
   └────┬────┘
        ▼
   ┌──────────┐ ◄──── revise (critic feedback prepended to the synthesis prompt)
   │ Research │  reuses the EXISTING retrieval + model-routing + synthesis code;
   └────┬─────┘  revision passes re-synthesise Exec Summary + Outlook only
        ▼
   ┌──────────────────┐  the existing LLM-as-judge, promoted to an inline node --
   │ Grounding-critic │  scores the draft for source-grounding (one judge, shared
   └────┬─────────────┘  with the offline eval; `agent/grounding.py`)
        ▼
   ┌────────────┐  unsupported% ≤ CRITIC_MAX_UNSUPPORTED_PCT → done; else send
   │ Supervisor │  back to Research, bounded at MAX_REVISIONS passes
   └────┬───────┘
        ▼
   final brief

• One judge, two callers. The inline critic and the offline grounding eval both call agent/grounding.py:grade_brief() -- there's a single definition of the judge prompt and scoring, not two copies that can drift.
• Schema-safe revisions. Revision passes reuse the already-grounded middle sections and only re-write the Executive Summary + Outlook through the same _synthesis_prompt, so the brief format can't break.
• Bounded loop. MAX_REVISIONS (default 2) caps the critic→research retries; the supervisor accepts the best effort if the budget is exhausted.
• Tracing. Each node (planner / research / critic / supervisor) is its own LangSmith span.

The primary deployment is on OCI (above). AWS ECS is a secondary, single-container deployment of the API only.

The FastAPI backend is containerized and runs on AWS ECS Fargate

Read the rest on GitHub