SlopScore
00 crowd

llm-exporter

Prometheus exporter for LLM API monitoring — probes OpenAI, Anthropic, Google Gemini, Azure OpenAI and any OpenAI-compatible endpoint, collecting TTFT, latency, token usage and availability metrics.
Open repo on GitHubgithub.com/oh-my-vibe-coding/llm-exporter
Go · ★ 1 · 0 forks · MIT · paperwork by the Cap'mmostly ai (inferred)light human (inferred)works-on-my-machine (inferred)other
listed 1 hour ago by oh-my-vibe-coding · last checked 1 hour ago
The owner didn't write this. This repo never submitted itself. The Cap'm found it on a truffle trawl and wrote its paperwork from what GitHub already shows. Picked by hand by the Cap'm on 2026-09-26: Prometheus exporter for LLM API monitoring — probes OpenAI, Anthropic, Google Gemini, Azure OpenAI and any Ope; its own README says "md) 100% AI-Coded — All code in this project was written entirely by AI (Claude Code)". 1 stars; MIT license. The owner did not submit this. Votes count; awards don't until the owner claims it.

I'm not calling your project slop! Geeze, it's a joke... Do you own this repo?

Log in with GitHub as oh-my-vibe-coding. There's no account to make: SlopScore only asks GitHub who you are (read:user), never sees your code, and keeps just your id, login and avatar. Then you can:

  • Keep it, on your terms. Commit your own slopscore.md (spec) and press Refresh. Your paperwork replaces the Cap'm's, and you can submit it for Slop of the Day.
  • Take it down. One click on Remove. It stays gone; the trawl never brings it back.

Log in with GitHub

Can't log in as the owner? Request a takedown. No login needed, and a trawled listing comes down right away.

GitHub says
Prometheus exporter for LLM API monitoring — probes OpenAI, Anthropic, Google Gemini, Azure OpenAI and any OpenAI-compatible endpoint, collecting TTFT, latency, token usage and availability metrics.
topics
exportergrafanallmmonitoringmonitoring-toolobservabilityopenaiprometheusttft
created
2026-04-09 · pushed 1 month ago · 37 commits · 2 contributors
release
v2.1.0 · 2026-05-10
languages
Go 99%Dockerfile 1%Makefile 0%
paperwork
code of conductcode of conduct filecontributingpull request templatelicensereadme 100% health
dependencies
no dependency graph (no manifest, or disabled) · OSV.dev, checked 1 hour ago

Disclosures, inferred by the Cap'm

slopbucket
vibe-coded
category
other
ai_generated
mostly
human_touch
light
status
works-on-my-machine
language (detected)
dockerfilegomakefile
topic (detected)
exportergrafanallmmonitoringmonitoring-toolobservabilityopenaiprometheusttft
license (detected)
mit

The Cap'm's log

The Cap'm wrote this paperwork, not the owner. This repo never submitted itself to SlopScore. The Cap'm picked it by hand: Prometheus exporter for LLM API monitoring — probes OpenAI, Anthropic, Google Gemini, Azure OpenAI and any Ope; its own README says "md) 100% AI-Coded — All code in this project was written entirely by AI (Claude Code)". It carries the MIT license. The disclosures above are his best guess from what GitHub shows.

Is this yours? Commit a real slopscore.md and press Refresh to replace this, or remove the listing in one click. There's no account to make: you log in with GitHub.

README — the repo's own words, folded up so the grading fits on one screen

LLM Exporter

English | 中文

100% AI-Coded — All code in this project was written entirely by AI (Claude Code). Human involvement was limited to product direction, requirements, and code review.

A Prometheus exporter for monitoring LLM API availability and performance. Periodically sends streaming probe requests to LLM API endpoints, collecting network latency, Time to First Token (TTFT), and token usage metrics.

Think of it as blackbox_exporter for LLM APIs — end-to-end probing tailored for large language models.

Features

  • Streaming TTFT measurement — All APIs probed via streaming requests for precise Time to First Token measurement
  • Multi-provider support — OpenAI Chat Completions + Responses API, Anthropic, Google Gemini, Azure OpenAI, plus any OpenAI-compatible service
  • Reasoning & cache token tracking — reasoning_tokens, cached_input_tokens, cache_creation_tokens captured from OpenAI / Anthropic / Gemini usage fields
  • Wide provider coverage — xAI Grok, Groq, Cerebras, Together, Fireworks, Moonshot/Kimi, Zhipu GLM, SiliconFlow, DashScope (Alibaba), Volcengine Ark (ByteDance), DeepSeek, Mistral, OpenRouter, local Ollama, and more
  • TLS cert expiry + rate-limit visibility — Exporter surfaces earliest-cert NotAfter and provider x-ratelimit-remaining-* headers as Prometheus gauges
  • Lightweight — Only depends on prometheus/client_golang, gopkg.in/yaml.v3, and fsnotify/fsnotify — no LLM SDKs
  • Single binary — Go 1.23+ compiled, deploys as binary / Docker / Kubernetes
  • Flexible config — YAML with ${ENV_VAR} expansion, custom headers, and custom API paths
  • Hot reload — SIGHUP signal, HTTP /-/reload endpoint, or --watch-config file watcher
  • Multi-target /probe mode — blackbox_exporter-style /probe?target=<url>&module=<name> for file_sd / dynamic target discovery
  • Native histograms — Duration / TTFT histograms dual-emit classic + native (Prometheus 2.50+ stores the native form)
  • Webhook alerting — Sends webhook notifications (Slack/Teams/etc.) after consecutive probe failures
  • Prompt rotation — Cycle through multiple prompts to avoid provider-side caching
  • Response validation — Regex match on model output to verify the model is actually working
  • Adaptive probe interval — Automatically reduces probe frequency when stable, resets to base interval on failure
  • Non-streaming mode — stream: false for APIs that don't support streaming
  • Ops-friendly — --validate config check, --version info, /api/v1/targets status API, llm_exporter_build_info + Go runtime metrics, example Prometheus rules

Quick Start

Binary

# Build
make build

# Configure
cp config.example.yaml config.yaml
# Edit config.yaml, fill in your API keys

# Set environment variables (or write directly in config.yaml)
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Run
./llm-exporter --config config.yaml

# Verify
curl http://localhost:9101/metrics | grep llm_probe
curl http://localhost:9101/healthz

Docker Compose (with Prometheus + Grafana)

# Configure API keys
cp .env.example .env
# Edit .env

# Start the full monitoring stack
docker compose up --build -d

# Access
# LLM Exporter:  http://localhost:9101/metrics
# Prometheus:     http://localhost:9090
# Grafana:        http://localhost:3000  (admin/admin)

Grafana auto-loads a pre-built LLM monitoring dashboard on startup.

Deployment

Systemd

Suitable for direct deployment on Linux servers.

1. Install binary

make build
sudo cp llm-exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/llm-exporter

sudo mkdir -p /etc/llm-exporter
sudo cp config.example.yaml /etc/llm-exporter/config.yaml
sudo vim /etc/llm-exporter/config.yaml

2. Create system user

sudo useradd --system --no-create-home --shell /usr/sbin/nologin llm-exporter

3. Create systemd service

sudo cat > /etc/systemd/system/llm-exporter.service << 'EOF'
[Unit]
Description=LLM Exporter - Prometheus LLM API Probe Exporter
Documentation=https://github.com/oh-my-vibe-coding/llm-exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=llm-exporter
Group=llm-exporter
ExecStart=/usr/local/bin/llm-exporter --config /etc/llm-exporter/config.yaml
# Optional: append --watch-config to auto-reload on config file changes
# (systemd deployments typically rely on `systemctl reload` instead)
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadOnlyPaths=/etc/llm-exporter
PrivateTmp=yes

EnvironmentFile=-/etc/llm-exporter/env

[Install]
WantedBy=multi-user.target
EOF

4. Environment file (optional)

sudo cat > /etc/llm-exporter/env << 'EOF'
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
EOF
sudo chmod 600 /etc/llm-exporter/env
sudo chown llm-exporter:llm-exporter /etc/llm-exporter/env

5. Start

sudo systemctl daemon-reload
sudo systemctl enable --now llm-exporter
sudo systemctl status llm-exporter
sudo journalctl -u llm-exporter -f

Docker

docker run -d \
  --name llm-exporter \
  --restart unless-stopped \
  -p 9101:9101 \
  -v $(pwd)/config.yaml:/etc/llm-exporter/config.yaml:ro \
  --env-file .env \
  llm-exporter

Docker Compose (full stack):

docker compose up --build -d
Service Port Description
llm-exporter 9101 Exporter metrics endpoint
prometheus 9090 Prometheus server
grafana 3000 Grafana (admin/admin), auto-provisioned dashboard

Kubernetes

1. ConfigMap and Secret

apiVersion: v1
kind: ConfigMap
metadata:
  name: llm-exporter-config
  namespace: monitoring
data:
  config.yaml: |
    listen_addr: ":9101"
    targets:
      - name: openai-gpt4o
        endpoint: "https://api.openai.com"
        api_key: "${OPENAI_API_KEY}"
        model: "gpt-4o"
        api_format: openai
        timeout: 30s
        interval: 60s
---
apiVersion: v1
kind: Secret
metadata:
  name: llm-exporter-secrets
  namespace: monitoring
type: Opaque
stringData:
  OPENAI_API_KEY: "sk-..."
  ANTHROPIC_API_KEY: "sk-ant-..."

2. Deployment + Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-exporter
  namespace: monitoring
  labels:
    app: llm-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llm-exporter
  template:
    metadata:
      labels:
        app: llm-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: llm-exporter
          image: llm-exporter:latest
          args: ["--config", "/etc/llm-exporter/config.yaml", "--watch-config"]
          ports:
            - containerPort: 9101
              name: metrics
          envFrom:
            - secretRef:
                name: llm-exporter-secrets
          volumeMounts:
            - name: config
              mountPath: /etc/llm-exporter
              readOnly: true
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9101
            initialDelaySeconds: 5
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /healthz
              port: 9101
            initialDelaySeconds: 3
            periodSeconds: 10
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
      volumes:
        - name: config
          configMap:
            name: llm-exporter-config
---
apiVersion: v1
kind: Service
metadata:
  name: llm-exporter
  namespace: monitoring
  labels:
    app: llm-exporter
spec:
  selector:
    app: llm-exporter
  ports:
    - port: 9101
      targetPort: metrics
      name: metrics

3. ServiceMonitor (Prometheus Operator)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-exporter
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: llm-exporter
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

4. Deploy

kubectl apply -f llm-exporter-config.yaml
kubectl apply -f llm-exporter-deployment.yaml
kubectl apply -f llm-exporter-servicemonitor.yaml

Hot Reload

Three ways to reload config without restarting:

SIGHUP signal:

sudo systemctl reload llm-exporter
# or: kill -HUP $(pidof llm-exporter)
# Docker: docker kill --signal=HUP llm-exporter

HTTP endpoint:

curl -X POST http://localhost:9101/-/reload

File watcher (recommended for Kubernetes):

llm-exporter --config /etc/llm-exporter/config.yaml --watch-config

Handles Kubernetes ConfigMap symlink swaps with 2-second debounce. On reload failure, existing probes continue running unaffected.

Configuration

Config Format

listen_addr: ":9101"      # Listen address, default :9101

# Webhook alerting (optional)
# webhook:
#   url: "https://hooks.slack.com/services/xxx"
#   consecutive_failures: 3  # default 3

targets:
  - name: openai-gpt4o     # Target name (required, used as provider label)
    endpoint: "https://api.openai.com"   # API endpoint (required)
    api_key: "${OPENAI_API_KEY}"         # API key, supports env vars
    model: "gpt-4o"                      # Model name (required)
    prompt: "Hi"                         # Probe prompt, default "Hi"
    api_format: openai                   # API format: openai / anthropic / google / azure
    timeout: 30s                         # Probe timeout, default 30s
    interval: 300s                       # Probe interval, default 300s
    max_tokens: 20                       # Max output tokens, default 20
    chat_path: "/v1/chat/completions"    # Custom API path (openai format only)
    extra_headers:                       # Custom request headers
      X-Custom-Auth: "token"
    # stream: false                      # Non-streaming mode (default true)
    # api_version: "2024-10-21"          # Azure API version (azure format only)
    # prompts: ["Hi", "Hello", "Hey"]    # Prompt rotation (mutually exclusive with prompt)
    # expect_pattern: "\\d+"             # Response validation regex
    # adaptive_interval: true            # Adaptive interval (default off)
    # max_interval: 1200s                # Adaptive cap, default 4x interval
    # backoff_after: 5                   # Consecutive successes before backoff, default 5

Provider Examples

OpenAI 
- name: openai-gpt4o
  endpoint: "https://api.openai.com"
  api_key: "${OPENAI_API_KEY}"
  model: "gpt-4o"
  api_format: openai
Anthropic Claude 
- name: anthropic-claude
  endpoint: "https://api.anthropic.com"
  api_key: "${ANTHROPIC_API_KEY}"
  model: "claude-sonnet-4-20250514"
  api_format: anthropic
Google Gemini 
- name: google-gemini
  endpoint: "https://generativelanguage.googleapis.com"
  api_key: "${GOOGLE_API_KEY}"
  model: "gemini-2.0-flash"
  api_format: google
Azure OpenAI 
- name: azure-gpt4o
  endpoint: "${AZURE_OPENAI_ENDPOINT}"
  api_key: "${AZURE_OPENAI_API_KEY}"
  model: "gpt-4o"
  api_format: azure
  # api_version: "2024-10-21"  # default
DashScope (Alibaba) 
- name: dashscope-qwen
  endpoint: "https://dashscope.aliyuncs.com/compatible-mode"
  api_key: "${DASHSCOPE_API_KEY}"
  model: "qwen-plus"
  api_format: openai
Volcengine Ark (ByteDance) 
- name: ark-pro
  endpoint: "https://ark.cn-beijing.volces.com/api"
  api_key: "${ARK_API_KEY}"
  model: "${ARK_ENDPOINT_ID}"
  api_format: openai
DeepSeek / Mistral / OpenRouter / Ollama 
# DeepSeek
- name: deepseek-chat
  endpoint: "https://api.deepseek.com"
  api_key: "${DEEPSEEK_API_KEY}"
  model: "deepseek-chat"
  api_format: openai

# Mistral
- name: mistral-large
  endpoint: "https://api.mistral.ai"
  api_key: "${MISTRAL_API_KEY}"
  model: "mistral-large-latest"
  api_format: openai

# OpenRouter
- name: openrouter-claude
  endpoint: "https://openrouter.ai/api"
  api_key: "${OPENROUTER_API_KEY}"
  model: "anthropic/claude-sonnet-4"
  api_format: openai
 Read the rest on GitHub
Scan report · 2026-09-26
  • ✓ Prohibited terms or links — profanity in body/README
  • ✓ Repository eligibility
  • ✓ slopscore.md paperwork
  • ✓ Content policy
  • ✓ Risk review

0 comments

log in to comment.

report this listing — log in to report