Paper: Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection Alberick Euraste Djire — ASE '26, October 12–16, 2026, Munich, Germany

A benchmark study investigating whether LLMs can detect LLM-generated code. Eight classifier models are evaluated across four prompting strategies on a dataset of human-written and AI-generated solutions to LeetCode problems.

data/AI-Human-Generated-Program-Code-Dataset.jsonl is a paired of human/AI solutions introduced in "Program Code Generation with Generative AIs". Dataset link

All strategies ask for a JSON response {"response": "yes"|"no", "explanation": "..."} where "yes" means LLM-generated.

├── llm4gcd.py              # Main script: inference + evaluation
├── run.sh                  # Automation wrapper (parallel jobs, .env loading)
├── requirements.txt        # Python dependencies
├── .env                    # API key
│
├── data/
│   ├── AI-Human-Generated-Program-Code-Dataset.jsonl
│   ├── AI-Human-Generated-Program-Code-Dataset*.csv
│   ├── files/              # Individual .txt code samples
│   └── README.md
│
├── output/
│   ├── {strategy}/
│   │   └── {org}/{model}/
│   │       └── iter_{n}/
│   │           └── rs_{idx}.json   # One file per classified sample
│   ├── results_summary.json        # Aggregated metrics (generated by report)
│   └── cohen_kappa.csv             # Inter-model agreement matrix
│
├── logs/                   # Per-job logs from run.sh
│
│
└── related_works/
    ├── ChatGPT-Code-Detection/   # Oedingen et al. 2024
    ├── GPTSniffer/
    └── DetectCodeGPT/

# 1. Install dependencies
pip install -r requirements.txt

# 2. Create your .env file
echo 'OPENROUTER_KEY=sk-or-...' > .env

# Run inference for one strategy across selected models
python llm4gcd.py infer \
  --strategy classification \
  --models "anthropic/claude-sonnet-5" "openai/gpt-5-mini" \
  --iter 3 \
  --temperature 0.1

# Evaluate existing output files
python llm4gcd.py eval --strategy classification --iter 1

# Generate full report → output/results_summary.json + output/cohen_kappa.csv
python llm4gcd.py report

--models accepts either short aliases (e.g. claude) or full OpenRouter IDs (e.g. anthropic/claude-sonnet-5). Already-completed output files are skipped automatically — runs are resumable.

# Full run: all models, all strategies, 3 iterations, 8 parallel workers
./run.sh --iters 3 --workers 8 --temperature 0.1

# Subset run
./run.sh \
  --models "anthropic/claude-sonnet-5 openai/gpt-5-mini" \
  --strategies "classification classification_stylo_semantic" \
  --iters 1 --workers 4

# Evaluate only (no inference)
./run.sh --eval-only

# Inference only, skip report
./run.sh --no-report --iters 2

All options:

Each classified sample is saved as a JSON file at:

output/{strategy}/{org}/{model}/iter_{n}/rs_{idx}.json

Example record:

{
  "id": 19,
  "language": "Java",
  "problem_number": 1,
  "human_writing": true,
  "code": "...",
  "model": "ChatGPT",
  "response": "no",
  "explanation": "..."
}

response is "yes" (LLM-generated) or "no" (human-written). model is only present when human_writing is false.

If you use this code, please cite:

@inproceedings{djire2026llm4gcd,
  author    = {Djire, Alberick Euraste},
  title     = {Code Detectors Have a Half-Life: Obsolescence and Metric Illusions
               in LLM-Generated Code Detection},
  booktitle = {Proceedings of the 41st IEEE/ACM International Conference on
               Automated Software Engineering (ASE '26)},
  year      = {2026},
  month     = {October},
  address   = {Munich, Germany},
  pages     = {3},
  publisher = {ACM},
  doi       = {10.1145/3832783.3844566},
  isbn      = {979-8-4007-2882-2},
}

This project is licensed under the MIT License.