Build research-ready datasets from EU legislative data — recitals, articles, points, and the network between them.

eurlex-builder is a configurable Python pipeline that turns the EUR-Lex / Cellar corpus into provision-level Parquet tables for quantitative research. It fetches every directive, regulation, decision, or communication you ask for, extracts text at the granularity you need (whole article, numbered paragraph, or lettered point), collects metadata + inter-document relations, and ships the result as one DuckDB file plus four Parquet tables.

It is, to our knowledge, the first open-source tool to extract EU legislative text at configurable sub-article granularity across EUR-Lex's heterogeneous document formats — from modern HTML to 1970s text-only pages to scanned PDFs.

The package accompanies Seidl and Kosti (2026), "Mapping Europe's Digital Acquis: A Granular History of EU Digital Policymaking", SocArXiv preprint, doi:10.31235/osf.io/7hv94_v1. Replication package: osf.io/suf9m.

• Sub-article granularity. One config switch chooses whether each article is one row, one row per numbered paragraph (Art. 5(1)), or one row per lettered point (Art. 5(1)(a)) — following the EU Joint Practical Guide.
• Six HTML structures + PDF fallback. Automatic detection across EUR-Lex's HTML eras (Standard OJ, Manual CSS, class-based, text-only, consolidated-norm, classless) with Docling/pymupdf for older docs that have no machine-readable HTML.
• Two query modes. Fixed (give us a list of CELEX IDs / procedure numbers) or descriptive (date range + doc types + optional EuroVoc keyword filter).
• Inter-document relations. Citations, amendments, legal basis, repeals, consolidations — all in one table for network analysis.
• Guarded translation built in. Non-English-only documents are translated via pinned Helsinki-NLP Opus-MT revisions, separately at document level (works.full_text) and per-unit (text_units.text_translated). Token-bounded retries and quality guards reject degenerate output without replacing the source text.
• Reproducible + resumable. A single YAML defines the dataset; a checkpoint table in DuckDB makes the pipeline restart-safe and incremental.
• Auditable runs. DuckDB stores every validated config, its SHA-256 hash, runtime versions, and completion status; validate checks structural integrity without modifying the database.
• Parallel mode. Multi-threaded fetching with parallel: true; sequential writes keep DuckDB contention-free.

Full-text retrieval works for every document type; granular extraction does not. Whatever the pipeline reaches, works.full_text is fetched and stored — the caveats here concern the decomposition into text units. Regulations, directives, and decisions are validated against official documents and are what the accompanying paper is built on. Communications extract well but have known gaps; proposals (PC) and staff working documents (SC) lose units to modern templates, and case law (sector 6) yields zero text units — silently (and is reachable only in fixed mode). Read Verification status before running anything outside the validated three.

pip install "eurlex-builder[all]"

Create a config.yaml. You can either request specific acts by CELEX ID (fixed mode) or search by date range and document type (descriptive mode):

Fixed mode — specific acts:

metadata:
  project_name: "GDPR + AI Act"

data:
  mode: "fixed"
  celex_ids:
    - "32016R0679"    # GDPR
    - "32024R1689"    # AI Act

processing:
  text_extraction:
    article_granularity: paragraph

output:
  output_directory: "./output"

Descriptive mode — all acts matching a search:

metadata:
  project_name: "EU digital regulations 2020–2025"

data:
  mode: "descriptive"
  document_types:
    - "regulation"
    - "directive"
  start_date: 2020-01-01
  end_date: 2025-12-31
  filter_keywords:           # optional EuroVoc filter
    - "artificial intelligence"
    - "data protection"

processing:
  text_extraction:
    article_granularity: paragraph
  parallel: true
  max_workers: 8

output:
  output_directory: "./output"

Run:

eurlex-builder run config.yaml

You'll get output/eurlex_builder.duckdb plus four Parquet files: works.parquet, text_units.parquet, relations.parquet, eurovoc.parquet. The database filename is always eurlex_builder.duckdb — it's the output_directory that separates projects, so give each config its own (e.g. ./output/digital-regs).

import polars as pl
text_units = pl.read_parquet("output/text_units.parquet")

# All paragraph-1 rows of GDPR Article 5
text_units.filter(
    (pl.col("celex_id") == "32016R0679")
    & (pl.col("type") == "article")
    & (pl.col("number") == "5")
)

Requires Python 3.11+.

# Core only (HTML extraction + parquet output)
pip install eurlex-builder

# With PDF support (Docling + pymupdf)
pip install "eurlex-builder[pdf]"

# With translation (Opus-MT via Hugging Face)
pip install "eurlex-builder[translate]"

# Everything (recommended for production runs)
pip install "eurlex-builder[all]"

For development, install from a checkout instead:

git clone https://github.com/tseidl/eurlex-builder.git
cd eurlex-builder
python -m venv .venv && source .venv/bin/activate

# Core only (HTML extraction + parquet output)
pip install -e .

# With PDF support (Docling + pymupdf)
pip install -e ".[pdf]"

# With translation (Opus-MT via Hugging Face)
pip install -e ".[translate]"

# Everything (recommended for production runs)
pip install -e ".[all]"

# Plus dev tools (pytest)
pip install -e ".[dev]"

A single YAML file defines the dataset. Only the data section is required; everything else has sensible defaults. Unknown keys and malformed CELEX IDs are rejected rather than silently ignored.

data:
  mode: "fixed"
  celex_ids:
    - "32016R0679"
  procedure_numbers:
    - "2021/0106"      # resolved to CELEX IDs via SPARQL

data:
  mode: "descriptive"
  document_types:
    - "regulation"
    - "directive"
    - "decision"
    - "communication"
  start_date: 1979-01-01
  end_date: 2026-04-30
  filter_keywords:
    - "artificial intelligence"
    - "digital"
  include_corrigenda: false
  include_consolidated_texts: false

EU drafting convention recognises three operative levels (Joint Practical Guide of the Council, Commission, and Parliament):

Article 5
  Paragraph 5(1)
    Point 5(1)(a)
    Point 5(1)(b)

eurlex-builder lets you emit each as a row:

A short preamble before paragraph 1. gets its own row with paragraph_num = "0". Articles in amending acts (lists of edits to other regulations) are flagged subtype = 'amendment_item'; substantive replacement text within them stays untagged so downstream classifiers can keep it.

Recitals are emitted identically across all granularity settings.

For legacy HTML, numbered table rows are accepted as recitals only when they form a credible drafting sequence. Nested numbered tables remain inside their parent recital, embedded sibling lists are excluded only when the main sequence resumes, and a table continuation is joined to preceding paragraph recitals only when every earlier number is present. For legacy PDFs, a page-break fragment such as (1) of Regulation ... remains part of the active recital instead of starting a false numbered sequence.

Run the pipeline.

The pipeline resumes by default, skipping checkpointed docs. After an interrupted --fresh rebuild, re-run without --fresh to resume the new rebuild rather than clearing its completed checkpoints again. --limit is intended for canary runs and cannot be combined with --fresh.

If any document attempted in this invocation fails, the command exits with status 1 after saving and exporting successful results. Its run manifest records complete_with_failures. Historical failures outside the attempted selection do not affect this exit status. Use --retry-failed to retry failures. A Python call to Pipeline.run() returns RunResult(processed, failed) with counts for the invocation; discovery and other fatal errors still raise.

Keyword filtering stops with an error if no EuroVoc concepts match or all matches are rejected. Set filter_keywords: [] explicitly for an unfiltered date/type search. Failed selection leaves existing checkpoints intact.

Translate non-English content. Resumable; skips already-translated rows.

Translation is interruption-safe at row level. Rejected outputs remain in the source language and are recorded in _translation_failures; an ordinary resume skips unchanged current-policy rejections, while --retry-rejected tries them again. A source-text or policy-version change invalidates the corresponding ledger entry automatically.

Add post-hoc metadata via SPARQL — no re-fetching content. Adds entry-into-force date, ELI, author institutions, subject matter, procedure type/reference/legal basis, EuroVoc descriptors, and repeal relations.

Completion is tracked separately for metadata, relations, and eurovoc, so categories can be run independently and resumed later. A successful response includes category completion sentinels; only then can an empty refresh remove stale values. Databases enriched by older releases are migrated to all-category checkpoints from enriched_at instead of unexpectedly re-fetching every work.

Print checkpoint summary (processed / failed counts + failure reasons).

Run read-only integrity checks for checkpoint/work consistency, orphaned rows, stable unit identity, duplicate keys/order, translated-fallback markers, and degenerate stored translations. The command exits non-zero when an error is found and reports expected data gaps and recorded translation rejections as warnings.