BRENNEN

Industries / Government, education and community / Lobbying

Data

Lobbying data work turns filings, meeting logs and bill texts into databases that reporters, researchers and the public can search and compare.

6

Milestones achieved, from 1998 to 2026.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1998

    Federal filings become a database

    The Center for Responsive Politics, now OpenSecrets, built a database of US federal lobbying filings starting with 1998; it standardises the spelling of names, rolls subsidiaries up to their parent companies and links reports to bills and agencies.

    LessonThe raw filings need heavy cleaning before they can be compared.

    opensecrets.org

    Credit

    Center for Responsive Politics (OpenSecrets)

  2. 2013

    LobbyPlag matches lobby texts

    Volunteers compared leaked lobby papers with amendments proposed in European Parliament committees on the EU's data protection law, the General Data Protection Regulation, to show "which changes proposed by lobbyists went straight into amendments", and asked the public to help check about 1,700 more.

    LessonThe tracing worked only because papers had leaked and volunteers checked them by hand.

    web.archive.orggithub.com

    Credit

    OpenDataCity; Europe versus Facebook; La Quadrature du Net and others for the papers

  3. 2014

    EU meeting logs go public

    From 1 December 2014, EU Commissioners and their offices must publish their meetings with lobbyists within two weeks, under a Commission decision. LobbyFacts, run by the campaign groups Corporate Europe Observatory and LobbyControl, collects these logs and the Transparency Register and keeps register history back to February 2012 that it says "is no longer available anywhere else".

    eur-lex.europa.eulobbyfacts.eu

    Credit

    European Commission; Corporate Europe Observatory and LobbyControl

  4. 2019

    Copycat bills traced at scale

    Reporters ran nearly 1 million bills from all 50 US states and Congress through a text-similarity program and found at least 10,000 bills almost entirely copied from model legislation over eight years, more than 2,100 of which became law.

    LessonModel bills appear in no lobbying disclosure, and the method caught only near-verbatim copying.

    usatoday.com

    Credit

    USA TODAY, The Arizona Republic and the Center for Public Integrity

  5. 2020

    Filings linked to bills

    In Song Kim and Dmitriy Kunisky published a dataset linking US bills and their sponsors to the interest groups that lobbied on them, built from more than two million text descriptions in lobbying reports; Kim's LobbyView site at MIT now lists more than a million reports. The same year the Sunlight Foundation, which built open-government data tools and says it served "more than 4.2 billion API calls", shut down.

    LessonLobbying data depends on grant-funded labs and nonprofits, which can close.

    doi.orglobbyview.orgsunlightfoundation.com

    Credit

    In Song Kim (MIT); Dmitriy Kunisky

  6. 2026

    One US filing site

    The Senate's lobbying data page announced in early 2026 that it would close after 30 June 2026, and search for federal lobbying reports moved at the end of July 2026 to LDA.gov, a joint House and Senate site whose public API serves every filing as structured data.

    lda.govstateandfed.comweb.archive.org

    Credit

    the Clerk of the House; the Secretary of the Senate

Watch for

  1. EU register cross-checks payments

    Since 5 June 2025 the EU Transparency Register automatically compares the amounts clients declare paying consultancies with what the consultancies declare receiving, and its 2025 annual report says the linking "has greatly helped with consistency and compliance".

    Achieved whenAn outside body, such as the European Ombudsman, the European Court of Auditors or an academic study, measures fewer mismatches.

    transparency-register.europa.eutransparency-register.europa.eu

    Claimed by

    Joint Transparency Register Secretariat, June 2025 and May 2026

  2. State data joins OpenSecrets

    OpenSecrets was formed in June 2021 by merging the Center for Responsive Politics with the National Institute on Money in Politics, whose FollowTheMoney site held US state data, in order to combine federal, state and local data. FollowTheMoney now says its data runs through the 2024 election year and that the site "isn't maintained as we integrate with OpenSecrets".

    Achieved whenOpenSecrets publishes state lobbying data for 2025 and later.

    commondreams.orgfollowthemoney.org

    Claimed by

    OpenSecrets, June 2021, restated on FollowTheMoney in November 2026

Open

  1. 01

    One format for lobby registers

    An open, documented data format for lobby registers, converters that put the US, EU, Canadian, Irish, French and German registers into it, and a match of the same organisations across them.

    The brief
    Credit · unclaimed
  2. 02

    Ground truth for lobby footprints

    A public set of pairs, each linking a passage in a lobby group's position paper to an amendment in the European Parliament, labelled by two reviewers as copied, reworded or unrelated, so that methods for tracing lobby text into law can be scored.

    The brief
    Credit · unclaimed

Open 01 · the brief

One format for lobby registers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An open, documented data format for lobby registers, converters that put the US, EU, Canadian, Irish, French and German registers into it, and a match of the same organisations across them.

Why it is still open

International IDEA assessed 16 registers in 2024 and found that most let the public download their data but that "formal coordination between lobbying registers is of pivotal importance – and so far largely unaddressed". Transparency International EU found in 2024 that the countries it studied lack unique identifiers for officials and companies, "which renders these datasets almost impossible to link".

idea.inttransparency.eu

Built with what exists

The LDA.gov API and the other registers' downloads; company identifiers such as the Legal Entity Identifier (LEI) where they exist; an AI model to match organisation names across languages; a spreadsheet for two reviewers.

  1. First step

    Map the fields of the US and EU registers to a draft format and match the 100 largest US lobbying clients to EU register entries.

  2. Done looks like

    A published format with converters for six registers; organisations matched across registers with at least 95% precision on 300 pairs checked by two reviewers; the combined data refreshed monthly for six months.

Open 02 · the brief

Ground truth for lobby footprints

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public set of pairs, each linking a passage in a lobby group's position paper to an amendment in the European Parliament, labelled by two reviewers as copied, reworded or unrelated, so that methods for tracing lobby text into law can be scored.

Why it is still open

Researchers at EPFL in Switzerland who traced alignments between lobby groups and members of the European Parliament wrote in 2024 that "there exists no large database of verified MEP-lobby alignments", so they could check their methods only indirectly, against retweets and meeting records.

aclanthology.org

Built with what exists

Position papers from the EU Transparency Register and lobby groups' websites; Parliament's published committee amendments; LobbyPlag's GDPR data as a first source of known copies; an AI model to propose candidate pairs; a spreadsheet for two reviewers.

  1. First step

    Label 100 candidate pairs drawn from LobbyPlag's GDPR amendments and measure how often two reviewers agree.

  2. Done looks like

    2,000 labelled pairs across at least five laws; the two reviewers agree on at least 80% of labels; existing methods scored against the set; data and guidelines published.