~/research/policy-evidence-kg

Background

Research-assistant project, built at the supervisor’s request — the Policy Evidence Knowledge Graph Platform, a monorepo of three interlocking modules that support evidence-grounded question answering over climate-policy scientific literature.

Architecture

The platform is organised as three loosely-coupled services that talk to each other over HTTP:

1. KG backend (Python + Neo4j)

2. Constrained Q&A Agent

The agent is a constrained summarizer, not a free chat bot. It can only answer based on KG retrieval results.

3. Web frontend (React + Vite)

Anti-hallucination control

Stacked validation layers ensure full answer traceability:

LayerWhat it checks
Citation whitelistAgent can only cite papers present in the KG.
Directional conflict detectionClaims contradicting known KG facts are flagged.
Unit normalizationNumeric units normalized before comparison; mismatches surface as conflicts.
NLI / LLM-as-judge post-hoc reviewOptional fallback validation on assembled answers.

Dual-track confidence scoring

Two independent scoring tracks, weighted to produce an overall score:

User feedback snapshots drive OLS calibration and feature-weight hot-updates so the scoring function adapts to real usage.

End-to-end pipeline

PDF parsing → finding extraction → snippet-claim alignment
            → Neo4j ingestion → LLM-as-judge auto-evaluation
              (accuracy, traceability, consistency)

The auto-evaluation module scores Agent responses across three dimensions to support iterative system improvement.

Top-level chrome (across all pages)

Present on every route:

Home / Landing page (/)

Climate Change Wiki home — the Knowledge Graph entry card showing the X → Y traversal pattern (Topic → Policy Instrument → Outcome → Evidence Papers), with Drivers and Outcomes root categories rendered as image cards (Policy and Regulation 287 papers, Physical Climate Shock 142 papers with sub-categories, Technology and Market Shifts 156 papers, Other Drivers 12 papers; Outcomes row partially visible below).

The home view exposes the platform’s core schema: every question begins by choosing a Topic (X) → Policy Instrument → Outcome (Y), which resolves to a set of Evidence Papers. Drivers and Outcomes are the two top-level taxonomies — drivers split into 4 root categories (with Physical Climate Shock further broken into 6 sub-categories: Acute Extreme Weather, Wildfire & Drought, Chronic Temperature Anomalies, Chronic Sea Level Rise, Resource & Ecological Constraint, Other), and outcomes into 8. Each card surfaces the underlying paper count (287 / 142 / 156 / 12 …) so the user can gauge evidence density before drilling in.

Functions on this page:

Q&A interface (/qa)

Purpose: the agent’s primary surface. User asks a question, the system picks a tier (T1 – T5) based on intent classification, renders the appropriate structured answer.

Functions on this page:

T1 — Concept retrieval (“What is X? / What kinds of X exist?”)

Coverage: ~10% of incoming queries. Output contract: a knowledge-card page only — the agent’s job is to answer the concept and nothing more, with citations to multiple references.

For concept-style questions (e.g. “what is carbon pricing?”), the agent renders the answer as a structured multi-card page rather than a flat paragraph. The walkthrough below goes through every card on the page in numerical / page-order, with each screenshot embedded at the first card it shows.

C0 — Core definition + canonical citations

T1 concept-card answer page for the query 'what is carbon pricing?'. Left sidebar: conversation history grouped Today / Earlier (carbon-tax impact, library coverage, GDP impact, contradictory conclusions, etc.). Right pane: the answer renders as a T1 Concept-Retrieval page — purple Concept-Card header, under it a C0 Core-Definition card with a one-sentence essence (carbon pricing internalises the environmental cost of GHG emissions into a visible price signal) and two cited paragraphs (Stiglitz & Stern 2017 — High-Level Commission on Carbon Prices; Aldy & Stavins 2012 — J. Economic Perspectives).

The C0 card leads with a one-sentence essence (“carbon pricing turns GHG-emission environmental costs into a visible price signal, making emitters pay per ton — using market mechanisms to drive reductions”), then expands into canonical academic definitions with per-paragraph citation chips. Every claim is anchored to a specific paper in the KG — no free-form prose without provenance. The page renders with a fixed slot order, so the cards below this one always follow in the same sequence (only their content varies per query).

C2 — Global implementation landscape

C2 Global-Implementation-Landscape card with four KPI tiles — 75+ jurisdictions, ~24% global GHG coverage, $104B 2023 carbon-price revenue (World Bank 2024), $25/t median price (well below the social cost of carbon). Below the tiles, jurisdictions grouped by geography: Europe (Sweden, EU (ETS+CBAM), UK, Norway, France), North America (BC Canada, California, RGGI 11-state), Asia-Pacific (China national + 8 pilots, Korea, New Zealand, Tokyo) — each a clickable chip that filters the rest of the cards. Below it, C3 Key-Timeline card shows chronological policy milestones: 1991 Sweden first global carbon tax, 2005 EU ETS Phase I, 2008 BC fiscally-neutral carbon tax, 2009 RGGI first regional ETS in North America, 2013 California C&T, with a horizontal scrollbar. Footnote names Pigou (1920) and Coase (1960) as the theoretical roots, plus the US 1990 Clean Air Act SO₂ trading as a precursor.

C2 surfaces the empirical scale of the concept — 75+ jurisdictions, ~24% global GHG coverage, $104B 2023 revenue, $25/t median price (well below the social cost of carbon). Below the KPI tiles, jurisdictions are grouped by geographic cluster (Europe / North America / Asia-Pacific) and rendered as clickable chips. Clicking China filters every other card on the page to China-related evidence; clicking again clears the filter.

C3 — Key timeline

Chronological policy milestones rendered as a horizontal time-strip: 1991 Sweden first global carbon tax · 2005 EU ETS Phase I · 2008 BC fiscally-neutral carbon tax · 2009 RGGI · 2013 California C&T and beyond. A footnote names the theoretical roots (Pigou 1920, Coase 1960) and the US 1990 Clean Air Act SO₂ trading as a precursor. Clicking any milestone chip opens a side drawer with the evidence cluster for that era (papers tagged within the milestone’s date window).

C4 — Concept relation graph

C4 Concept-Relation-Graph card: the queried concept (Carbon Pricing) at the centre, surrounded by three lanes — parent concepts (Pigouvian Taxation, Externality Internalization, Market-Based Instrument), companion mechanisms (Revenue Recycling, MRV / monitoring & verification, allocation — free vs auction), and risks / side-effects (Carbon Leakage, Regressivity, Social Cost of Carbon). Below it, C5 Library-Research-Coverage card: 12 / 50 papers study carbon pricing, time span 2008–2025, 9 countries, 75% SSCI top journals; a method-distribution stacked bar shows DiD 5 / SC 3 / CGE 2 / theory 2 (8 of 12 are causal-identification studies); top country coverage lists EU (ETS) 5 papers, BC Canada 2, California 2, China (pilots) 2, other 1; a warning at the bottom names coverage blind-spots — developing countries (India, SEA, Latin America), post-CBAM causal evidence (from 2026+), and long-term effects of China's national ETS (>4 years).

C4 places the concept inside its conceptual neighbourhood — parents (Pigouvian Taxation, Externality Internalization, Market-Based Instrument), companions (Revenue Recycling, MRV, free-vs-auction allocation), risks (Carbon Leakage, Regressivity, Social Cost of Carbon) — drawn directly from the KG’s typed edges. Each node is clickable: clicking Revenue Recycling routes to a fresh T1 query for that companion concept while preserving the parent-concept breadcrumb.

C5 — Library research coverage

The meta-card. It quantifies how well the library itself covers the concept (12 of 50 papers, time span 2008–2025, 9 countries, 75% top-journal share, method distribution DiD 5 / SC 3 / CGE 2 / theory 2 — 8 of 12 are causal-identification studies). A Top country coverage list shows EU (ETS) 5 papers, BC Canada 2, California 2, China (pilots) 2, other 1. A warning band at the bottom explicitly names coverage blind-spots — developing countries (India, SEA, Latin America), post-CBAM causal evidence (from 2026+), long-term effects of China’s national ETS (>4 years). This is the inverse of a hallucination-prone system: it surfaces what it doesn’t know before the user has to ask.

Evidence cards — paper-by-paper

Lower portion of the T1 answer: evidence cards for individual papers — California cap-and-trade significantly reduced emissions but raised distributional concerns (US California, CA C&T); Murray & Rivers 2015 (Energy Policy) — BC carbon tax empirical synthesis, 5%–15% emission reductions with near-neutral economic impact via revenue recycling (CA BC, BC Carbon Tax); Cao Ho Hu Jorgenson 2021 (JEEM) — China ETS pilot CGE assessment with sector heterogeneity (CN China, China ETS pilots); Aldy & Stavins 2012 (J. Econ. Perspectives) — foundational theory review on price vs quantity instruments. Below them, C7 Want-to-Know-More chips: 'how effective is carbon pricing at emission reduction?', 'carbon tax vs ETS — which works better?', 'does carbon pricing cause carbon leakage?', 'how to design revenue recycling optimally?', 'how well has China's national ETS performed so far?', 'why are carbon prices generally low?' (each chip routes to the appropriate T2 answer type). Bottom-left: an Answer-Confidence bar at 85/100, with provenance breakdown — 3 authoritative external definitions · 12 internal-library citations · external World Bank data.

After C5, the page renders paper-by-paper evidence cards — one per relevant in-library paper, each carrying the headline finding, country / tool tags, and a click-through into the full Evidence-Library entry. Examples in the screenshot: California cap-and-trade significantly reduced emissions but raised distributional concerns (CA C&T); Murray & Rivers 2015 (Energy Policy) — BC carbon tax empirical synthesis (5%–15% emission reductions, near-neutral economic impact via revenue recycling); Cao Ho Hu Jorgenson 2021 (JEEM) — China ETS pilot CGE assessment; Aldy & Stavins 2012 (J. Econ. Perspectives) — foundational theory review.

C7 — Follow-up question chips

C7 is the want-to-know-more card — the system’s understanding of what kinds of questions naturally come next. In the example: “how effective is carbon pricing at emission reduction?”, “carbon tax vs ETS — which works better?”, “does carbon pricing cause carbon leakage?”, “how to design revenue recycling optimally?”, “how well has China’s national ETS performed so far?”, “why are carbon prices generally low?”. Each chip is typed — the prefix routes to the appropriate answer-tier (mostly T2 here, since C7’s job in T1 is to bridge from definition → impact / comparison).

Confidence bar (page footer)

At the very bottom: an Answer Confidence bar at 85/100, with a provenance breakdown — 3 authoritative external definitions · 12 internal-library citations · external World Bank data. The bar is always pinned at the answer footer (across T1/T2/T3); how that 85 decomposes is described under T1 confidence breakdown below.

T1 routing — when the agent picks concept-card mode

Interactive behaviour (not visible in static screenshots)

T1 confidence breakdown

The 85/100 in this example decomposes into the dual-track scoring described earlier:

T2 — Synthesis questions (impact / comparison)

Coverage: ~40% of incoming queries — by far the largest bucket. Anything of the form “does X affect Y?”, “which X is better?”, “how do X+Y stack?”, “how is X going in China?”. Retrieval is two-layer:

  1. Layer 1 — synthesis-literature corpus. First pass searches a curated Zotero-backed pool of review / meta-analysis papers for the headline answer (the “what does the consensus say” layer).
  2. Layer 2 — per-paper corpus. Second pass searches the full per-paper library for the supporting specific findings (the “what does each underlying study say” layer).

Both layers are then integrated into a single structured answer. The walkthrough below goes through every component of that answer in page order, with each screenshot at the first component it shows.

Source-quality dashboard (top of the page)

T2 answer header for the query 'what tools are available for emission reduction, and how effective are they?', labelled Literature-Review Retrieval · Zotero · Climate-tools synthesis articles (50 papers). Top: a Literature-Source panel showing Zotero / Climate-tools synthesis articles / 50 papers / 2011–2026 year range, with five KPI tiles — Total papers 50, directly on mitigation policy 28, with quantitative effect ranges 15, Meta-analysis reviews 4, year span 2011–2026. Below: a horizontal bar chart of tool-type coverage by paper count — Carbon Pricing (Carbon Tax + ETS) 18 papers, Subsidies & Clean Energy 6, International Coordination & Trade Policy 5, Technology Innovation Policy 4, Energy-Efficiency Policy 3, Carbon Offsets (CDM) 2, CDR (carbon removal) 2. Below the chart, section '01 — Carbon Pricing' opens with the strongest evidence base (18 papers), a row for Carbon Tax tagged Direction Consensus ↓ Reduction, with a meta-analysis effect range of −4% to −21%; primary citation chip: Döbbeling-Hildebrandt et al. 2024 Nat.Comm., with a one-line abstract explaining that the meta-analysis assessed 80 effect evaluations across 21 carbon-pricing mechanisms — 17 of which produced immediate and significant reductions of −5% to −21% (publication-bias-corrected −4% to −15%); price levels are mostly low across mechanisms, higher prices yield larger effects.

T2 answers lead with a source-quality dashboard — not just what’s the answer, but what’s the evidence base. The page header reads Literature-Review Retrieval · Zotero · Climate-tools synthesis articles (50 papers) — explicitly naming the corpus this answer is drawn from. Five KPI tiles below it: Total 50 · directly on mitigation policy 28 · with quantitative effect ranges 15 · meta-analyses 4 · year span 2011–2026. The user knows the size and shape of the evidence base before any claim is made.

Tool-type coverage chart

Directly under the dashboard (same screenshot), a horizontal bar chart breaks down the 50 papers by tool category — carbon pricing dominates (18 papers), then subsidies & clean energy (6), international coordination & trade (5), tech innovation (4), energy efficiency (3), carbon offsets / CDM (2), CDR (2). The visual ordering makes evidence density per tool obvious at a glance, and tells the user up-front which tool sections below will be evidence-rich vs thin.

Section 01 — Carbon Pricing

The first per-tool section opens below the coverage chart. 01 — Carbon Pricing gets the strongest-evidence-base banner (18 papers). Its top row is Carbon Tax — direction consensus ↓ reduction, with a meta-analysis effect range of −4% to −21%. The primary citation chip is Döbbeling-Hildebrandt et al. 2024 Nat.Comm., whose one-line abstract explains the meta-analysis covered 80 effect evaluations across 21 carbon-pricing mechanisms — 17 of which produced immediate, significant reductions of −5% to −21% (publication-bias-corrected −4% to −15%); price levels are mostly low across mechanisms, higher prices yield larger effects.

Sections 02 – 07 (other tool categories)

The full T2 answer continues with parallel sections for every tool category that has ≥ 2 papers in the corpus. Each section opens with a strongest-single-citation banner, lists 2–4 supporting findings, and ends with an internal sub-table for any sub-tools the category contains (e.g. carbon pricing splits into carbon-tax and ETS rows; subsidies split by technology).

Cross-tool comparison table

Two large cards. Top — Cross-Tool Comparison Table with columns Tool / Direction / Quantitative effect / Key citations / Core caveat, rows for: Carbon Tax (↓, −4% to −21% across schemes; citations D-H 2024, Metcalf & Stock 2023, Köppl 2023; caveat: prices generally low, regressivity needs revenue recycling); ETS (↓, confirmed but magnitude design-dependent; Verde 2020, Borghesi 2023; caveat: stacked policies may have reverse effects per Perino 2025); Renewable-energy subsidies (↓, dynamic cost low — Wind PTC MVPF highest; Hahn 2025, Gillingham & Stock 2018; caveat: EV subsidies MVPF lower than Wind PTC, static cost highest); Energy-efficiency policy (↓, positive but systematically over-estimated; Gerarden 2017, Hahn 2025; caveat: engineering forecasts biased, need place-specific diagnostics); Carbon offsets / CDM (↑ in red — net emission increase / additionality failure; Calel et al. 2025; caveat in red: >52% lack additionality, global CO2 net increase); Climate clubs (↓, theoretically high but needs sanctions; Nordhaus 2015, Clausing & Wolfram 2023; caveat: politically hard, subsidy race risk); Import tariffs / CBAM (↓, 5.4 Gt reduction palm-oil case · $15/tCO2; Hsiao 2026 Econometrica; caveat: needs coordination + commitment, otherwise weak); CDR (~, complementary tool — lowers total cost; Edenhofer et al. AREE; caveat: non-permanence risk, governance immature); Policy mix (↓↓, better than single tools; Blanchard et al., Peñasco 2021, Stiglitz 2019; caveat: design complex, stacking effects must be careful per Perino 2025). Bottom — Questions this corpus can't answer (evidence gaps) — 4 cards: Developing-country-specific evidence (evidence heavily concentrated in OECD; China / India / SEA carbon-pricing causal evidence severely lacking; Döbbeling-H 2024 explicitly names this gap); Policy-mix causal identification (most existing evidence is single-tool; policy-mix incremental effects, mix vs sum, lack rigorous causal studies; Perino 2025 framework is theoretical, needs empirical test); Long-term 10y+ effects (most causal studies cover 3–8 years post-implementation; investment leakage, technology lock-in long-term effects under-studied; Verde 2020 names this as a top future-research direction); Climate-adaptation tools (the corpus is almost entirely mitigation-focused; adaptation-policy effect evaluation basically absent — only analytical frameworks like Carleton et al. / Ferreira et al., no systematic policy-effect synthesis).

After sections 02–07, the page surfaces a cross-tool comparison table — the synthesising layer of the whole answer. Every tool gets a row with direction (↓ reduction / ↑ increase / ~ neutral), quantitative range, the strongest citation set, and a core caveat — the one-line “but…” a reader would otherwise miss. The CDM row is intentionally rendered red: the corpus finds it has a failure mode (52%+ lack additionality → net CO₂ increase globally), and the system flags it visually rather than burying it in prose. The Policy-mix row at the bottom — direction ↓↓ — shows that tool mixes outperform single tools, with citations to Blanchard et al., Peñasco 2021, Stiglitz 2019.

Evidence-gaps card (page footer)

Below the comparison table, the evidence-gaps card is the meta-honesty layer — four named blind-spots, each with a citation supporting why we know we don’t know:

By placing these gaps directly after the comparison table, the agent makes its own coverage limits part of the answer rather than letting them surface as silent hallucinations downstream.

Filters and interactivity

T2 routing — when this mode fires

Cross-paper synthesis logic

T3 — Internal library metadata (“how many papers? do we have AER articles?”)

Coverage: ~10% of incoming queries. Purpose: answers questions about the library itself — “how many papers do we have?”, “do we have any AER articles?”, “how many papers on China ETS?”, etc. Not LLM-generated.

Implementation: the answer is looked up from a pre-prepared metadata sheet (library_meta_questions_v1.xlsx) maintained by the curator. The sheet enumerates the expected meta-questions and their canonical answers; the system matches the user prompt to the closest sheet row. This makes T3 deterministic and fast — and avoids the LLM hallucinating numbers about the corpus.

Functions on this page:

T4 — Out-of-scope / hard questions the system can’t answer yet

Coverage: ~20% of incoming queries. Current state: these are the questions the system can’t answer — anything that needs external knowledge not in the corpus, ephemeral facts, specific local-government queries outside the climate-policy literature, or questions about adjacent domains the library doesn’t cover. Examples from the spec:

Current behaviour: the chatbox responds with a clear “I can’t answer this right now” rather than fabricating. This is the conservative-by-design refusal — a hardcoded sentinel response that explains why the question is out-of-scope. Long-term: the team hopes to drop the refusal rate as the library expands and external-knowledge tools (live web retrieval, government-policy crawlers, real-time fact lookup) come online. Not solved yet.

Functions on this page:

Knowledge Graph visualization (/graph)

Purpose: force-directed view of the KG. Used to explore neighbourhoods around any node (Paper / Finding / Driver / Outcome / Concept).

Functions on this page:

Evidence Library (/evidence)

Purpose: browseable corpus index. Two views over the same underlying records.

Functions on this page:

Paper detail page (/papers/:id)

Purpose: all extracted info for a single paper.

Functions on this page:

Finding detail page (/findings/:id)

Purpose: all metadata for a single extracted finding.

Functions on this page:

Topic / taxonomy browser (/topics)

Purpose: explore the platform’s taxonomy (the X → Y schema) as a tree rather than as cards on the home page.

Functions on this page:

Data ingestion (/ingest)

Purpose: admin entry point for adding new papers to the library. Upload PDFs, the system extracts findings via LLM, builds an evidence index, and writes everything into Neo4j — all in one click. Duplicate detection prevents re-ingesting the same paper twice.

Ingest page for the Climate Change Wiki. Header reads 'Import new papers — pick PDF files, confirm, one-click ingest; the system runs the full processing flow automatically'. Three live KPI tiles: 163 papers · 2170 findings · 4524 Neo4j nodes (with a Refresh button). Left column: a large drag-and-drop zone ('Drop PDF files here, or click to select; multi-select supported'), and below it the four-step processing flow — (1) Parse PDF — extract text + structure, (2) Extract findings — LLM identifies drivers + evidence, (3) Build evidence index — construct citable snippets, (4) Write to KG — persist to Neo4j; footnote: only newly-uploaded papers are processed, library content is never re-processed. Right column: 'Pending papers — 7' panel with a dedup banner (✓ New 4, ✕ Duplicate 3, plus a 'Remove all duplicates' button). Three papers are marked ✕ (red, content-duplicate, already in library): davis et al 2014 cash for coolers, dechezleprêtre et al 2025 fighting climate change international attitudes, cole et al 2023 policies for electrifying the light duty vehicle fleet. Four papers marked ✓ (green, new): 2507.17186v2 (10.3 MB), agentic large language models for day to day route choices (1) (4.8 MB), IJCSE V12I4P103 (0.2 MB). A grey disabled submit button at the bottom reads 'Resolve 3 duplicates before ingesting'.

Functions on this page:

Admin & curation tools (admin-only)

Purpose: post-hoc fixes to the KG that don’t require re-running the LLM ingestion pipeline. Reachable from the global menu when the signed-in user has the admin role.

Functions on this page:

Feedback collection (inline on every answer)

Purpose: capture per-answer feedback to drive offline weight calibration.

Functions:

Settings & preferences (/settings)

Purpose: per-user preferences and account.

Functions on this page:

Walkthrough screenshots of any pages still marked screenshot pending are available on request — reach out via email or LinkedIn.

Status

In progress (research-assistant role, 2026.01–present).

← back