Extended Semantic Web Conference (ESWC 2025) & Semantic Web 2026

LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Models and AI Agents

Sameer Sadruddin1,*, Eleni Poupaki2, Alex Watkins3, Bora Karasulu3, Adriaan J. M. Mackus2, Erwin Kessels2, Sören Auer1,4, and Jennifer D’Souza1,*

1TIB Leibniz Information Centre for Science and Technology   2Eindhoven University of Technology   3University of Warwick   4L3S Research Center, Leibniz University of Hannover

* Equal contribution

Four-stage LLMs4SchemaDiscovery workflow covering initial schema mining, preliminary refinement, final refinement, and ontology grounding.
Overview of the schema-miner workflow: Stage 1 initial schema mining, Stage 2 preliminary refinement with curated literature and expert feedback, Stage 3 final refinement with a broader corpus, and Stage 4 ontology grounding.

Workflow Description

schema-minerpro is a human-in-the-loop workflow for turning scientific process descriptions and literature into ontology-grounded schemas. This page summarizes the research workflow; for installation, configuration, and command-line usage, see the software README.

Stage 1

Initial Schema Mining

The workflow begins with a compact process specification written by a domain expert. An LLM uses this specification to draft an initial JSON schema that captures core entities and properties, such as materials, parameters, measurements, and process outputs.

Stage 2

Preliminary Refinement

The draft schema is refined with a small, high-quality corpus of domain-relevant scientific papers. Domain experts review the intermediate schema and provide feedback, helping the model add missing properties, clarify descriptions, merge duplicates, and align terminology with the literature.

Stage 3

Final Refinement

A broader and more heterogeneous corpus is then used to test whether the schema remains stable across different reporting styles, process variants, and edge cases. This stage improves generalizability while preserving the domain structure established in the earlier stages.

Stage 4

Ontology Grounding

The final schema is aligned with formal ontology concepts so that extracted properties become machine-actionable. The grounding agent first checks for direct lexical matches, then uses FAISS-based semantic search and LLM reasoning for ambiguous properties. Domain experts validate the resulting quantity kinds, units, URIs, and grounding decisions.

Agentic ontology grounding workflow using lexical search, FAISS semantic search, LLM calls, and expert validation.
The grounding agent checks schema properties against ontology terms, uses FAISS-backed semantic search for ambiguous cases, and returns ontology-aligned metadata for expert validation.
QUDT quantity schema connecting quantity value, quantity kind, and unit.
Quantity schema used to connect extracted physical properties to QUDT quantity kinds, numerical values, and units.

Example Output from schema-miner

atomic layer deposition (ALD) experimental schema

Complete ALD experimental schema generated by schema-miner, showing linked schema classes and quantity normalization terms.
Complete ALD experimental schema output with process, material, quantity, and unit relationships.
Annotated ALD schema view showing core schema properties, process-specific properties, and QUDT-based quantity normalization.
Focused schema view connecting core process properties to QUDT-based quantity normalization.

Results Snapshot

The evaluation compares schemas generated by GPT-4o, GPT-4-turbo, and LLaMA 3.1 (8B) across refinement stages. Quantitative metrics capture schema variance, while qualitative review identifies model stability and grounding quality.

Representative Stage 3 Experimental Schema Similarity
Use case Model pair ROUGE-L BERT-F1
ALD GPT-4o vs GPT-4-turbo 0.3690 0.8046
ALE GPT-4o vs GPT-4-turbo 0.3552 0.7631

Primary quantitative metrics: ROUGE-L, BLEU, and BERTScore. See the paper for the complete cross-stage matrices.

Qualitative Findings

For ALD, GPT-4o and LLaMA 3.1 (8B) maintained the most coherent schema structures across refinement stages. For ALE, GPT-4o and GPT-4-turbo were the most reliable at capturing the process structure. The hybrid grounding strategy reduced unnecessary LLM calls by using direct ontology lookups before semantic search.

Tutorial Notebooks

Use the provider-specific notebooks for guided, end-to-end runs of the schema-minerpro workflow.

Hugging Face GPU

Local model notebook

Run the three-stage workflow with a local Hugging Face model on a CUDA-enabled GPU.

Example model: mistralai/Ministral-3-3B-Instruct-2512

Open notebook

KISSKI SAIA

Academic Cloud API notebook

Run the workflow through the SAIA OpenAI-compatible endpoint without a local GPU.

Endpoint: chat-ai.academiccloud.de/v1

Open notebook

OpenRouter

Remote API notebook

Run the same staged workflow with OpenRouter through schema-miner's SAIA-compatible backend.

Endpoint: openrouter.ai/api/v1

Open notebook

File naming: tutorial outputs use a model-specific <model> stem such as mistralai-Ministral-3-8B-Instruct-2512, qwen3-30b-a3b-instruct-2507, or qwen3-235b-a22b. Stage 2 and Stage 3 detect available batchN/ folders and write matching schema-batchN/, feedback-batchN/, and log files.

Citation

ESWC 2025

@InProceedings{10.1007/978-3-031-94578-6_14,
  author    = {Sadruddin, Sameer and D'Souza, Jennifer and Poupaki, Eleni and Watkins, Alex and Babaei Giglou, Hamed and Rula, Anisa and Karasulu, Bora and Auer, S{\"o}ren and Mackus, Adrie and Kessels, Erwin},
  editor    = {Curry, Edward and Acosta, Maribel and Poveda-Villal{\'o}n, Maria and van Erp, Marieke and Ojo, Adegboyega and Hose, Katja and Shimizu, Cogan and Lisena, Pasquale},
  title     = {LLMs4SchemaDiscovery: A Human-in-the-Loop Workflow for Scientific Schema Mining with Large Language Models},
  booktitle = {The Semantic Web},
  year      = {2025},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {244--261},
  isbn      = {978-3-031-94578-6}
}

Semantic Web 2026

@article{sadruddin2026schema,
  title = {SCHEMA-MINERpro: Agentic AI for Ontology Grounding Over LLM-Discovered Scientific Schemas in a Human-in-the-Loop Workflow},
  author = {Sadruddin, Sameer and Poupaki, Eleni and Watkins, Alex and Karasulu, Bora and Mackus, Adriaan J. M. and Kessels, Erwin and Auer, S{\"o}ren and D'Souza, Jennifer},
  journal = {Semantic Web},
  volume = {17},
  number = {3},
  pages = {22104968261431521},
  year = {2026},
  doi = {10.1177/22104968261431521}
}

ESWC 2025 paper DOI: 10.1007/978-3-031-94578-6_14. Semantic Web 2026 paper DOI: 10.1177/22104968261431521.