Methodology

Primary sources are aggregated from 150+ digital archives, libraries, and databases using automated downloaders. Each item is tracked in PostgreSQL with metadata: source name, language, download status, item counts, and original URLs.

These archives are fragmented across institutions in France, Haiti, the US, and the Caribbean, with no unified catalog. Many exist only as uncatalogued scans or deteriorating microfilm.

The collection spans 8 categories — archives, databases, periodicals, primary sources, secondary sources, legal documents, maps/GIS data, and the Digital Library of the Caribbean. All downloaders are documented and reproducible.

Historical documents are processed through docTR, a state-of-the-art OCR model optimized for document understanding. GPU acceleration (NVIDIA and AMD) is used to process at scale across French, English, Spanish, and Haitian Kreyòl.

Standard OCR tools fail on these materials — 18th-century French typefaces with archaic ligatures, Kreyòl with no standardized orthography before the 20th century, and manuscripts damaged by tropical humidity. We tune recognition parameters per-collection to maximize accuracy.

OCR accuracy varies by document quality and age. Scanned 18th–19th century materials may contain errors due to faded text, printing irregularities, or damage. All limitations are flagged in source metadata.

Extracted text is segmented into semantically meaningful chunks using boundary detection (section breaks, paragraphs, logical divisions), size optimization for embedding models (200–500 tokens), and context preservation for standalone readability.

Each chunk is encoded into a 1024-dimensional vector using BGE-M3, a multilingual embedding model supporting French, English, Spanish, and Haitian Kreyòl. Embeddings are stored in Qdrant, enabling semantic search by meaning rather than exact keyword matches.

This also enables cross-lingual retrieval: a query in English like 'slave revolt 1791' matches French documents about 'révolte des esclaves' and Kreyòl texts — a capability rare in digital humanities tools, where researchers typically must search each language separately.

A full-text BM25 index provides keyword search as a complementary retrieval path, catching exact-match queries that semantic search alone may miss.

GLiNER, a generalist named entity recognition model, extracts structured information across seven categories: people (historical figures, enslaved individuals), places (cities, plantations, colonies), events (battles, treaties, uprisings), organizations, dates, legal concepts, and economic terms.

Entities are deduplicated and resolved to canonical forms — "Toussaint Louverture" and "Toussaint L'Ouverture" are merged into a single node.

Entities and their relationships are stored in Neo4j. The graph captures MENTIONED_IN (entity in document), RELATED_TO (co-occurrence), PARTICIPATED_IN (people in events), and LOCATED_IN (entities tied to places).

The knowledge graph enables exploratory research, network analysis, and relationship discovery across centuries of sources.

The chat interface uses hybrid retrieval combining three parallel paths: vector search (semantic similarity), full-text BM25 (keyword matching), and graph traversal (entity-based document discovery). Results are merged using Reciprocal Rank Fusion (RRF), then reranked by a cross-encoder (bge-reranker-v2-m3) for precision.

Retrieved passages are passed to Nemotron-3-Nano-30B (a hybrid Mamba-2 + MoE architecture with 3B active parameters per token) running on TRT-LLM. The model generates responses with inline citations.

Unlike most AI chat tools, every claim is traced to a specific passage that researchers can verify. Most retrieval-augmented generation systems leave citation accuracy to the LLM alone — we add a separate NLI verification step (DeBERTa) that checks whether each citation actually supports its claim.