13 KiB
DEVIDENCE
Purpose
DEVIDENCE is the evidentiary and preservation architecture of the digital search commons. Its purpose is to preserve not only that a resource or assertion was observed, but enough of the surrounding evidentiary chain to explain what was observed, how it was acquired, how later analytical observations were derived, how the material changed through time, and where independent observations agree or disagree.
For web material, DEVIDENCE can be understood as the foundation of a more evidentiary form of web archive. A conventional archive can answer, “What representation of this URL was captured at this time?” DEVIDENCE extends that capability by allowing preserved representations to be repeatedly analyzed so that metadata, entities, citations, claims, references, relationships, and changes can themselves become attributable and searchable observations.
The goal is not to replace web archives, WARC, SIP-01, or search engines. The goal is to connect preservation, analysis, provenance, and search without collapsing them into one representation.
The Evidentiary Chain
A basic DEVIDENCE web chain is:
RESOURCE
|
| encountered through
v
ACQUISITION
|
| produced/imported
v
REPRESENTATION
|
| preserved as
+------------------> ARCHIVE ARTIFACT
| WARC / ARC / WACZ / other
|
| analyzed through
v
DERIVATION
|
| produced
v
OBSERVATION / ASSERTION
|
+--------------------+
| |
v v
DSEARCH projection SIP projection
rich search kind 39697 / other
The preserved representation is the durable evidentiary anchor. Search indexes and protocol projections are downstream products.
Preservation Is Not Search
DEVIDENCE distinguishes durable evidence state from rebuildable or republishable search state.
DURABLE
preserved representations
acquisition provenance
derivation provenance
evidentiary observations
relationships among observations
REBUILDABLE
OpenSearch indexes
entity indexes
full-text indexes
graph indexes
ranking structures
REPUBLISHABLE
SIP events
relay copies
other transport-specific projections
If a search index is lost, it should be possible to rebuild it from durable evidence. If a relay disappears, signed projections should be reproducible or republishable according to their contracts. Losing the preserved representation and provenance is more serious because later analysis may not be able to reconstruct what was originally observed.
Ingestion Rather Than One Crawler
DEVIDENCE should not assume that all representations were obtained through one live crawler. The acquisition layer is an ingestion framework with multiple adapters.
Potential sources include:
LIVE_HTTP
WARC
ARC
WACZ
ARCHIVE_ORG
COMMON_CRAWL
LOCAL_FILE
site backups
crawler exports
document collections
future archival formats
These are input formats and acquisition routes, not the DEVIDENCE ontology.
A normalized representation record should preserve the source-specific information that can be established without inventing information the source did not contain.
For historical imports, at least two times must remain distinct:
observed_at
when the source representation was originally captured,
to the extent that the source establishes this
ingested_at
when the local DEVIDENCE participant imported it
A WARC record captured in 2018 and imported in 2026 remains a 2018 representation imported in 2026. The importer must not appear to have personally observed the live resource in 2018.
Actors in Historical Backfill
Backfill can involve several different actors:
original observer
captured the historical representation
archive/preserver
retained or distributed it
importer
brought it into a DEVIDENCE system
deriver
analyzed the historical representation
publisher
exposed a search or protocol projection
These roles may belong to different organizations or software systems. Preserving the distinction prevents a newly produced analytical observation from being mistaken for an original historical observation.
Provenance Ceiling
Different source formats support different strengths of provenance.
A WARC record may preserve target URI, capture time, HTTP headers, record identity, payload, content digests, and other acquisition context. ARC may preserve less. A loose HTML file may preserve exact bytes while leaving the original acquisition circumstances unknown.
DEVIDENCE should retain the strongest provenance supported by the available source and make missing provenance visible. It should not synthesize fields merely to make heterogeneous imports look equally complete.
This is the provenance ceiling of the imported evidence.
Repeated Analysis of Preserved Material
A preserved representation may support new observations long after acquisition.
representation R1
|
+--> extractor v1 --> observation set A
|
+--> extractor v2 --> observation set B
|
+--> entity linker --> observation set C
|
+--> citation parser --> observation set D
|
+--> later model --> observation set E
Acquisition and interpretation are therefore separable.
This permits analytical capability to improve without requiring a vanished or changed resource to be reacquired. It also permits different analytical methods to coexist and disagree.
Enhanced Metadata as Evidence
DEVIDENCE may derive richer metadata from preserved representations, including:
title and description
language and topics
people
organizations
places
dates
citations
references
quoted material
documents
links
media references
authorship signals
claims or assertions
relationships
structural sections
change observations
detector/model observations
A derived item remains an observation attributed to the derivation that produced it.
If an extractor identifies a relationship between a person and an organization, DEVIDENCE records that the extractor observed textual or structural evidence suggesting that relationship. The system does not convert the extracted relationship into unqualified truth.
Useful derivation provenance may include the exact source representation, source span or locator, extractor or model identity, version, configuration, normalization method, and confidence or uncertainty where the method legitimately supplies it.
Disagreement Is Evidence
Two observers or derivations may produce different outputs from what appears to be the same resource.
The disagreement may reflect:
different acquisition times
different returned representations
geographic variation
personalization
authentication or cookies
redirect differences
different extraction methods
different model versions
different metadata interpretation
observer error
DEVIDENCE should preserve enough lineage to investigate these possibilities rather than flattening them into one “correct” record.
Multiple analytical observations can be retained side by side. Later consumers can compare or evaluate them according to their own methods.
Change Through Time
A resource may have a sequence of historical representations:
resource R
2014 -> representation A
2016 -> representation B
2019 -> representation C
2023 -> representation D
2026 -> representation E
The archive can then support both representation retrieval and derived temporal questions, such as when an entity first appeared, when a citation was added, when wording changed, when a link disappeared, or when independently captured representations began to diverge.
A changed extraction output is not automatically a resource change. DEVIDENCE must preserve whether the underlying representation changed or only the derivation method changed.
Better-Than-Snapshot Search
The richer archive becomes searchable at more than the page level.
A conventional URL/date lookup may return an archived representation. DEVIDENCE-backed DSEARCH can additionally expose searchable observations about what appears within the representation.
Conceptually:
search: "June Almeida"
|
v
entity observation
|
v
derivation record
|
v
representation
|
v
WARC record
|
v
resource / URL
The useful property is traceability. Searchable derived evidence should retain a path back to the representation that supported it.
Dual Projection: SIP and DSEARCH/DEVIDENCE
A rich DEVIDENCE record may support several public projections.
For ordinary web-search interoperability, a participant may derive a SIP-01 kind 39697 event containing the URL, title, description, topics, language, observer identity, time, and other fields allowed by the SIP contract.
For richer search, DSEARCH may index resources, representations, captures, entities, citations, claims, relationships, historical versions, and provenance.
DEVIDENCE evidence graph
|
+------------+------------+
| |
v v
SIP-01 projection DSEARCH projection
lightweight rich searchable graph
interoperable and evidence context
| |
v v
Nostr relays indexes / APIs
The SIP event is a deliberately lossy projection. It is not expected to permit reconstruction of the complete DEVIDENCE chain.
Conversely, a valid SIP event does not require a DEVIDENCE record. DSEARCH must remain able to consume useful thin observations from independent participants.
Historical SIP Projection
Historical material requires special care. A metadata extraction performed today over a historical representation should not be published in a form that implies the original live resource was observed today in that historical state, nor should the importer be made to appear as the historical observer.
A historical archived representation may be indexed as a distinct resource where its archive locator is itself stable and retrievable. Other historical observations may be better exposed only through the rich DSEARCH/DEVIDENCE surface until a thinner binding can represent their temporal semantics without distortion.
The principle is:
historical representation
!= current resource observation
Binding Rich and Thin Records
Where a rich participant publishes both forms, DEVIDENCE should retain explicit cross-identities:
resource_id
acquisition_id
representation_id
derivation_id
observation_id
SIP event id
DSEARCH projection/index identity
The exact serialization is a later contract question. The semantic requirement is that a rich result can trace back to its evidence and that a protocol projection can be identified as a projection of a richer observation where such a relationship exists.
Evidence Depth Without Universal Trust Scores
Search results can differ greatly in the evidence available behind them.
One result may provide only a signed search observation. Another may additionally provide a preserved representation and acquisition record. Another may have several independent captures and competing derivations.
DEVIDENCE should make those differences inspectable without automatically converting them into a universal ranking or truth score.
The existence of deeper provenance increases what can be examined. It does not guarantee the truth of the underlying claim.
Housatonic as the First Empirical Corpus
The Housatonic indexing project is intended to exercise the DEVIDENCE model with real material rather than serve as a special-case ontology.
The first operating pipeline should be:
discover Housatonic resource
↓
scope / robots / acquisition policy
↓
live HTTP acquisition or historical import
↓
preserve exact representation
↓
WARC / source archive binding
↓
derive normalized text and metadata
↓
derive selected entities/references/claims
↓
create DEVIDENCE observations
↓
derive SIP-01 projection where appropriate
↓
publish to NA1 / later NA2
↓
build richer DSEARCH projection
↓
verify both search surfaces
The corpus should also be backfilled from available WARC, ARC, WACZ, Archive.org, Common Crawl, local archives, or other historical sources where provenance can be preserved adequately.
The result should support both ordinary SIP/Nostr search and richer historical/evidentiary search without forcing either interface to become the other.
Empirical Accretion of the Model
DEVIDENCE should grow from recurring real evidentiary problems.
When a real representation or analytical case cannot be described without losing important information, the gap should be recorded. New semantic distinctions should be introduced only when they solve an observed problem and should then be tested against unrelated corpora or evidence types.
This prevents WARC, HTTP, Nostr, Indexstr, or any other implementation from becoming the hidden ontology merely because it supplied the first working implementation.
The continuing question is not “What fields can our current software produce?” but “What distinctions must remain explicit if a later participant is to understand, reproduce, challenge, compare, or reuse the evidence?”