13 KiB
Digital Search Commons — Terms of Reference
Purpose
These Terms of Reference define the common vocabulary used across DSEARCH and DEVIDENCE. They are intended to preserve shared meaning at interoperability boundaries while allowing independently operated systems to use their own internal terminology, schemas, identifiers, storage systems, and software structures.
A local system does not need to use these terms internally. It needs only to map the parts of its operation relevant to participation into the corresponding shared meanings.
Commons and Project Terms
A commons is a shared environment formed by independently situated participants, shared resources and capabilities, and relationships through which participants contribute to, draw from, preserve, compare, govern, and steward what is held in common. A commons can require specialized stewardship while still preserving meaningful independence among participants.
A digital commons is a commons whose shared resources, records, relationships, interfaces, and means of participation are substantially represented, exchanged, processed, or used digitally.
A digital search commons is a digital commons organized around discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information so that it can be found and used.
DSEARCH is the architectural and specification effort concerned with interoperable search and discovery within the digital search commons. It defines common search-domain semantics, Base Layer contracts, adaptation boundaries, conformance expectations, and the relationships required for independently built systems to exchange useful search information.
DEVIDENCE is the architectural and specification effort concerned with evidence, preservation, provenance, representations, derivation, historical state, and the relationships required to explain how an observation or searchable assertion came to exist.
DSEARCH and DEVIDENCE overlap around observations and evidence, but they are not synonymous. DSEARCH may consume observations for discovery without requiring their complete evidentiary history. DEVIDENCE may preserve representations and derivations even when no DSEARCH projection has yet been produced.
Participants and Responsibility
A participant is a person, organization, service, software agent, or technical system acting within the commons while retaining control of its own operation.
An operator is the person or organization controlling or administering a technical participant.
An observer is a participant, or a process acting on behalf of a participant, that examines a resource, representation, event, measurement, or other subject and produces an observation.
Operational responsibility, cryptographic identity, observer identity, and participant identity may coincide in an implementation but remain semantically distinct. A single operator may control multiple participants or observers. Several keys do not by themselves establish several independent operators.
Resources, Representations, and Content
A resource is something identifiable that can become the subject of discovery, retrieval, observation, preservation, description, or reference.
A resource identity identifies that subject across relevant acts of discovery or observation. Resource identity is not necessarily identical to a URL, file path, Nostr event identifier, archive locator, or other substrate-specific identifier.
A representation is a particular material presentation or response obtained from, associated with, or preserved for a resource under particular conditions. An HTTP response body, a PDF file, a Nostr event, a database export, or a captured media object may each constitute a representation when treated as the material examined.
A representation identity identifies a particular representation independently of the resource from which it was obtained. Content-addressed hashes may contribute to representation identity but do not exhaust its semantics.
Content is the material carried by or derived from a representation. Where a use case requires a distinction between exact bytes, normalized text, rendered media, or another interpretation of content, the relevant profile or contract should state that distinction explicitly.
A resource can yield different representations over time or under different conditions. Identical representations can be associated with several resources. Resource identity and representation identity therefore remain distinct.
Acquisition and Capture
An acquisition is an act through which a participant obtains or imports a representation. A live HTTP fetch, a WARC record import, an ARC import, a WACZ import, a filesystem ingestion, or another archive import may each be an acquisition.
An acquisition identity identifies that acquisition act or record.
A capture source describes the source through which a representation entered the system. Examples include LIVE_HTTP, WARC, ARC, WACZ, ARCHIVE_ORG, COMMON_CRAWL, and LOCAL_FILE. These names are profile or implementation values rather than Base Layer requirements.
Observed time identifies when the source representation was originally observed or captured to the extent that this can be established.
Ingested time identifies when the local participant imported or accepted the representation into its own system.
Observed time and ingested time are distinct. Importing a 2018 WARC record in 2026 does not mean that the importer observed the live resource in 2018.
A historical representation is a representation associated with an earlier observation or capture. It does not by itself establish the current state of the resource.
Observation, Assertion, and Evidence
An observation is a record of an evidentiary act in which an observer examines a subject and records what was found, when the examination occurred, and such conditions and provenance as are required to interpret the result.
An observation identity identifies that particular evidentiary act or record.
An assertion or claim is a statement made by a participant, observer, source representation, extractor, or analyst. A claim remains attributable to its source and does not become true merely because it is stored, signed, replicated, or indexed.
Evidence is information that can support, challenge, contextualize, reproduce, compare, or otherwise bear upon an assertion or interpretation. Evidence may include preserved representations, observations, measurements, derivation records, independent captures, signatures, provenance relationships, or contradictory observations.
A signature establishes a relationship between a signing identity and a statement according to the relevant cryptographic mechanism. It does not establish that the statement is true.
Derivation and Analysis
A derivation is a transformation or analytical process that produces one representation, observation, assertion, feature, or search projection from another representation or evidence object.
A derivation identity identifies that analytical act or record.
Derivation provenance may include the input representation, software or model identity, version, configuration, source span, normalization method, and other conditions needed to understand or reproduce the result.
A change in derivation output does not necessarily mean that the underlying resource changed. The architecture must be able to distinguish a changed representation from a changed extractor, changed model, or changed analytical method.
An analytical observation is an observation produced by examining an existing representation or evidence record rather than by directly reacquiring the original live resource. Historical material may therefore support new observations long after the original acquisition.
Provenance
Provenance is information describing the origin, custody, production, transformation, publication, or replication of a resource, representation, observation, assertion, or other record.
For practical reasoning, provenance may be divided into several related forms:
- Acquisition provenance — how a representation was obtained.
- Representation provenance — what exact material was preserved and where it originated.
- Derivation provenance — how an output was produced from earlier evidence.
- Assertion provenance — who or what produced a claim or observation.
- Publication provenance — where and how an observation was submitted or exposed.
- Replication provenance — where additional copies or projections were observed.
These forms may be represented by separate records rather than compressed into one object.
A provenance ceiling describes the strongest provenance claims supported by the source material actually available. A WARC record may preserve more acquisition context than a loose HTML file. A system should not invent missing provenance merely to normalize different source formats into an apparently uniform record.
Publication, Replication, and Availability
Publication makes a record or projection available through a participant, transport, index, archive, or other path.
Replication establishes an additional copy or independently retrievable instance of a record.
Replication and independent observation are different. A thousand copies of one observation improve availability but remain one underlying observation unless additional independent evidentiary acts occurred.
Availability concerns whether a record or resource can be obtained through a particular path at a particular time.
Retrieval success establishes availability through the tested path. It does not by itself establish currentness, truth, or independent corroboration.
Currentness, Change, and Historical Standing
Currentness describes how strongly the available evidence supports the proposition that an observation still describes a resource at a later time.
A precise observation timestamp and currentness answer different questions. An old observation may be excellent historical evidence while providing little support for current state.
A change observation records evidence that two representations, states, or analytical outputs differ in a way relevant to the applicable contract.
A record may remain historically preserved while being marked or related as superseded, withdrawn, obsolete, contradicted, replaced, or otherwise no longer representative of current standing.
Preservation and authority are distinct. Withdrawal and erasure are distinct.
Coverage, Search, and Evaluation
Coverage describes the scope within which a participant or collection claims or demonstrates observation, preservation, indexing, or search capability.
A search projection is a representation derived from richer state for the purpose of discovery or retrieval. Search projections may omit evidence that remains available elsewhere.
Ranking and reputation are evaluative methods or inputs. They may assist consumers in weighing results while remaining attributable methods rather than universal truth authorities.
An evidence-depth description identifies what provenance or evidentiary material is available behind a result. It is not inherently a trust score.
Base Layer, Profiles, Bindings, and Adapters
The Base Layer is the common semantic meeting point whose concepts and contracts are intended to survive changes in implementation substrate.
A profile specializes Base Layer concepts for a domain or collection, such as web acquisition, scientific measurement, or a particular corpus.
A binding describes how Base Layer or profile contracts are carried through a particular technology or substrate, such as Nostr or HTTP.
An adapter maps a participant's local concepts and implementation structures into the corresponding shared contracts and maps shared records back into forms meaningful to the participant.
A projection is a deliberately narrower representation produced from richer state for interoperability, search, transport, or another bounded purpose.
A valid projection need not contain the complete provenance graph from which it was derived.
SIP and Rich Evidence Coexistence
SIP-01 kind 39697 is treated as an interoperable web-search projection, not as the complete DEVIDENCE record for a web acquisition.
A participant may publish a valid SIP-01 observation without maintaining DEVIDENCE records. Such a participant remains useful to DSEARCH.
A richer participant may bind the SIP event to a DEVIDENCE observation, representation, acquisition, and derivation chain. The richer chain does not invalidate or supersede the ordinary SIP surface.
The architecture therefore preserves this distinction:
valid SIP observation
!= complete provenance record
historical representation
!= current live-web observation
signed observation
!= true assertion
replicated observation
!= independently corroborated observation
These distinctions are expected to constrain later contracts and conformance work.