8.0 KiB
Digital Search Commons — Split Migration Notes
Purpose
These notes describe how the existing monolithic digital_search_commons.md should be migrated into the new document set. They are not intended to become a normative architecture document.
The current root document already contains most of the foundational reasoning needed for the split: the definition of a commons, participant independence, adaptation at the interoperability boundary, separation of distribution from decentralization, forms of decentralization, resource/content/observation identity, currentness, resilience, substrate capture, governance, conformance, and the principle that shared meaning should survive implementation change.
The new split should preserve those arguments while moving them to the document whose authority best matches the subject.
Proposed Repository Layout
digital_search_commons.md # stable landing page / map
terms_of_reference.md # common vocabulary
dsearch.md # search/discovery architecture
devidence.md # evidence/provenance/preservation architecture
split_migration_notes.md # temporary migration aid
Keeping digital_search_commons.md as the landing document preserves the existing repository path and lets existing links continue to lead readers into the architecture.
Material Moving to Terms of Reference
The current Terms of Reference section should become the foundation of terms_of_reference.md.
The definitions of commons, digital commons, digital search commons, participant, operator, observer, and observation belong there. The existing distinctions among resource identity, content identity, and observation identity should also move there, expanded by the newly demonstrated need to distinguish representation, acquisition, derivation, observed time, ingested time, publication, and replication.
The Terms of Reference should remain comparatively compact and semantic. Governance arguments, implementation examples, and long threat discussions should not accumulate there.
Material Moving to DSEARCH
The following existing sections belong principally in dsearch.md:
Decentralization in DSEARCH
A Digital Search Commons
Architecture of the Commons
System Governance
System Threats and Defenses
Principles at a Glance
Role of This Document
Locating Architectural Disagreement
Using the Vision as a Reference
A Living Root
The Continuing Work
They should be edited rather than merely copied. Evidence-preservation material that is now more naturally DEVIDENCE should be referenced from DSEARCH instead of forcing DSEARCH to carry the complete evidentiary ontology.
The DSEARCH document should continue to own search-domain concerns such as interoperability, participant specialization, coverage, currentness as used for search, publication/replication distinctions, result provenance, decentralization, resilience, governance, bindings, adapters, conformance, and substrate capture.
Material Moving to DEVIDENCE
The current document already anticipates much of DEVIDENCE in its discussion of evidentiary decentralization, divergent evidence, provenance, resource/content/observation identity, corroboration, contradiction, preservation, and historical standing.
The new DEVIDENCE document develops those ideas into a dedicated architecture around:
acquisition
representation
preservation
derivation
analytical observation
historical backfill
provenance chains
change through time
agreement/disagreement
reprocessing
evidence-depth
archive formats
cross-projection identity
This material should no longer be forced into DSEARCH solely because search consumes it.
New Semantic Distinctions Introduced by Current Work
The Housatonic/Indexstr/relay work has exposed several distinctions that should be integrated explicitly.
Representation versus resource
The existing document distinguishes resource, content, and observation. The archive work shows that an explicit representation concept is needed between resource and derived content/observation. Exact HTTP bytes preserved in WARC are not simply the resource and are not identical to later normalized text or extracted metadata.
Acquisition versus observation
A live fetch, WARC import, ARC import, and local file ingestion can all introduce a representation. Their acquisition provenance differs. Importing a historical representation today does not make the importer the historical observer.
Observed time versus ingested time
Historical backfill requires these times to remain separate.
Derivation versus acquisition
An extractor run over a preserved representation is a new evidentiary act. A changed derivation can arise even when the source representation is unchanged.
Historical representation versus current observation
Historical archive material must not be projected as evidence of present state without later evidence supporting currentness.
Search projection versus durable evidence
SIP-01, OpenSearch records, and DSEARCH indexes should be treated as projections. WARC and DEVIDENCE records form deeper durable evidence state.
Thin and rich participation
A valid SIP-01 participant remains useful without DEVIDENCE. A rich participant can expose deeper provenance while producing the same ordinary SIP projection.
Dual Projection Contract
The split should establish a first-class coexistence rule:
one evidence/resource base
|
+--> SIP-compatible projection
|
+--> DSEARCH/DEVIDENCE projection
The architecture must not turn this into a migration from SIP to DSEARCH/DEVIDENCE.
SIP provides a lightweight decentralized interoperable surface. DSEARCH/DEVIDENCE provides richer search and evidentiary depth. They can evolve independently while remaining linked where a participant supports both.
Historical Backfill
The new DEVIDENCE document should explicitly allow ingestion from WARC, ARC, WACZ, Internet Archive, Common Crawl, local archives, and future formats.
These sources enter through adapters. They do not define the common ontology.
Backfilled evidence should preserve original capture provenance, archive provenance, import provenance, and later derivation provenance as separate relationships where the source data allows it.
Better-Wayback Direction
The architectural objective emerging from the current work can be summarized as:
Preserve exact representations as an evidentiary archive, repeatedly derive attributable observations from them as analytical capability improves, and make both the resources and those observations searchable while retaining a path back to the evidence that produced them.
This should be developed in DEVIDENCE, while DSEARCH defines how the resulting resources and observations become discoverable across independent participants.
Cleanup Needed in the Current Root Text
The current root document contains several editorial artifacts that should not be carried forward mechanically, including the marker --- pick up here ---, the apparent typo reacLayersh, bracketed drafting text such as [perform as], and a malformed sentence around governance resilience.
The split is therefore an opportunity to produce clean successor documents rather than cut the existing file at heading boundaries.
Recommended Migration Sequence
First, add the four new documents without deleting the current monolith. Review terminology and cross-document boundaries.
Second, replace digital_search_commons.md with the map/landing version after the successor documents have been accepted.
Third, preserve the last monolithic version in Git history rather than maintaining a second archival copy in the repository.
Fourth, begin separate Base Layer invariant and contract documents only after the Terms of Reference, DSEARCH, and DEVIDENCE boundary is accepted.
The split should be considered complete when a reader can answer three questions without ambiguity:
What does this term mean?
-> Terms of Reference
What does the search commons require?
-> DSEARCH
What evidence exists behind this observation and how was it produced?
-> DEVIDENCE