127 lines
8.1 KiB
Markdown
127 lines
8.1 KiB
Markdown
# Digital Search Commons
|
|
|
|
## Map of the Architecture
|
|
|
|
The digital search commons is the larger environment in which independently operated participants discover, observe, preserve, describe, exchange, compare, retrieve, and evaluate information so that it can be found and used. It is larger than any single search engine, archive, crawler, protocol, database, relay, company, implementation, or transport.
|
|
|
|
The architecture is now described through four coordinated documents rather than one monolithic root text:
|
|
|
|
- **`digital_search_commons.md` — Map.** This document locates the other documents and explains how their scopes fit together.
|
|
- **`terms_of_reference.md` — Terms of Reference.** This document defines the shared vocabulary used across the architecture.
|
|
- **`dsearch.md` — DSEARCH.** This document describes the search/discovery commons, its Base Layer, interoperability boundaries, decentralization, resilience, governance, and search-oriented contracts.
|
|
- **`devidence.md` — DEVIDENCE.** This document describes preservation, acquisition, representation, provenance, derivation, historical backfill, evidentiary observations, disagreement, and the richer evidence surface from which searchable projections may be produced.
|
|
|
|
The split is intended to make the architecture easier to reason about without separating concerns that need to remain connected.
|
|
|
|
```text
|
|
DIGITAL SEARCH COMMONS
|
|
|
|
|
+-----------------+-----------------+
|
|
| |
|
|
v v
|
|
DSEARCH DEVIDENCE
|
|
discovery / search / acquisition / preservation /
|
|
interoperability / provenance / derivation /
|
|
coverage / routing evidence / historical state
|
|
| |
|
|
+-----------------+-----------------+
|
|
|
|
|
shared Terms of Reference
|
|
|
|
|
v
|
|
Base Layer contracts
|
|
|
|
|
+-----------------+-----------------+
|
|
| |
|
|
v v
|
|
bindings / projections storage / archive
|
|
Nostr, HTTP, SIP-01, etc. WARC, ARC, WACZ, etc.
|
|
| |
|
|
+-----------------+-----------------+
|
|
|
|
|
v
|
|
implementations
|
|
crawlers, relays, archives,
|
|
search engines, AI systems,
|
|
indexes, applications, operators
|
|
```
|
|
|
|
DSEARCH and DEVIDENCE are not replacements for one another. DSEARCH is concerned principally with making independently produced resources, observations, capabilities, and search evidence discoverable and interoperable. DEVIDENCE is concerned principally with preserving what was observed, how it was acquired, how later observations were derived from it, and what evidentiary relationships can be established or challenged.
|
|
|
|
A participant may implement only DSEARCH-facing contracts. A SIP-01 indexer that publishes a valid kind `39697` observation can contribute useful search evidence without implementing the full DEVIDENCE model. A richer participant may preserve exact representations, acquisition records, derivation lineage, historical versions, and multiple analytical observations, and may expose those records through DEVIDENCE while also publishing a thinner SIP-compatible projection.
|
|
|
|
This creates a deliberate coexistence model:
|
|
|
|
```text
|
|
ONE RESOURCE / EVIDENCE BASE
|
|
|
|
|
+------------+------------+
|
|
| |
|
|
v v
|
|
SIP-compatible surface DSEARCH/DEVIDENCE surface
|
|
lightweight projection rich evidence/search view
|
|
ordinary Nostr relays resources, captures,
|
|
kind 39697 / NIP-50 representations, derivations,
|
|
observations, relationships
|
|
```
|
|
|
|
The SIP-accessible version and the DSEARCH/DEVIDENCE version are therefore different projections for different depths of use. SIP is not the obsolete version and DSEARCH/DEVIDENCE is not the replacement version. Each should remain independently useful.
|
|
|
|
## Architectural Direction
|
|
|
|
The commons defines meaning. Implementation technologies provide capabilities. A protocol, archive format, database, relay implementation, crawler, or search engine can become highly important without becoming the definition of the commons itself.
|
|
|
|
This principle applies to the technologies currently useful to the project:
|
|
|
|
```text
|
|
Indexstr = observer/indexer implementation
|
|
SIP-01 = interoperable web-search projection
|
|
Nostr = transport/publication substrate
|
|
Ditto/UNCAGED = relay implementations
|
|
OpenSearch = search/index implementation
|
|
WARC/ARC/WACZ = representation-preservation formats
|
|
DSEARCH = search/discovery semantics and interoperability
|
|
DEVIDENCE = evidence/provenance/preservation semantics
|
|
```
|
|
|
|
The architecture should continue to pass several replaceability tests. If Indexstr disappeared, another observer should be able to produce useful observations. If a relay disappeared, signed observations should remain reproducible or republishable elsewhere. If Nostr were replaced, preserved evidence should remain intelligible. If an extractor changed, the system should be able to distinguish a change in interpretation from a change in the underlying representation.
|
|
|
|
## Shared Development Method
|
|
|
|
DSEARCH and DEVIDENCE are to be refined against real data rather than completed solely as abstract ontologies. The project should preserve source observations, apply the smallest vocabulary that adequately represents them, record semantic gaps, collect concrete examples, introduce the smallest new distinction that resolves a recurring gap, and then test that distinction against other domains.
|
|
|
|
Concepts should remain at the narrowest level at which they are useful:
|
|
|
|
```text
|
|
CORE / BASE LAYER
|
|
resource, participant, observer, observation,
|
|
representation, acquisition, derivation,
|
|
publication, replication, currentness, provenance
|
|
|
|
DOMAIN PROFILE
|
|
web resource, URL, HTTP acquisition, redirect,
|
|
page, crawler, canonicalization, MIME type
|
|
|
|
IMPLEMENTATION
|
|
Nostr, SIP-01, kind 39697, Indexstr,
|
|
WARC, SQLite, OpenSearch, Ditto, UNCAGED
|
|
|
|
SITE / COLLECTION PROFILE
|
|
Housatonic, Google Sites roots,
|
|
collection-specific seeds and extraction rules
|
|
```
|
|
|
|
The purpose of this discipline is to prevent an implementation convenience from becoming a foundational meaning merely because it appeared first.
|
|
|
|
## Document Authority
|
|
|
|
The Terms of Reference supplies common meanings used by DSEARCH and DEVIDENCE. The DSEARCH document governs search/discovery architecture. The DEVIDENCE document governs evidence, representation, provenance, and derivation architecture. Lower-level contracts, schemas, conformance tests, bindings, and implementations should identify which of these meanings they implement.
|
|
|
|
Where a later implementation exposes a new distinction, the implementation does not silently redefine the architecture. The distinction should be brought back to the appropriate document and incorporated deliberately if operational evidence shows that it belongs there.
|
|
|
|
## Current Empirical Work
|
|
|
|
The Housatonic web-indexing effort is the first substantial corpus intended to exercise this split architecture. It will combine live crawling, historical backfill, preserved web representations, enhanced metadata extraction, SIP-01 publication, richer DSEARCH indexing, and DEVIDENCE provenance.
|
|
|
|
The purpose is not to make Housatonic a special case in the Base Layer. The purpose is to use a real, heterogeneous corpus to discover which distinctions the Base Layer actually needs.
|