Add digital_search_commons.md

2026-09-03 02:15:54 +00:00
parent b5993339cd
commit 4bde4e095e
1 changed files with 441 additions and 0 deletions
@@ -0,0 +1,441 @@
 > # DSEARCH: The Creation of a Digital Search Commons
 > **An idealized vision**
 > ## Terms of Reference
 > A **commons** is a shared environment in which independently ?situated? [engaged?] participants contribute to, draw from, preserve, and govern shared resources through commonly understood [and accepted] rules and practices.
 > [These rules and practices are not themselves within the commons; because of demands such as need for specialised knowledge to set them up and to maintain them; and the fact that the practical unweildines of a commons wide government of rules and practice, however politically ideal, raises demands  strongly difficult to be overcome.]
 > A **digital** commons is a commons whose shared resources, records, relationships [between user-entities and between the parts of the commons, including] the means of participation are substantially digital[and are exchanged and moved and used **digitally**.
 > A digital**search** commons is a digital commons organized around the discovery, observation, description, preservation, exchange, comparison, and [ad hoc? Contingent? utilitarian?] evaluation of information [to do with] **search**. Its [multitude of] participants [usually] include[s] crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, and[also] infrastructure operators. [Plus many other parts]
 > **DSEARCH** is an effort [at] defin[ing] the shared concepts, contracts, and [considerable] interoperability required [in] buil[ding] systems [allowing user-entities independent] participation in such a **digital search commons**.
 > The relationship among these four terms [(**commons** **digital** **search** and **digital search commons**)] is hierarchical. A **commons** [founding the rising build pyramid at the base] provides the general model of shared participation and [joint] stewardship. A**digital** commons applies that model to digital resources and relationships [at the next lesser level of foundation in the pyramid]. A digital **search** commons applies [the **commons** and **digital** ?foundations?] to [that entailed in] the **search** domain [sat at the next level up the pyramid of foundations] [is domain ambiguous?]. DSEARCH is [then] the architectural undertaking intend[ing] to make [the] specialized commons [outlined here] interoperable across independent implementations.[Is this enough to define DSEARCH? A  question not a complaint]
 > ## A Digital Search Commons
 > A **digital search commons** is a shared environment in which independently operating entities discover, observe, describe, exchange, compare, preserve, and evaluate information **through common structures and contracts**. Its purpose is to make search-related information usable across organizational[is 'organisational' too unspecific?], technical, and geographic boundaries while allowing each participant [-entity] to retain control of its own infrastructure, methods, policies, and implementation choices [; which are being used in accessing DSEARCH]. Crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, and other participants can therefore contribute to and consume from, within] the same commons [and] without first becoming [maleable? chattel-like? unconsidered?] parts of one [monolithic] vertically integrated system. [Bit of rhetorical colour Matt, sorry]
 > The commons exists at a level above any particular search engine, index, database, crawler, protocol, company, or network [which is present within it]. Those things participate [providing parts for working] the commons and implement[ing] portions of it. The commons itself consists of the shared meanings, records, relationships, and rules through which independently built systems can understand one another[The commons Matt? The commons maybe first and foremost is the whole thing? The body of user-entities and the services which DSEARCH provides them with? The 'nuts and bolts' are just means to the commons?]
 > At its foundation, the ?commons? [is to] distinguish the things being searched from the evidence collected about them. A **resource** identifies [is this a 'resource' is a thing that can be examined, or a 'resource'' 'picks out' a thing to be examined?] something that can be examined. **Content** identifies material obtained from that resource. [These are terminology but are not operations?] An **observation** records what a participant found when it examined the resource at a particular time. Additional records can [label and harbour data on?] describe provenance, relationships among observations, replication, coverage, currentness, corroboration, contradiction, reputation, and other properties that help participants [participants = user-entities? or is the evaluation done unseen and automatedly within the mechanisims?] evaluate the available evidence.
 > [common](n.)https://www.etymonline.com/word/commons
 > c. 1300, "a fellowship or brotherhood; early 14c., "people of a community or town, freemen, citizenry;" late 15c., "land held in common," from Old French commune and Medieval Latin communia, and partly from common (adj.). **Also compare commons**. Latin communis "common, general" (adj.) also served as a noun meaning "common property; state, commonwealth."]
 > These ?common[s]? structures allow different participants [persons?] to contribute different capabilities[is this lower and higher or is it variety alone?] while preserving specialization[.  Are specialisations presumed of a higher order than are'capabilities' here?]. One participant [ a person maybe very useful per se but has no adequate machinery? I say this to separate the person from the means?] may focus on crawling, another on historical preservation, another on a regional or subject-specific index, another on ranking, another on machine analysis, and another on a public search interface. Their [the persons'?] usefulness to the commons [is this the prime objective? usefulness to the commons?] comes from their ability to [pool, for free - in all senses?] exchange[,] compatible information while continuing to operate independently.
 > The ideal [**DSEARCH** allows, encourages, lays the ground for? therefore **diversity with interoperability**. Independent implementations can use [personal choice?] databases, programming languages, transports, storage systems, ranking methods, crawling strategies, and operational models while [at the same time being able to share] enough common meaning to [enable] exchange and [to make possible acts of evaluation of] search information. A crawler can produce observations while remaining a crawler. An archive can preserve content while remaining an archive. [The previous two sentences don't add anything in my eyes. They are meant to mean something but don't] A ranking system can evaluate evidence while retaining its own ranking method. A transport can carry observations while the Base Layer continues to define what those observations mean.
 > This separation gives the ?commons? [it's the machinery and its generation and storage of items we are talking about here?] a way to evolve through addition, replacement, and coexistence [a new term? Is this people or data?]. New crawlers, storage systems, protocols, indexes, ranking methods, and applications can enter the **DSEARCH** environment when they offer useful capabilities [This implies a filter refusing items not up to scratch?] Existing mechanisms can continue operating while alternatives are tested [This sentence is in a vacuum. The talk  has been of DSEARCH and Commons and of individual divergent contributors contributing to these.  The sentence gives no idea which of these items it is taking about?]. A successful implementation may become widely used while the commons remains larger than that implementation.[Yes DSEARCH is a means. The commons is the goal]
 > The commons [is this DSEARCH in fact?] [will] also [be able to] accommodate disagreement [is disagreement a good word here? if it means something technically incongruous.Is this the AI being clever? It is to human a word for mechanics? Especialy when coupled with 'accommodation'?] . Independent observers [new term???}] may retrieve different [separate - the word different applies to a comparison so it should be different content from one another's.  Separate is less confusing?] content, report different conditions, or reach different conclusions [from one another]. Their records can [happily] coexist [in DSEARCH and be compared through provenance, timing, identity, and other evidence. [Their assessments might differ from each other's but not necessarily conflict - disagree] Agreement can strengthen confidence where the evidence supports that conclusion [is agreeement a conclusion in this context here. Agreements mutuially support views of parties involved in them?], while contradiction remains visible and available for further evaluation.[Difffering is not always contradiction]
 > Search results can consequently become more than answers returned by an opaque [new term. Opaqueness has not yet been opened up as a factor re DSEARCH] index. They can be connected to an evidentiary environment that shows which resources [have been] examined, which content [has been] observed, [at what times] observations occurred, which observers [saw recorded captured?] produced them, how [diverse] observations relate to one another, how widely they are replicated [copies of others], where[abouts search scope] coverage appears strong or weak, and how current the available evidence [is appears correct? Surely everything is dated precisely?] appears to be.
 > The **Base Layer** of DSEARCH [build would] exist to make the **digital search commons**?] environment possible. It [would] define the [unique to DSEARCH, yet global within DSEARCH] concepts and contracts through which independent systems [using some transitionals] [will be able to] participate in the commons. Implementations [User-entities?] remain free to choose the mechanisms that suit their purposes, while the common[s] structures [comprising DSEARCH] preserve interoperability across those [discrete] choices [which suit].
 > The idealized objective [for DSEARCH} is therefore a search environment [with] strength [which] comes from independent [and vareigated] participation, [from] compatible evidence, [and] diverse implementations, and [from a] continuing [potentiality] to change. The commons gains usefulness [robustness? durability? relevance?] from the number and variety of systems that can contribute to it. [Its resilience] comes from [a studied technical diligence which will preserve for DSEARCH resort that is always open to change, change such as might involve a choice from new] multiple paths for implementation, operation, and future revision.
 > Three forms of resilience follow from this [idealised] objective. [These are:] **Technical resilience** [which] allows individual participants, services, copies, and implementations to fail while the wider commons continues to operate. **Architectural resilience** [which] allows technologies and implementations to be replaced while shared meanings remain [current and] intelligible. **Governance resilience** [which] allows several generations [is this only time or is it also additional developement?] of [DSEARCH] implementation [is this static or involves a progression?] to coexist while architectural change and migration proceed across independently operated participants. [The final sentence here is in need of attention. Unless the implementation is being considered to be in stasis and generations are being considered as time periods only, the connecting thought central to the sentence cannot hold up. But if these things are being considered in these ways, it is a view false to practical reality. Either way the sentence fails. Also the changes do not proceed across independently operated participants. Over the heads of them maybe? ]
 > These three forms express different aspects of the same goal: preserving the usefulness of the commons [to] participants, [as] technologies, and operating conditions change.[This sentence is not the whole of the matter. Keeping to the ideals as goals for the DSEARCH project is a prime concern. This concern will impinge on choices made for technology and operating conditions through DSEARCH's lifetime? It's a two way street. DSEARCH serves participants but it also in doing so advocates its values to them] Architectural resilience governs the relationship between meaning and implementation [I don't understand?]. Governance resilience governs change across independent operators.[This is so compresed it is misleading] Technical resilience governs containment of failure and resist[s] technical and social attempts to compromise the [DSEARCH] system. Some important failure modes [will] cross all three [resilience] boundaries, because a decision that begins as an architectural convenience [is a failure mode a convenience?] can later acquire operational and governance consequences.
 # DSEARCH: The Creation of a Digital Search Commons
 **An idealized vision**
 ## Terms of Reference
 A **commons** is a shared environment constituted by a community of independently situated participants, a field of shared resources or capabilities, and established relationships through which those participants contribute, draw upon, preserve, and steward what is held in common. Participation takes place under commonly understood and accepted rules and practices that give the shared environment enough coherence to remain usable through time.
 Those rules and practices require governance, but governance does not imply that every participant directly governs every rule. Some responsibilities demand specialized knowledge, sustained maintenance, adjudication, specification work, or administrative continuity. A commons can therefore combine broad participation with more concentrated stewardship of the structures that make participation possible. The relevant measure is whether those structures continue to serve the shared environment and preserve meaningful independence among its participants.
 The word **digital** describes the principal medium in which the commons operates. A **digital commons** is therefore a commons in which the shared resources, records, relationships, interfaces, and means of participation are substantially represented, exchanged, processed, and used digitally. The digital character applies both to what participants share and to the mechanisms through which they interact with one another and with the shared environment.
 A **digital search commons** is a digital commons organized around the domain of **search**: the discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information so that it can be found and used. Its participants may include people, organizations, crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, infrastructure operators, and other technical or institutional actors that contribute some useful part of that activity.
 Within this document, a **participant** is any independently controlled person, organization, service, software agent, or technical system that acts within the commons under some identifiable operational authority. Participants can occupy different roles. An **operator** controls or administers a technical participant. An **observer** is a participant or process that examines a resource and produces an observation. These roles may coincide in a particular implementation, while their meanings remain distinct because the commons may need to reason about them separately.
 **DSEARCH** is the architectural and specification effort directed toward making such a digital search commons possible. It defines the shared concepts, Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons while retaining their own implementations, infrastructure, methods, and policies.
 The relationship among these terms is hierarchical. The **commons** supplies the underlying model of shared participation and stewardship. The **digital commons** applies that model to an environment whose resources and interactions are substantially digital. The **digital search commons** specializes that environment around the requirements of search. **DSEARCH** is the architectural undertaking that develops the common meanings and technical agreements required for independently operated systems to participate in that specialized commons.
 The hierarchy can therefore be pictured as a rising set of foundations:
 `commons → digital commons → digital search commons → DSEARCH architecture enabling participation`
 Each level adds specificity while retaining what lies beneath it. The commons supplies the social and institutional form. Digital operation supplies the medium. Search supplies the domain. DSEARCH supplies the architecture through which that domain can function as an interoperable commons.
 ## A Digital Search Commons
 The digital search commons described above is the intended environment; DSEARCH is one means of making that environment possible. The commons consists of its participating community, the shared search-related resources and evidence available to that community, and the relationships through which participants contribute, retrieve, preserve, compare, and use them. DSEARCH supplies common structures that allow those activities to occur across independently designed systems.
 Its purpose is to make search-related information usable across administrative, technical, and geographic boundaries while preserving the operational independence of participants. Crawlers, archives, search engines, applications, AI systems, specialist indexes, researchers, and infrastructure operators can therefore contribute different capabilities to the same environment while retaining control of the machinery and methods through which they participate.
 The commons is consequently larger than any particular search engine, index, database, crawler, protocol, company, network, or DSEARCH implementation. These provide capabilities within the commons. The commons itself is the larger socio-technical environment formed by the participants, the shared information and evidence available among them, and the interoperable relationships that allow those contributions to become useful beyond the system that originally produced them.
 DSEARCH is concerned especially with those interoperable relationships. It seeks to provide enough shared meaning that independently built systems can understand what another participant is asserting, while leaving substantial freedom over how each participant performs its own work.
 At the foundation of search evidence are several concepts that need to remain distinguishable. A **resource** is something identifiable that can become the subject of discovery, retrieval, or observation. A **resource identity** identifies that subject across relevant acts of search or observation. **Content** is material obtained from, associated with, or presented by a resource under particular conditions. A **content identity** identifies particular content independently of the resource through which it was obtained.
 An **observation** is a record of an act in which an observer examined a resource at a particular time and under describable conditions. The observation can identify the resource examined, the content obtained, the observer responsible for the act, the time at which it occurred, and other information needed to interpret the resulting evidence. An **observation identity** identifies that particular evidentiary act or record.
 These distinctions are semantic rather than merely terminological. A resource can present changing content. Identical content can appear through several resources. Several observers can examine the same resource and obtain the same content while still producing several observations. DSEARCH therefore preserves the identities of resource, content, and observation separately because each answers a different question.
 Other records and relationships can provide provenance, replication information, coverage claims, timing, corroboration, contradiction, reputation, operator relationships, and other evidence useful in interpreting what has been observed. Some evaluation will occur automatically inside crawlers, indexes, ranking systems, AI systems, and other mechanisms; some will occur through decisions made by people or organizations. The Base Layer should preserve the evidence required for these different forms of evaluation while allowing each consumer to apply methods suited to its own purpose.
 The shared structures created through DSEARCH allow participants to contribute capabilities of different kinds and at different levels of specialization. One participant may operate a broad crawler. Another may concentrate on historical preservation. Another may maintain an index for a particular language, geography, subject, or collection. Another may contribute storage, ranking, analysis, provenance assessment, or an interface through which people search the accumulated evidence.
 A participant's value to the commons therefore comes from the useful capability or evidence it makes available and from the degree to which that contribution can be understood and used beyond its own implementation. Specialized participants need only perform the roles they are suited to perform. Their contributions become part of a larger search environment because shared contracts allow those specialized capabilities to interoperate.
 The ideal can therefore be described as **diversity with interoperability**. DSEARCH creates the architectural ground on which independently chosen databases, programming languages, transports, storage systems, ranking methods, crawling strategies, analysis methods, and operational models can exchange compatible search information. The independence lies in how participants perform their work; the interoperability lies in the meanings and boundary contracts through which their work can become useful to others.
 This distinction matters because a digital search commons gains much of its strength from specialization. A crawler can concentrate on producing reliable observations. An archive can concentrate on durable preservation. A ranking system can concentrate on evaluating available evidence according to a declared method. A transport can concentrate on moving records. The Base Layer provides the common meanings that allow these separate capabilities to form a larger whole.
 The architecture also gives the commons room to evolve. New crawlers, storage systems, transports, indexes, analytical methods, ranking systems, and applications can be introduced by participants as new needs and opportunities arise. Their adoption depends on the value they provide and, where they interact through DSEARCH, their ability to satisfy the relevant shared contracts. Existing mechanisms can continue serving participants while alternatives are developed, compared, and adopted.
 This condition is **coexistence**: several implementations, technologies, versions, or operating approaches can participate in the commons during the same period. Coexistence provides continuity while allowing change. It also allows useful technologies to become widely adopted without making the commons identical with any one of them.
 The commons also preserves **divergent evidence**. Independent observers may retrieve separate instances of content, encounter different conditions, or produce assessments that vary from one another. Such variation can arise from genuine changes in a resource, geographic differences, personalization, timing, access conditions, caching, methodology, or observer error. The commons gains information by retaining these differences together with enough provenance to interpret them.
 Several observations may corroborate one another. Others may differ without being logically contradictory. Some may directly conflict. The useful property is that their evidentiary relationships remain visible. Timing, provenance, observer identity, content identity, retrieval conditions, and other available evidence can then support later comparison and evaluation.
 Search results can consequently be connected to an evidentiary context. A result can point toward the resources examined, the content obtained, the times at which observations occurred, the observers that produced them, the relationships among several observations, the extent to which records have been replicated, the scope of the available coverage, and the evidence relevant to currentness.
 **Currentness** deserves particular care because a precise timestamp and a judgment of currentness answer different questions. An observation can say exactly when something was seen. Currentness concerns how strongly the available evidence supports the proposition that the observation still describes the resource at a later time. A recent confirming observation may provide strong evidence of currentness; an old but precisely dated observation may provide excellent historical evidence while offering little basis for a claim about present state.
 The **Base Layer** developed through DSEARCH exists to provide the common semantic foundation for this environment. It defines the concepts and contracts through which independently operated systems can exchange and interpret search-related evidence. Participants remain free to choose the mechanisms suited to their own purposes while implementing the shared boundary meanings required for participation in the commons.
 The idealized objective of DSEARCH is therefore to help create a digital search commons whose strength comes from independent and varied participation, compatible evidence, specialized capabilities, and a continuing capacity for change. The commons gains reach, robustness, and relevance as more useful participants and systems contribute to it. DSEARCH contributes resilience by preserving architectural choices that leave viable paths for replacement, extension, migration, and adaptation when technical or social conditions change.
 This objective carries values as well as mechanics. DSEARCH serves participants by making independent contribution useful across system boundaries, while its architecture also preserves the conditions that make such independence meaningful. Decisions about technologies, contracts, migration, identity, and governance should therefore be evaluated both for the immediate capability they provide and for their effect on the long-term character of the commons.
 Three related forms of resilience express this concern.
 **Technical resilience** is the capacity of the commons to continue useful operation when individual participants, services, copies, communication paths, or implementations fail. It depends on containment of failure, diversity of available paths, and enough independence among components that one local problem does not automatically become a commons-wide problem.
 **Architectural resilience** is the capacity to revise or replace implementation mechanisms while preserving the shared meanings needed for interoperability. It comes from keeping domain concepts explicit, separating those concepts from any one technical representation, and maintaining practical paths through which better implementations can succeed earlier ones.
 **Governance resilience** is the capacity of the shared architectural and specification process to evolve while independently controlled participants adopt changes according to different circumstances and schedules. Several contract versions, implementation generations, or capability sets may therefore coexist during periods of transition. Governance provides the means to define successors, communicate changes, support compatibility, and make migration tractable, while adoption remains an act performed by independently governed participants.
 These forms of resilience reinforce one another. Technical resilience preserves operation through local failure. Architectural resilience preserves freedom of implementation through technological change. Governance resilience preserves the ability of the shared framework to evolve as authority over actual implementations becomes increasingly distributed.
 Together they serve two directions at once. They help the commons remain useful as participants, technologies, and operating conditions change, and they help DSEARCH preserve the principles on which that usefulness depends as technical choices exert pressure on the architecture. The project therefore responds to changing conditions while also carrying its architectural commitments forward into the choices made under those conditions.
 A single decision can affect all three forms of resilience. A convenient implementation choice may initially improve technical capability, later become embedded in the architecture, and eventually create a governance problem when independently operated participants have organized their systems around it. The purpose of distinguishing these forms is therefore analytical: it gives DSEARCH several perspectives from which to examine the consequences of decisions whose effects may unfold over many years.
 ## Architecture of the Commons
 A durable digital search commons begins with **agreements about meaning**. Those agreements define what a resource is, what an observation represents, what replication accomplishes, what a claim of coverage establishes, how provenance is expressed, and which relationships among records participating systems are expected to understand. The technologies used to implement those agreements can change many times during the life of the commons, while the meanings need enough stability to remain intelligible across those changes.
 This produces a layered structure:
 `commons requirements → Base Layer contracts → implementation adapters`
 **Commons requirements** describe what distributed search needs to accomplish. **Base Layer contracts** express those requirements in forms that independently built systems can implement and test. **Implementation adapters** carry those contracts through particular transports, storage systems, signing mechanisms, synchronization protocols, network topologies, and other technologies.
 Several adapters can implement the same contract at the same time. One adapter may become dominant during a particular period because it solves many practical problems well. Another may be chosen for a specialized environment. A later technology may eventually replace both. The shared contract provides continuity across those implementation choices.
 This layering carries a real cost. Defining contracts before allowing an implementation technology to supply the domain model requires additional design work at the beginning. An implementation built this way may reach visible operation more slowly than one constructed directly around a mature technology that already provides most of the required machinery. That cost belongs in the architectural calculation because speed to a working system is valuable.
 The return on that cost is the ability to identify what a technology actually contributes, which parts of the system belong to the search domain itself, and what would have to change if that technology were eventually replaced. The total complexity of using an implementation substrate can therefore be described approximately as:
 `total complexity = commons semantics + substrate machinery + commons↔substrate adaptation`
 **Commons semantics** are the meanings and contracts the search commons must define regardless of implementation. **Substrate machinery** is the useful capability an implementation technology already supplies. **Commons↔substrate adaptation** is the additional work required to represent the commons through that technology while preserving the distinction between search-domain meaning and implementation detail.
 A technology creates genuine value when the useful machinery it supplies removes more work than the adaptation required to use it introduces. Architectural independence therefore provides a way to measure the contribution of mature technologies rather than a reason to avoid them.
 Two separate dimensions help make that assessment clear. The first concerns **when complexity is measured**. **Bootstrap simplicity** describes the effort required to reach an initial working implementation. **Lifecycle simplicity** describes the effort and constraint accumulated as the system is designed, tested, operated, secured, debugged, extended, documented, interconnected, migrated, and maintained over time. Lifecycle simplicity also includes the practical freedom available to correct architectural choices as experience accumulates and adoption grows.
 The second dimension concerns **where architectural meaning is defined**. A **domain-first** design begins with the concepts and contracts required by distributed search and then selects technologies to implement them. A **substrate-first** design begins with the structures and capabilities supplied by an implementation technology and increasingly expresses the application through those structures.
 These dimensions describe different questions. Bootstrap and lifecycle simplicity compare two time horizons. Domain-first and substrate-first describe the source of architectural meaning. A project can define its important contracts domain-first while making extensive use of an existing substrate underneath them, thereby gaining much of the substrate's bootstrap advantage while retaining substantial architectural freedom.
 Mature implementation technologies often perform strongly in terms of bootstrap simplicity because they already supply serialization, identity, storage, synchronization, discovery, deployment tooling, libraries, and operational experience. Using those capabilities can produce a working implementation considerably faster than recreating equivalent infrastructure from first principles.
 The lifecycle result depends on how much architectural dependence accompanies that convenience. When the commons defines its own concepts and uses a substrate primarily to carry, store, synchronize, or process them, the adaptation layer can remain thin. The commons receives much of the bootstrap advantage while preserving room to revise or replace the implementation beneath its contracts.
 A more substrate-first implementation may reduce initial work further by expressing domain concepts directly through the substrate's native records, identifiers, references, query conventions, and operational assumptions. That can be a strong answer under the conditions in which it is chosen. Its lifecycle cost appears as more search-domain meaning becomes encoded through substrate-specific structures, because changing the substrate can then require corresponding changes to stored data, indexes, adapters, tests, operational procedures, libraries, and independently developed implementations.
 The architectural choice therefore concerns the degree to which the commons borrows implementation capability while preserving its own semantic authority. Lifecycle simplicity depends on how much additional machinery accumulates, how tightly the layers become coupled, and how much freedom remains to revise one layer without forcing the rest of the system to move with it.
 The desired relationship is straightforward:
 **The commons defines the meaning; implementation technologies supply capabilities where those capabilities reduce total complexity.**
 Bootstrap simplicity and lifecycle simplicity are two time horizons over which that relationship should be evaluated. Domain-first and substrate-first describe whether the search domain continues to supply the architectural meaning or increasingly inherits that meaning from the technology chosen to implement it.
 ## System Governance
 ### Planning
 Every significant implementation dependency exchanges some amount of immediate capability for some amount of coupling. Planning therefore needs a repeatable discipline that makes this exchange visible and allows dependencies to be judged against the same architectural questions.
 A proposed technology should first be examined for the capability it already supplies in usable form. The analysis should then identify the additional conventions, schemas, representations, interpretations, compatibility mechanisms, and operational work needed to preserve the meaning of the commons through that technology. A strong dependency also allows an independently built alternative to implement the same Base Layer contract while preserving its meaning.
 Planning should also examine what happens under successful adoption. Stored data, tooling, documentation, testing, indexing, and operator practice may begin to depend upon a technology's particular representation. The more widely that dependency spreads, the more important its migration characteristics become. A technology that provides substantial capability with a small adaptation layer can be an excellent choice. A technology whose adaptation layer steadily expands deserves continuing review even when its initial implementation was highly successful.
 One question carries special weight because it concerns the meaning of the commons itself: whether a foundational concept can still be defined independently of the implementation technology carrying it. Resource identity, observation identity, replication, coverage, and participant standing belong to the Base Layer when their meaning needs to survive a change in substrate. When one of those concepts becomes expressible only through the native vocabulary of a particular technology, an implementation decision has begun to determine the architecture.
 Planning is therefore a standing discipline rather than a one-time review. Its purpose is to keep convenience decisions and meaning decisions visible even when the same engineering work touches both.
 ### Cooperation
 A commons composed of independently operated participants evolves through cooperation. A steward of the Base Layer can publish a successor contract, describe a migration path, provide compatibility tooling, and publish evidence supporting the change. Adoption then proceeds through the decisions and circumstances of crawlers, archives, indexes, applications, infrastructure operators, and other participants whose systems depend upon the earlier behaviour.
 Some participants will move quickly. Others will move later because of cost, priority, operational constraints, disagreement, abandonment, or competing technical obligations. This variation follows directly from the independence the commons is designed to preserve.
 Architectural evolution therefore proceeds through published contracts, compatibility, migration, and convergence across participants. **Cooperation becomes an architectural mechanism rather than merely a social preference.**
 Governance can make that cooperation tractable by making successor contracts clear enough for independent implementers to act on them without guessing. Conformance tests can allow participants to verify their own implementations. Compatibility windows can turn migration into a plannable process. Versions can coexist where necessary, capabilities can be discovered explicitly, and earlier contracts can remain legible while they continue to operate within the commons.
 Migration itself can produce evidence. Participants and tooling can inspect which contracts are supported, which transitions have completed, and which capabilities remain available. Deprecation can describe the intended architectural direction, while retirement criteria can define the conditions under which support for an earlier form can reasonably end.
 Within this model, **deprecation and migration describe different processes**. Deprecation expresses the architectural direction of the specification. Migration changes the operating population of implementations. Versioning, compatibility, migration, and retirement therefore belong to the architecture of the commons rather than merely to its documentation.
 ### Authority, Adoption, and Path Dependence
 **Architectural authority** is the standing to define the shared concepts and contracts of the commons. **Implementation authority** is control over the software, data, infrastructure, and operational practices through which those definitions are realized.
 Early in the life of a commons, both forms of authority may be concentrated among a small group. An architectural decision can then be revised relatively cheaply because few implementations, records, tools, or operators depend upon it.
 Successful adoption changes that relationship. Implementation authority spreads across every independent participant that builds upon the shared contracts, while architectural authority continues to describe the intended meaning of those contracts. The growing distance between the two creates **path dependence**.
 A representation chosen for early convenience can become expensive to change as independently operated systems store it, index it, interpret it, exchange it, document it, and build additional behaviour upon it. The distinctions that cost little to preserve during early development can therefore become difficult to recover after wide adoption.
 A mature commons eventually faces two simultaneous changes. The cost of correcting an architectural decision increases because more data, implementations, operators, tooling, and users depend upon it. At the same time, practical authority over those implementations becomes increasingly distributed. The original architects can still propose a better direction, but adoption of that direction becomes a coordination and migration problem across the commons.
 The important lifecycle question is therefore whether the architecture preserves enough semantic independence, versioning capacity, replaceability, and migration freedom to change direction after widespread adoption.
 This relationship also creates one of the commons's most consequential systemic risks: an implementation technology can gradually acquire authority over meanings that originally belonged to the search domain. That process is addressed below as **substrate capture**.
 ## System Threats and Defenses
 A commons designed for independent participation faces threats that are technical, operational, and social. Many important threats move across those boundaries, because the same condition can begin in software, spread through operational practice, and eventually become a governance problem.
 ### Manufactured Plurality
 An open commons can attract participants that create many apparent identities in order to simulate independent agreement, corroboration, popularity, or reputation. This is the **Sybil problem** in its broader form: one actor can manufacture the appearance of many actors.
 Strong cryptographic identity helps establish which key produced a statement. It provides much less information about whether several keys represent several independent people, organizations, crawlers, infrastructure operators, or acts of observation. Ten valid signatures establish ten signing events; independence requires additional evidence.
 The defense therefore belongs in the commons's model of independence and reputation. Cryptographic identity, operational identity, administrative control, infrastructure relationships, observational independence, historical behaviour, and evidentiary weight can be represented as related properties while retaining their distinct meanings.
 A participant can remain pseudonymous while the commons still records evidence useful for assessing whether several apparently independent observations arose from genuinely independent activity.
 ### Confused Evidence
 A distributed system can easily confuse the number of copies of evidence with the number of independent sources of evidence. Many copies of one observation improve availability. Many independent observations can contribute corroboration. Both properties are valuable, and each describes a different kind of evidence.
 Replication therefore remains distinct from observation throughout the Base Layer. A transport or storage system can create hundreds of copies of one record while the underlying observation remains one observation.
 Similar distinctions apply elsewhere. A valid signature establishes a relationship between a key and a statement. Successful retrieval establishes that a record was available through a particular path. Agreement among several records gains evidentiary value to the extent that the observations and operators behind them are meaningfully independent. Each of these claims should retain the scope that its evidence actually supports.
 The defense against confused evidence is explicit modeling. The Base Layer should preserve distinctions among publication, replication, observation, signature, currentness, independence, reputation, and other evidentiary properties so that applications can evaluate them without reconstructing their meaning from circumstance.
 ### Substrate Capture
 A slower threat can arise from ordinary architectural success. **Substrate capture occurs when the representations, abstractions, or operating assumptions of an implementation technology begin to determine the architecture of the commons above it.**
 The process is usually gradual. A new requirement begins as a question about what the commons needs to express. As capture grows, the same requirement is increasingly approached through the native records, identifiers, tags, queries, storage conventions, APIs, or extension mechanisms of the current substrate. Concrete technology always influences engineering, but the architectural boundary shifts when concepts become easy to express primarily because they fit the substrate and difficult to express when they fall outside it.
 At that point, representation has begun to influence requirement. The coupling then spreads through stored records, libraries, indexes, tests, documentation, operational procedures, and compatibility expectations. Other implementations reproduce the conventions because interoperability requires them, and migration begins to require separation of enduring domain meaning from assumptions that accumulated around the substrate.
 Substrate capture is therefore different from extensive use of a technology. A commons can rely heavily on one substrate while preserving a clear and testable boundary between the Base Layer and the implementation. Capture begins when that boundary becomes difficult to state, test, or reproduce independently.
 Successful adoption can make this progressively harder to reverse. Once substrate-specific conventions have spread across independently controlled implementations, architectural authority may retain the ability to propose a replacement while implementation authority is distributed among participants that must decide whether and when to migrate.
 The defense is preventative: independently defined Base Layer contracts, replaceable adapters, independent semantic identities, conformance tests that validate meaning rather than source shape, explicit versioning, feasible migration paths, and continuing examination of the adaptation term in the complexity equation.
 ### Fragmented Security Surface
 Independent implementation distributes both capability and security responsibility. This distribution strengthens the commons when failures remain contained and trust boundaries remain explicit.
 Exposure grows when participants cross those boundaries casually: by parsing outside data without adequate validation, sharing credentials or mutation authority, depending upon another participant as though its availability were guaranteed, accepting assertions that require independent evaluation, or allowing the state of one component to become authority over unrelated parts of the commons.
 The defense is containment and explicit trust. Outside input should be validated according to clear contracts. Authority should be narrowly scoped. Credentials and mutation rights should remain separable. Failures should remain local where possible. Provenance should be authenticated where authentication contributes useful evidence, and consumers should receive enough context to evaluate the conditions under which records were produced.
 Security of the commons therefore grows from isolation, constrained authority, and verifiable interaction across independently operated systems.
 ### Assumed Erasure
 Information copied to independently operated systems can remain available after one participant withdraws it. This property creates both technical and social consequences because withdrawal, persistence, and current standing describe different states.
 A good-faith participant may assume that a withdrawn record has disappeared everywhere. Another participant may continue to hold the record for archival, legal, operational, or adversarial reasons. The Base Layer can represent these realities directly rather than relying on a global deletion assumption.
 Records can acquire states or relationships such as **withdrawn**, **superseded**, **contradicted**, **obsolete**, or **replaced by a later observation**. Historical evidence can therefore remain preserved while its current standing is expressed clearly.
 This allows the commons to distinguish preservation from authority and withdrawal from erasure.
 ### Stalled Correction
 Because architectural evolution proceeds through cooperation, some participants will continue operating older contracts after successors appear. This can result from neglect, economic cost, abandonment, technical difficulty, disagreement, or deliberate resistance.
 The architecture should therefore make partial migration an expected operating condition. Compatibility windows can provide time for transition. Version negotiation and capability discovery can expose supported behaviour. Conformance tooling can clarify what migration requires. Successor contracts can identify changed semantics directly, while migration evidence can show which participants have completed a transition.
 Retirement criteria can describe the conditions under which support for older behaviour can reasonably end. A resilient commons remains intelligible and operational while several generations of implementation coexist.
 ## Principles at a Glance
 The principles below compress arguments developed throughout the document into a reference form.
 **The commons is larger than any implementation.** A protocol, database, crawler, index, application, or transport can become an important part of DSEARCH while the commons remains the shared environment through which many such systems interoperate.
 **Meaning belongs to the Base Layer.** Implementation mechanisms carry, store, sign, synchronize, index, and transform Base Layer objects. The Base Layer preserves the shared meaning that allows those mechanisms to change.
 **Publication and replication describe different states.** Publication makes information available. Replication establishes additional copies and contributes to availability.
 **Replication and independent observation describe different forms of evidence.** Copies strengthen availability. Independent observations can strengthen corroboration.
 **Signatures establish provenance rather than truth.** A signature establishes a relationship between a key and a statement. Confidence in the statement arises from additional evidence.
 **Resource identity, content identity, and observation identity remain distinct.** A resource can change through time, the same content can appear at several resources, and several observations can concern the same resource and content.
 **Retrieval success establishes availability through a path.** Currentness depends on observation time and later evidence about the resource.
 **Reputation and ranking are evaluative inputs.** They can help participants weigh evidence while remaining attributable methods rather than universal truth authorities.
 **Deprecation describes architectural direction; migration changes implementations.** The commons should therefore provide versioning, coexistence, migration evidence, and retirement criteria.
 **Independence produces coexistence.** Different technologies, versions, policies, and migration schedules are expected properties of an independently operated commons.
 **Replaceability is easiest to preserve early.** Dependencies are simpler to isolate before stored data, tooling, independent implementations, and operator practice accumulate around them.
 **Successful adoption redistributes practical authority.** As implementations spread, the original architects gain a larger ecosystem while losing the ability to revise every participating system directly. Architecture should anticipate that transition.
 **Today's answer is one answer under today's conditions.** Durability comes from preserving the ability to adopt better answers when conditions change.
 The idealized DSEARCH commons is therefore an environment whose contracts allow independently developed answers to cooperate, compete, be evaluated, and eventually be replaced while shared meaning remains intelligible through change.
 ## Role of This Document
 This document describes the idealized architecture and governing principles of the DSEARCH commons. It serves as the **root architectural reference** from which more specific DSEARCH specifications, implementation decisions, conformance work, and operational systems are derived.
 Its role is to establish meanings and architectural properties that later work can preserve. More specific documents can then define invariants, contracts, schemas, conformance tests, reference implementations, bindings, and applications without reopening the foundational question at every layer.
 The resulting hierarchy can be understood approximately as:
 ```text
 DSEARCH vision and commons principles
        ↓
 Base Layer invariants
        ↓
 Base Layer contracts and schemas
        ↓
 conformance tests and test vectors
        ↓
 reference implementations
        ↓
 implementation bindings and adapters
        ↓
 applications, crawlers, indexes,
 archives, AI systems, and other participants
 ```
 Each level answers a different kind of question and carries a different kind of authority.
 ### Vision and Commons Principles
 This document occupies the first level. It describes the nature of the commons, the architectural independence expected among participants, the relationship between shared meaning and implementation, the forms of resilience the commons seeks to preserve, and the governance consequences of independent adoption.
 These principles provide the context in which lower-level decisions are evaluated. A later implementation should be able to explain how it serves the commons described here while leaving the root vision stable enough to serve other implementations as well.
 ### Base Layer Invariants
 The next level converts the general principles into statements that should remain true across conforming implementations. Examples include `publication != replication`, `signed != true`, `replication != independent observation`, `resource identity != content identity != observation identity`, `retrieval success != currentness`, `reputation != truth authority`, and `deprecation != migration`.
 These invariants are more precise than the vision while remaining independent of wire format or implementation technology. They establish boundaries that later contracts are expected to preserve.
 ### Base Layer Contracts and Schemas
 Contracts define the objects, relationships, required semantics, and observable behaviours through which the invariants become implementable. Schemas provide agreed representations through which those contracts can be exchanged.
 The distinction matters because a contract defines meaning while a schema expresses that meaning in a particular form. Where practical, the contract should remain understandable independently of any one serialization.
 ### Conformance Tests and Test Vectors
 Conformance work determines whether independently developed implementations preserve the contracts. Effective tests qualify observable semantics rather than incidental source structure.
 A content-identity test should establish that qualifying implementations derive equivalent identities from qualifying content. A replication test should establish that copied records remain distinguishable from independent observations. A supersession test should establish the semantics of the relationship across exchange. The internal class structure, function names, database schema, and source layout can remain implementation choices.
 Test vectors provide common evidence against which implementations can be compared. Together, conformance tests and test vectors turn architectural agreement into something participants can verify.
 ### Reference Implementations
 Reference implementations demonstrate one or more valid ways to implement the contracts. They provide working code, examples, fixtures, developer guidance, and operational experience.
 Their value comes from demonstration rather than authority. Independent implementations should be able to differ substantially internally while satisfying the same conformance requirements. When conformance begins to require reproduction of the reference implementation's internal structure, the specification has started to inherit implementation shape.
 ### Implementation Bindings and Adapters
 Bindings describe how Base Layer contracts are carried through particular technologies. A Nostr binding may describe how DSEARCH observations are represented, signed, published, retrieved, and synchronized through Nostr. An HTTP binding may express the same Base Layer contracts through HTTP resources and requests. A content-addressed storage binding may use another mechanism, while future systems may use technologies that have yet to be developed.
 Several viable bindings provide evidence that the Base Layer has retained meaningful independence from any one substrate. One binding may become overwhelmingly successful and remain the best answer for many years. Its success can be embraced while the shared meanings continue to belong to DSEARCH.
 ### Applications and Participants
 Applications, crawlers, search engines, archives, AI systems, ranking systems, specialist indexes, storage operators, and other participants operate above these shared contracts. They can make different choices about what information to collect, preserve, rank, display, or evaluate, and each may implement only the portions of the commons relevant to its function.
 Their independence is part of the architecture. Interoperability arises from the contracts they share.
 ## Locating Architectural Disagreement
 The hierarchy provides a way to identify where a disagreement belongs before attempting to resolve it.
 A dispute about whether independent observation should remain distinct from replication belongs at the level of Base Layer invariants. A dispute about how an observation represents retrieval time belongs at the contract or schema level. A dispute about whether a particular test actually proves the contract belongs at the conformance level. A dispute about the internal design of a reference implementation belongs at the implementation level. A dispute about whether Nostr, HTTP, a message bus, or another technology provides the best implementation of a contract belongs at the binding or substrate level.
 These questions influence one another, but locating them correctly prevents an implementation capability from silently settling an architectural question.
 A technology may have a useful identifier while DSEARCH still needs to determine whether that identifier has the required semantics for a resource, content object, observation, or participant. A network may replicate records effectively while DSEARCH still needs to define what replication means. A protocol may provide search while the Base Layer continues to define coverage, currentness, ranking, and evidentiary support according to the needs of the commons.
 The first useful question in such a disagreement is therefore:
 **At which layer is this decision being made?**
 Once that level is clear, the relevant architectural or implementation alternatives can be compared on the proper terms.
 ## Using the Vision as a Reference
 This document is intended to remain comparatively stable while lower layers evolve more frequently. A proposed architecture, dependency, funding tranche, implementation, or protocol binding can therefore cite it when explaining which commons principles the work advances and which architectural properties it preserves.
 A substrate proposal can be evaluated against the architecture described here according to the work it removes, the adaptation it introduces, and the semantic independence it preserves. A funding proposal can identify the portion of the hierarchy advanced by the funded work. A Base Layer specification can derive its invariants from these principles. A conformance project can identify which contracts it qualifies. An implementation comparison can measure several technologies against the same requirements without allowing the vocabulary of one candidate technology to define the problem in advance.
 This provides continuity across work that may otherwise look unrelated. Architecture research, cryptographic identity, replication experiments, crawler development, search interfaces, storage systems, AI consumption, conformance tooling, and implementation bindings can all contribute to one commons while remaining separate engineering efforts.
 The root document provides the common frame through which those efforts can be related.
 ## A Living Root
 Serving as the root architectural reference gives this document a special role, while the same principle of revisability applied elsewhere also applies here.
 Experience may expose a distinction that the vision failed to preserve. Independent implementation may reveal that an assumption cannot be expressed consistently. Security analysis may demonstrate that a governance or trust model needs revision. Operational evidence may show that a principle creates substantially greater complexity than anticipated.
 The vision can therefore change through explicit architectural revision. When a foundational principle changes, the project should recognize that the architecture of the commons is changing and examine the consequences for invariants, contracts, conformance, implementations, migrations, and participants downstream.
 Architectural change should enter this layer deliberately and with evidence rather than arriving indirectly through the behaviour of one implementation.
 This creates the same discipline at the top of the hierarchy that the document asks of the layers below it:
 `make meaning explicit → observe implementation evidence → revise deliberately`
 The document's stability therefore comes from conscious, evidenced, visible change.
 ## The Continuing Work
 The idealized vision establishes the direction of the commons. The continuing work is to make its properties progressively more explicit, testable, and independently implementable.
 The next layer identifies the Base Layer invariants implied by this document. Those invariants can then be expressed through the smallest useful contracts that preserve them. Test vectors and conformance tests can qualify those contracts before a single reference implementation becomes indistinguishable from the specification. Independent implementations can then provide evidence that the contracts are complete enough to reproduce their intended meaning across genuinely different internal architectures.
 Implementation substrates can be tested according to the useful machinery they supply and the adaptation work they introduce. Evidence from those experiments can be preserved so that architectural decisions remain open to later review under changed conditions.
 The purpose of this work is to create a search commons whose shared meanings remain stable enough for independent systems to cooperate and flexible enough for better implementations to replace earlier ones.
 That is the central architectural task of DSEARCH.