diff --git a/digital_search_commons.md b/digital_search_commons.md index 251f774..7c4d525 100644 --- a/digital_search_commons.md +++ b/digital_search_commons.md @@ -4,19 +4,45 @@ ## Terms of Reference -A **commons** is a shared environment constituted by a community of independently situated participants, a field of shared resources or capabilities, and established relationships through which those participants contribute, draw upon, preserve, and steward what is held in common. Participation takes place under commonly understood and accepted rules and practices that give the shared environment enough coherence to remain usable through time. +> A **commons** is a shared environment constituted by a community of independently-situated participants, [providing and using] a field of shared resources or capabilities, and [a corpus of] established relationships through which those participants contribute, draw upon, preserve, and steward [the entirety that] is held in common. Participation takes place under commonly understood and accepted rules and practices[, and these] give the shared [technical] environment enough coherence to remain [stable and] usable through time. -Those rules and practices require governance, but governance does not imply that every participant directly governs every rule. Some responsibilities demand specialized knowledge, sustained maintenance, adjudication, specification work, or administrative continuity. A commons can therefore combine broad participation with more concentrated stewardship of the structures that make participation possible. The relevant measure is whether those structures continue to serve the shared environment and preserve meaningful independence among its participants. +> Those rules and practices [demand] xrequirex governance, but governance does not imply that every participant directly governs every rule [or practice]. Some responsibilities [of governance] demand specialized knowledge, sustained maintenance, adjudication, specification work, or [an] administrative continuity. A commons can therefore combine broad [distributed] participation with more concentrated [specialised] stewardship of the structures that make [distributed] participation possible. The relevant measure [here] is whether those structures continue to serve the shared environment [including its aims and intents] and [so] preserve meaningful independence among its participants. -The word **digital** describes the principal medium in which the commons operates. A **digital commons** is therefore a commons in which the shared resources, records, relationships, interfaces, and means of participation are substantially represented, exchanged, processed, and used digitally. The digital character applies both to what participants share and to the mechanisms through which they interact with one another and with the shared environment. +> The word **digital** describes the principal medium [through] xinx which the commons operates. A **digital commons** is therefore a commons in which the shared resources, records, relationships, interfaces, and means of participation are substantially represented, exchanged, processed, and used digitally. [This] digital character applies both to what [the independent] participants [do and hold and] share and to the mechanisms [of the shared enviroment] through which they interact with one another [via that] xandwithx the shared environment. -A **digital search commons** is a digital commons organized around the domain of **search**: the discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information so that it can be found and used. Its participants may include people, organizations, crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, infrastructure operators, and other technical or institutional actors that contribute some useful part of that activity. +> A **digital search commons** is a digital commons organized around the domain of **search**: the discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information[;] so that [that information] xitx can be found and used. xItsx [The] participants [in a **digital search commons**] may include people, organizations, crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, infrastructure operators, and other technical or institutional actors[; all of which] xthatx contribute [to] some useful part of that activity. -Within this document, a **participant** is any independently-controlled person, organization, service, software agent, or technical system that acts within the commons under some identifiable endowed operational authority. Participants can occupy different roles. An **operator** controls or administers a technical role. An **observer** is a participant or process that examines a resource and produces an observation. These roles may coincide in a particular implementation, while their meanings of the implementation remain distinct because the commons may need to reason about them separately. +> Within this [present] document, a **participant** is any independently-controlled person, organization, service, software agent, or technical system that acts within the commons[.] [This action wil be done] under some identifiable operational authority [endowed to a participant for it to be done]. Participants can occupy different roles. An **operator** controls or administers a technical role. An **observer** is a participant or process that examines a resource and produces an observation. These roles may coincide in [any single] particular implementation [(deed)], while the xirx meanings [of these roles for] [that single] implementation remain distinct[.] [This is] because [participants in] the commons may need to reason about them separately. + +> **DSEARCH** is the architectural and specification effort directed toward making such a **digital search commons** possible. [**DSEARCH** is to] xItx define xsx the shared concepts, [the] Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons[.] [At the same time these indenpendent operators] xwhilex retain xingx their own [private] implementations, infrastructure, methods, and policies. + +> The conceptual relationship [for **DSEARCH**] is hierarchical. A **commons** supplies [it with] the underlying model of shared participation and stewardship. A **digital commons** applies that model to an environment whose resources, relationships, and interactions are substantially digital. A **digital search commons** specializes that environment around the requirements of search. **DSEARCH** [project is to] develop xsx the shared semantic and technical architecture through which independently operated systems can participate in th[e shared] xatx specialized commons. + +A **commons** is a shared environment constituted by a community of independently situated participants, a field of shared resources and capabilities that those participants provide and use, and an established body of relationships through which they contribute to, draw upon, preserve, and steward what is held in common. Participation takes place under commonly understood and accepted rules and practices that give the shared environment sufficient coherence, stability, and continuity to remain usable through time. + +Those rules and practices demand governance. Some responsibilities of governance require specialized knowledge, sustained maintenance, adjudication, specification work, or administrative continuity. A commons can therefore combine broad and distributed participation with more concentrated forms of specialized stewardship over the structures that make such participation possible. The relevant measure is whether that stewardship continues to serve the shared environment, its purposes, and the independence of the participants who constitute it. + +The word **digital** describes the principal medium through which the commons operates. A **digital commons** is therefore a commons in which shared resources, records, relationships, interfaces, and means of participation are substantially represented, exchanged, processed, and used digitally. This digital character applies both to what independent participants create, hold, use, and share, and to the mechanisms of the shared environment through which they interact with one another. + +A **digital search commons** is a digital commons organized around the domain of **search**: the discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information so that information can be found, understood, and used. Participants in a digital search commons may include people, organizations, crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, infrastructure operators, and other technical or institutional actors, each contributing some useful part of that activity. + +Within this document, a **participant** is an independently controlled person, organization, service, software agent, or technical system that acts within the commons under some identifiable operational authority. Participants may occupy different roles according to the activity being performed. An **operator** is the person, organization, or authority that controls or administers a technical participant. An **observer** is a participant, or a process acting under the authority of a participant, that examines a resource and produces an observation. These roles may coincide within a particular implementation, but their meanings remain distinct because the commons may need to reason separately about the actor, the operational authority behind it, and the act of observation itself. + +**DSEARCH** is the architectural and specification effort directed toward making such a **digital search commons** possible. It defines the shared concepts, Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons while retaining their own implementations, infrastructure, methods, internal terminology, and policies. + +The conceptual relationship underlying **DSEARCH** is hierarchical. A **commons** supplies the underlying model of shared participation and stewardship. A **digital commons** applies that model to an environment whose resources, relationships, and interactions are substantially digital. A **digital search commons** specializes that environment around the requirements of search. **DSEARCH** develops the shared semantic and technical architecture through which independently operated systems can participate in that specialized commons. + +Within that architecture, DSEARCH provides a common semantic meeting point rather than a common internal implementation. Independent operators may describe their own systems using different terms, identifiers, data models, schemas, and internal representations. At the boundary of participation, an adaptation layer maps those local meanings into the corresponding DSEARCH terms and contracts, and maps DSEARCH concepts back into forms meaningful to the local system. + +The hierarchy can therefore be understood as a sequence of increasingly specific foundations: + +`commons → digital commons → digital search commons → DSEARCH architecture → Base Layer terms and contracts → independent operator mappings` + +The **commons** supplies the social and institutional form. The **digital commons** supplies the digital medium. The **digital search commons** supplies the search domain. **DSEARCH** supplies the architecture for interoperability within that domain. The **Base Layer** supplies the shared semantic vocabulary and contracts through which participation becomes intelligible across independently operated systems. Operator mappings then connect those common meanings to the distinct terminology, structures, and machinery used within each implementation. + +The resulting arrangement preserves a deliberate balance: common meaning at the boundary, diversity within the participant. The commons gains interoperability without requiring internal uniformity, while independently operated systems retain the freedom to organize themselves according to their own purposes and circumstances. -**DSEARCH** is the architectural and specification effort directed toward making such a digital search commons possible. It defines the shared concepts, Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons, while retaining their own implementations, infrastructure, methods, and policies. -The conceptual relationship is hierarchical. A **commons** supplies the underlying model of shared participation and stewardship. A **digital commons** applies that model to an environment whose resources, relationships, and interactions are substantially digital. A **digital search commons** specializes that environment around the requirements of search. **DSEARCH** develops the shared semantic and technical architecture through which independently operated systems can participate in that specialized commons. Within that architecture, **DSEARCH terms** provide a common vocabulary at the interoperability boundary. Independent operators may use their own terminology, internal data models, identifiers, schemas, and representations. Their systems therefore need not describe their own operations internally in DSEARCH language. Instead, a translation or adaptation layer maps the relevant local concepts into DSEARCH concepts when information is contributed to the commons, and maps DSEARCH concepts into locally meaningful forms when information is consumed. @@ -30,6 +56,61 @@ Each level adds specificity while retaining what lies beneath it. The commons su This arrangement allows the digital search commons (**DSEARCH**) to obtain shared meaning without requiring uniformity from operators of incoming translatable individual terms. Operators remain free to describe and implement their own systems in forms suited to their purposes; **DSEARCH** provides the semantic meeting place within which those different systems can understand one another. +The **D** in **DSEARCH** refers to **decentralization**, but decentralization describes a structural property of the system rather than the meaning of the commons itself. A commons is a form of shared participation and stewardship. Decentralization is one of the architectural strategies DSEARCH uses to preserve independence, resilience, plurality, and freedom from capture within that commons. + +The distinction matters because a commons can contain concentrated infrastructure while retaining broadly shared participation and stewardship, just as technically distributed infrastructure can exist under highly concentrated architectural, economic, or governance authority. DSEARCH therefore treats decentralization as a set of separable properties whose value depends on what they preserve for the digital search commons. + +A related distinction applies to **distributed** systems. Distribution describes the placement of computation, storage, or activity across multiple systems or locations. Decentralization describes the distribution of effective control, dependency, and authority. A system can be highly distributed while remaining strongly centralized in the authority that defines, operates, or controls it. **DSEARCH** is concerned with both questions, while giving special attention to the forms of decentralization that preserve the independence of the commons. + +## Decentralization in DSEARCH + +The digital search commons and decentralization describe different aspects of the **DSEARCH** ideal. The **commons** describes the shared environment: its participants, shared meanings, evidence, resources, relationships, and practices of stewardship. **Decentralization** describes how control, dependency, implementation, and evidentiary authority are arranged within that environment. + +DSEARCH therefore treats decentralization as instrumental. Its purpose is to preserve useful properties of the commons rather than to satisfy an abstract requirement that every component be equally dispersed or that concentration never occur. A particular search engine may become extremely popular. One storage system may serve most participants for a period. A particular transport or implementation binding may become the practical default. Such concentration can be efficient and beneficial while participants retain realistic alternatives and while success in one layer does not acquire unintended authority over the others. + +The relevant question is therefore less whether the system is “decentralized” in the singular than **which forms of decentralization are present, what each one protects, and where concentration would create unacceptable dependence or authority**. + +**Infrastructure decentralization** concerns the physical and network machinery through which the commons operates. Storage, communication, crawling, indexing, synchronization, and other services can be supplied through multiple independently controlled systems and failure domains. The objective is practical continuity: the commons should have paths through which useful activity can continue when an individual provider, network, server, or technology becomes unavailable or unsuitable. Infrastructure decentralization therefore concerns dependence as much as numerical distribution. Ten servers under one administrative authority provide a different kind of resilience from ten independently controlled systems. + +**Implementation decentralization** concerns the ability of independently developed software to participate through the same shared contracts. The Base Layer provides common meaning at the interoperability boundary while allowing participants to choose their own internal terminology, software architecture, databases, programming languages, algorithms, and operational models. Several implementations of a contract provide more than redundancy. They demonstrate that the contract exists independently of the first implementation that expressed it. + +This is why a reference implementation serves as an example rather than as the effective constitution of the commons. When independent implementations can satisfy the same semantic contracts through substantially different internal designs, implementation authority remains distributed. When compatibility increasingly requires reproduction of one implementation's internal assumptions, practical decentralization has begun to contract even if many separately operated copies of the software exist. + +**Governance decentralization** concerns the relationship between architectural stewardship and practical authority over participating systems. DSEARCH requires some sustained architectural work: concepts have to be defined, contracts maintained, ambiguities resolved, conformance evidence produced, and successor specifications developed. Specialized stewardship can perform those tasks more effectively than attempting to make every architectural decision through commons-wide direct administration. + +The important decentralizing property appears in what that stewardship can command. A **DSEARCH** steward may publish a successor contract and provide compelling reasons to adopt it, but independently operated participants retain authority over their own implementations. As adoption grows, practical authority becomes increasingly distributed across those operators. Governance resilience therefore comes from transparent specification, conformance, compatibility, migration, and persuasion rather than from an expectation that architectural authority carries unilateral operational control. + +This also explains why architectural authority itself deserves continuing scrutiny. A steward that defines shared contracts performs a necessary coordinating function. The commons remains healthier when that function is constrained by explicit specifications, observable evidence, independent implementation, and practical opportunities for alternative interpretations or successors to be demonstrated. Stewardship supplies coherence; independent participation supplies a limit on the reach of that stewardship. + +**Economic decentralization** concerns the ability to participate without one commercial position becoming a compulsory gate to the commons. Participants may charge for services, build businesses, finance infrastructure, operate proprietary systems behind conforming interfaces, or become commercially dominant through superior execution. Economic activity can strengthen the commons by providing resources for operation and development. + +The decentralizing concern is capture rather than commerce. A commercial actor acquires architectural significance when access to an essential service, identifier, implementation, dataset, network, or other dependency becomes sufficiently concentrated that participation in practice depends upon that actor's continuing permission or commercial policy. DSEARCH should therefore preserve viable paths through which competing providers, self-operated systems, community infrastructure, and future alternatives can satisfy the same shared contracts. + +**DSEARCH** protects the commons by preserving economic plurality rather than prescribing an economic model. Tokenization is one possible mechanism available to participants. Its legitimacy comes from remaining optional and bounded to the resources and policies of those who adopt it. Non-tokenized participants retain full standing in the commons, while tokenized systems remain free to innovate above the same Base Layer. Economic success can produce influence through adoption, but neither tokens, stake, payment, nor market capitalization acquire inherent authority over shared semantics, evidence, or participation. + +**Evidentiary decentralization** concerns authority over what the commons can know. Search operates through observations and claims produced under differing conditions by different participants. DSEARCH therefore preserves provenance, observer identity, timing, independence, corroboration, contradiction, and other evidence that allows several views of a resource to be compared. + +No numerical count of nodes, keys, signatures, or copies establishes evidentiary decentralization by itself. One operator can control many systems. One observation can be replicated widely. Several services can derive their information from the same upstream source. Evidentiary decentralization depends on preserving enough information to distinguish genuine plurality of observation from plurality of representation. + +This property is particularly important because it reaches beyond technical topology. A system with thousands of independent servers could still possess a highly centralized evidentiary model if every server ultimately treats one source as authoritative. Conversely, a smaller infrastructure can preserve meaningful evidentiary plurality when independently controlled observers produce and expose distinct evidence that consumers can compare. + +These dimensions can vary independently. A system may have decentralized infrastructure and concentrated governance. It may have many independent implementations whose economic access depends upon one provider. It may have diverse operators while relying upon one representation that has become architecturally mandatory. It may have thousands of cryptographic identities while the apparent plurality of observations originates from a small number of controlling actors. + +DSEARCH therefore avoids treating decentralization as a binary label. The architecture asks what is distributed, what remains concentrated, why the concentration exists, what authority follows from it, and whether participants retain practical alternatives when conditions change. + +This permits deliberate concentration where concentration is useful. A widely adopted transport can reduce deployment cost. A strong reference implementation can accelerate participation. Specialized maintainers can improve the quality of shared contracts. A commercially successful service can make the commons substantially easier to use. These outcomes are compatible with the DSEARCH ideal when their success remains bounded by interoperable contracts and realistic replaceability. + +The boundary is **capture**. Concentration becomes architecturally significant when success in one layer gives an actor, technology, representation, or institution effective control over meanings or participation that properly belong to the wider commons. + +This produces a more useful objective than decentralization everywhere for its own sake: + +**DSEARCH seeks sufficient decentralization at each relevant layer to preserve independent participation, plural evidence, replaceable implementation, distributed practical authority, and freedom from compulsory gatekeepers.** + +The degree and form required at each layer can change with circumstances. What matters is that concentration remains visible, its consequences can be evaluated, and the architecture retains credible paths through which alternatives can emerge. + +Seen this way, decentralization is one means by which DSEARCH serves the digital search commons. The commons is the goal. Decentralization helps preserve the conditions under which that commons can remain diverse, interoperable, revisable, and resistant to capture. + + ## A Digital Search Commons The digital search commons described above is the intended environment; **DSEARCH** is a projected means of making that environment possible. The commons consists of its participating community, the shared search-related resources and evidence available to that community, and the relationships through which participants contribute, retrieve, preserve, compare, and use these things. DSEARCH supplies common structures that allow these activities to occur across independently designed systems. @@ -84,7 +165,8 @@ Three related forms of resilience express this concern. **Governance resilience** is the capacity of the shared architectural and specification process to evolve while independently controlled participants adopt changes according to different circumstances and schedules. Several contract versions, implementation generations, or capability sets may therefore coexist during periods of transition. Governance provides the means to define successors, communicate changes, support compatibility, and make migration tractable, while adoption remains an act performed by independently governed participants. -These forms of resilience reinforce one another. Technical resilience preserves operation through local failure. Architectural resilience preserves freedom of implementation through technological change. Governance resilience preserves the ability of the shared framework to evolve as authority over actual implementations becomes increasingly distributed. +These forms of resilience reinforce one another. Technical resilience preserves operation through local failure. Architectural resilience preserves freedom of implementation through technological change. Governance resilience preserves the ability of the shared framework to evolve as +over actual implementations becomes increasingly distributed. Together they serve two directions at once. They help the commons remain useful as participants, technologies, and operating conditions change, and they help DSEARCH preserve the principles on which that usefulness depends as technical choices exert pressure on the architecture. The project therefore responds to changing conditions while also carrying its architectural commitments forward into the choices made under those conditions. @@ -216,6 +298,16 @@ Successful adoption can make this progressively harder to reverse. Once substrat The defense is preventative: independently defined Base Layer contracts, replaceable adapters, independent semantic identities, conformance tests that validate meaning rather than source shape, explicit versioning, feasible migration paths, and continuing examination of the adaptation term in the complexity equation. +### Economic Capture + +**Economic capture occurs when a commercial participant's market position, pricing power, or control over some function the commons has come to depend on begins to determine the architecture of the commons, rather than merely operating within it.** + +The mechanism mirrors substrate capture closely enough that the same warning signs apply, with commercial success standing in for technical convenience. A commercial operator earns adoption by solving real problems well — perhaps by running the most reliable relay, maintaining the most complete index, or providing the infrastructure most other participants find it easiest to build on. Other participants come to depend on that operator's specific behavior, pricing, availability, and interface. As dependence deepens, architectural decisions that would threaten that operator's business begin to carry a cost the commons did not intend to create, and the boundary between "this operator's product" and "the definition of the commons" becomes difficult to state, test, or reproduce independently — the same test that distinguishes substrate capture from ordinary heavy use of a technology, applied here to ordinary commercial success. + +Economic capture is therefore distinct from a commercial participant simply becoming widely used, in exactly the way substrate capture is distinct from a technology simply being widely used. Wide adoption earned by providing genuine value is the expected and healthy outcome of open competition within the commons. Capture is the further step in which that adoption becomes leverage over what the commons is permitted to mean or require. + +The defense follows the same preventative logic as the defense against substrate capture, extended to economic relationships. The Base Layer should remain implementable without payment to any party, so that a dominant commercial operator's advantage rests on the quality of its service rather than on gatekeeping access to the commons itself. No single implementation, however successful, should be treated as the only conforming one. Governance bodies responsible for Base Layer invariants should not be funded or controlled exclusively by any one commercial interest, for the same reason a specification should not be authored solely by the vendor whose product it is likely to bless. And a commercial operator's pricing, access policies, or product decisions should remain its own business choices rather than being allowed to redefine what a Base Layer term such as observation, coverage, or reputation means for the commons as a whole. + ### Fragmented Security Surface Independent implementation distributes both capability and security responsibility. This distribution strengthens the commons when failures remain contained and trust boundaries remain explicit. @@ -226,6 +318,26 @@ The defense is containment and explicit trust. Outside input should be validated Security of the commons therefore grows from isolation, constrained authority, and verifiable interaction across independently operated systems. +### Resource Exhaustion and Denial of Service + +A commons built on independently operated infrastructure can be attacked by exhausting that infrastructure rather than by corrupting its meaning. This threat runs in two directions, and a search commons has to defend against both — including one direction that is unusual to this kind of system. + +**The commons as target.** Any individual relay, index, crawler, or API can be flooded with connections, queries, or publication requests intended to make it unavailable to legitimate participants. Because the commons depends on many independently operated services rather than one central one, an attack against a single operator should not by itself threaten the commons — this is exactly what technical resilience and the containment principles under Fragmented Security Surface are meant to provide. A related but distinct version of this threat operates on data rather than network traffic: a participant can publish an overwhelming volume of low-quality or fabricated observations, straining the storage, indexing, and synchronization capacity that other participants must expend to process them. This data-volume attack is often paired with manufactured plurality, since many apparent identities publishing at volume can be harder to distinguish from genuine growth in participation than a single source flooding traffic from one place. + +**The commons as attacker.** A search commons exists to encourage independent examination of resources, and that purpose can turn against the resources being examined. Many independently operated crawlers, each behaving reasonably by its own local standard, can collectively re-examine the same small or lightly resourced target so often that their combined activity becomes indistinguishable from a distributed denial-of-service attack — with no single participant responsible and no single participant even aware that a problem exists. This risk is distinctive to a commons whose stated purpose is to encourage observation, and it deserves attention in its own right rather than being treated as a special case of the first direction. + +The defenses for both directions share a common foundation, and several of them can reuse concepts the Base Layer already defines rather than introducing new machinery solely for this purpose. + +Rate limiting, request budgets, and cost mechanisms such as proof-of-work or reputation-gated throughput can blunt flooding in either direction at the point of ingestion or retrieval. + +Isolation and local containment, as already described under Fragmented Security Surface, keep an attack against one operator from cascading into a commons-wide failure. + +**Coverage and currentness information can double as coordination**, giving the commons a defense specific to its second direction. A participant considering whether to examine a resource can consult existing observation and coverage records to see that the resource was recently and adequately examined by another observer, reducing redundant re-examination without requiring any central scheduling authority. This reuses evidentiary concepts already defined in the Base Layer rather than adding a new coordination protocol. + +**Observation etiquette can become an expectation of the observer role.** Where a resource expresses its own access preferences or capacity limits, participants acting as observers can be expected to honor them as part of what it means to conform to the observer role in good standing, similar in spirit to established crawling conventions but expressed as a Base Layer expectation rather than left entirely to individual operator discretion. + +A resilient commons therefore treats denial of service not only as an attack to repel but as a foreseeable consequence of its own success that has to be designed around from the outset. + ### Assumed Erasure Information copied to independently operated systems can remain available after one participant withdraws it. This property creates both technical and social consequences because withdrawal, persistence, and current standing describe different states. @@ -312,7 +424,8 @@ These principles provide the context in which lower-level decisions are evaluate ### Base Layer Invariants The next level converts the general principles into statements that should remain true across conforming implementations. Examples include `publication != replication`, `signed != true`, `replication != independent observation`, `resource identity != content identity != observation identity`, `retrieval success != currentness`, `reputation != truth authority`, and `deprecation != migration`. - remaining independent of wire format or implementation technology. They establish boundaries that later contracts are expected to preserve. + +These invariants are more precise than the vision while remaining independent of wire format or implementation technology. They establish boundaries that later contracts are expected to preserve. ### Base Layer Contracts and Schemas