dsearch/documents
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update digital_search_commons.md

Browse Source
This commit is contained in:
anomalist
2026-09-03 11:04:09 +00:00
parent 4bde4e095e
commit 71770d73c4
1 changed files with 25 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
digital_search_commons.md
+25 -65
View File
@@ -1,51 +1,3 @@
> # DSEARCH: The Creation of a Digital Search Commons
> **An idealized vision**
> ## Terms of Reference
> A **commons** is a shared environment in which independently ?situated? [engaged?] participants contribute to, draw from, preserve, and govern shared resources through commonly understood [and accepted] rules and practices.
> [These rules and practices are not themselves within the commons; because of demands such as need for specialised knowledge to set them up and to maintain them; and the fact that the practical unweildines of a commons wide government of rules and practice, however politically ideal, raises demands strongly difficult to be overcome.]
> A **digital** commons is a commons whose shared resources, records, relationships [between user-entities and between the parts of the commons, including] the means of participation are substantially digital[and are exchanged and moved and used **digitally**.
> A digital**search** commons is a digital commons organized around the discovery, observation, description, preservation, exchange, comparison, and [ad hoc? Contingent? utilitarian?] evaluation of information [to do with] **search**. Its [multitude of] participants [usually] include[s] crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, and[also] infrastructure operators. [Plus many other parts]
> **DSEARCH** is an effort [at] defin[ing] the shared concepts, contracts, and [considerable] interoperability required [in] buil[ding] systems [allowing user-entities independent] participation in such a **digital search commons**.
> The relationship among these four terms [(**commons** **digital** **search** and **digital search commons**)] is hierarchical. A **commons** [founding the rising build pyramid at the base] provides the general model of shared participation and [joint] stewardship. A**digital** commons applies that model to digital resources and relationships [at the next lesser level of foundation in the pyramid]. A digital **search** commons applies [the **commons** and **digital** ?foundations?] to [that entailed in] the **search** domain [sat at the next level up the pyramid of foundations] [is domain ambiguous?]. DSEARCH is [then] the architectural undertaking intend[ing] to make [the] specialized commons [outlined here] interoperable across independent implementations.[Is this enough to define DSEARCH? A question not a complaint]
> ## A Digital Search Commons
> A **digital search commons** is a shared environment in which independently operating entities discover, observe, describe, exchange, compare, preserve, and evaluate information **through common structures and contracts**. Its purpose is to make search-related information usable across organizational[is 'organisational' too unspecific?], technical, and geographic boundaries while allowing each participant [-entity] to retain control of its own infrastructure, methods, policies, and implementation choices [; which are being used in accessing DSEARCH]. Crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, and other participants can therefore contribute to and consume from, within] the same commons [and] without first becoming [maleable? chattel-like? unconsidered?] parts of one [monolithic] vertically integrated system. [Bit of rhetorical colour Matt, sorry]
> The commons exists at a level above any particular search engine, index, database, crawler, protocol, company, or network [which is present within it]. Those things participate [providing parts for working] the commons and implement[ing] portions of it. The commons itself consists of the shared meanings, records, relationships, and rules through which independently built systems can understand one another[The commons Matt? The commons maybe first and foremost is the whole thing? The body of user-entities and the services which DSEARCH provides them with? The 'nuts and bolts' are just means to the commons?]
> At its foundation, the ?commons? [is to] distinguish the things being searched from the evidence collected about them. A **resource** identifies [is this a 'resource' is a thing that can be examined, or a 'resource'' 'picks out' a thing to be examined?] something that can be examined. **Content** identifies material obtained from that resource. [These are terminology but are not operations?] An **observation** records what a participant found when it examined the resource at a particular time. Additional records can [label and harbour data on?] describe provenance, relationships among observations, replication, coverage, currentness, corroboration, contradiction, reputation, and other properties that help participants [participants = user-entities? or is the evaluation done unseen and automatedly within the mechanisims?] evaluate the available evidence.
> [common](n.)https://www.etymonline.com/word/commons
> c. 1300, "a fellowship or brotherhood; early 14c., "people of a community or town, freemen, citizenry;" late 15c., "land held in common," from Old French commune and Medieval Latin communia, and partly from common (adj.). **Also compare commons**. Latin communis "common, general" (adj.) also served as a noun meaning "common property; state, commonwealth."]
> These ?common[s]? structures allow different participants [persons?] to contribute different capabilities[is this lower and higher or is it variety alone?] while preserving specialization[. Are specialisations presumed of a higher order than are'capabilities' here?]. One participant [ a person maybe very useful per se but has no adequate machinery? I say this to separate the person from the means?] may focus on crawling, another on historical preservation, another on a regional or subject-specific index, another on ranking, another on machine analysis, and another on a public search interface. Their [the persons'?] usefulness to the commons [is this the prime objective? usefulness to the commons?] comes from their ability to [pool, for free - in all senses?] exchange[,] compatible information while continuing to operate independently.
> The ideal [**DSEARCH** allows, encourages, lays the ground for? therefore **diversity with interoperability**. Independent implementations can use [personal choice?] databases, programming languages, transports, storage systems, ranking methods, crawling strategies, and operational models while [at the same time being able to share] enough common meaning to [enable] exchange and [to make possible acts of evaluation of] search information. A crawler can produce observations while remaining a crawler. An archive can preserve content while remaining an archive. [The previous two sentences don't add anything in my eyes. They are meant to mean something but don't] A ranking system can evaluate evidence while retaining its own ranking method. A transport can carry observations while the Base Layer continues to define what those observations mean.
> This separation gives the ?commons? [it's the machinery and its generation and storage of items we are talking about here?] a way to evolve through addition, replacement, and coexistence [a new term? Is this people or data?]. New crawlers, storage systems, protocols, indexes, ranking methods, and applications can enter the **DSEARCH** environment when they offer useful capabilities [This implies a filter refusing items not up to scratch?] Existing mechanisms can continue operating while alternatives are tested [This sentence is in a vacuum. The talk has been of DSEARCH and Commons and of individual divergent contributors contributing to these. The sentence gives no idea which of these items it is taking about?]. A successful implementation may become widely used while the commons remains larger than that implementation.[Yes DSEARCH is a means. The commons is the goal]
> The commons [is this DSEARCH in fact?] [will] also [be able to] accommodate disagreement [is disagreement a good word here? if it means something technically incongruous.Is this the AI being clever? It is to human a word for mechanics? Especialy when coupled with 'accommodation'?] . Independent observers [new term???}] may retrieve different [separate - the word different applies to a comparison so it should be different content from one another's. Separate is less confusing?] content, report different conditions, or reach different conclusions [from one another]. Their records can [happily] coexist [in DSEARCH and be compared through provenance, timing, identity, and other evidence. [Their assessments might differ from each other's but not necessarily conflict - disagree] Agreement can strengthen confidence where the evidence supports that conclusion [is agreeement a conclusion in this context here. Agreements mutuially support views of parties involved in them?], while contradiction remains visible and available for further evaluation.[Difffering is not always contradiction]
> Search results can consequently become more than answers returned by an opaque [new term. Opaqueness has not yet been opened up as a factor re DSEARCH] index. They can be connected to an evidentiary environment that shows which resources [have been] examined, which content [has been] observed, [at what times] observations occurred, which observers [saw recorded captured?] produced them, how [diverse] observations relate to one another, how widely they are replicated [copies of others], where[abouts search scope] coverage appears strong or weak, and how current the available evidence [is appears correct? Surely everything is dated precisely?] appears to be.
> The **Base Layer** of DSEARCH [build would] exist to make the **digital search commons**?] environment possible. It [would] define the [unique to DSEARCH, yet global within DSEARCH] concepts and contracts through which independent systems [using some transitionals] [will be able to] participate in the commons. Implementations [User-entities?] remain free to choose the mechanisms that suit their purposes, while the common[s] structures [comprising DSEARCH] preserve interoperability across those [discrete] choices [which suit].
> The idealized objective [for DSEARCH} is therefore a search environment [with] strength [which] comes from independent [and vareigated] participation, [from] compatible evidence, [and] diverse implementations, and [from a] continuing [potentiality] to change. The commons gains usefulness [robustness? durability? relevance?] from the number and variety of systems that can contribute to it. [Its resilience] comes from [a studied technical diligence which will preserve for DSEARCH resort that is always open to change, change such as might involve a choice from new] multiple paths for implementation, operation, and future revision.
> Three forms of resilience follow from this [idealised] objective. [These are:] **Technical resilience** [which] allows individual participants, services, copies, and implementations to fail while the wider commons continues to operate. **Architectural resilience** [which] allows technologies and implementations to be replaced while shared meanings remain [current and] intelligible. **Governance resilience** [which] allows several generations [is this only time or is it also additional developement?] of [DSEARCH] implementation [is this static or involves a progression?] to coexist while architectural change and migration proceed across independently operated participants. [The final sentence here is in need of attention. Unless the implementation is being considered to be in stasis and generations are being considered as time periods only, the connecting thought central to the sentence cannot hold up. But if these things are being considered in these ways, it is a view false to practical reality. Either way the sentence fails. Also the changes do not proceed across independently operated participants. Over the heads of them maybe? ]
> These three forms express different aspects of the same goal: preserving the usefulness of the commons [to] participants, [as] technologies, and operating conditions change.[This sentence is not the whole of the matter. Keeping to the ideals as goals for the DSEARCH project is a prime concern. This concern will impinge on choices made for technology and operating conditions through DSEARCH's lifetime? It's a two way street. DSEARCH serves participants but it also in doing so advocates its values to them] Architectural resilience governs the relationship between meaning and implementation [I don't understand?]. Governance resilience governs change across independent operators.[This is so compresed it is misleading] Technical resilience governs containment of failure and resist[s] technical and social attempts to compromise the [DSEARCH] system. Some important failure modes [will] cross all three [resilience] boundaries, because a decision that begins as an architectural convenience [is a failure mode a convenience?] can later acquire operational and governance consequences.
# DSEARCH: The Creation of a Digital Search Commons
**An idealized vision**
@@ -60,41 +12,49 @@ The word **digital** describes the principal medium in which the commons operate
A **digital search commons** is a digital commons organized around the domain of **search**: the discovery, observation, description, preservation, exchange, comparison, retrieval, and context-dependent evaluation of information so that it can be found and used. Its participants may include people, organizations, crawlers, search engines, archives, applications, AI systems, specialist indexes, researchers, infrastructure operators, and other technical or institutional actors that contribute some useful part of that activity.
Within this document, a **participant** is any independently controlled person, organization, service, software agent, or technical system that acts within the commons under some identifiable operational authority. Participants can occupy different roles. An **operator** controls or administers a technical participant. An **observer** is a participant or process that examines a resource and produces an observation. These roles may coincide in a particular implementation, while their meanings remain distinct because the commons may need to reason about them separately.
Within this document, a **participant** is any independently-controlled person, organization, service, software agent, or technical system that acts within the commons under some identifiable endowed operational authority. Participants can occupy different roles. An **operator** controls or administers a technical role. An **observer** is a participant or process that examines a resource and produces an observation. These roles may coincide in a particular implementation, while their meanings of the implementation remain distinct because the commons may need to reason about them separately.
**DSEARCH** is the architectural and specification effort directed toward making such a digital search commons possible. It defines the shared concepts, Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons while retaining their own implementations, infrastructure, methods, and policies.
**DSEARCH** is the architectural and specification effort directed toward making such a digital search commons possible. It defines the shared concepts, Base Layer contracts, interoperability requirements, and supporting means of conformance through which independently built and independently operated systems can participate in the commons, while retaining their own implementations, infrastructure, methods, and policies.
The relationship among these terms is hierarchical. The **commons** supplies the underlying model of shared participation and stewardship. The **digital commons** applies that model to an environment whose resources and interactions are substantially digital. The **digital search commons** specializes that environment around the requirements of search. **DSEARCH** is the architectural undertaking that develops the common meanings and technical agreements required for independently operated systems to participate in that specialized commons.
The conceptual relationship is hierarchical. A **commons** supplies the underlying model of shared participation and stewardship. A **digital commons** applies that model to an environment whose resources, relationships, and interactions are substantially digital. A **digital search commons** specializes that environment around the requirements of search. **DSEARCH** develops the shared semantic and technical architecture through which independently operated systems can participate in that specialized commons.
Within that architecture, **DSEARCH terms** provide a common vocabulary at the interoperability boundary. Independent operators may use their own terminology, internal data models, identifiers, schemas, and representations. Their systems therefore need not describe their own operations internally in DSEARCH language. Instead, a translation or adaptation layer maps the relevant local concepts into DSEARCH concepts when information is contributed to the commons, and maps DSEARCH concepts into locally meaningful forms when information is consumed.
The purpose of the common vocabulary is semantic interoperability. A crawler may internally call something a `fetch`, another system may call the corresponding act a `capture`, and another may represent it through several database objects. Where each of those local structures satisfies the meaning of a DSEARCH **observation**, their adapters can express them through the same Base Layer contract. The shared DSEARCH term describes the interoperable meaning rather than prescribing the operator's internal implementation.
The hierarchy can therefore be pictured as a rising set of foundations:
`commons → digital commons → digital search commons → DSEARCH architecture enabling participation`
`commons → digital commons → digital search commons → DSEARCH architecture → Base Layer terms and contracts → operator mappings`
Each level adds specificity while retaining what lies beneath it. The commons supplies the social and institutional form. Digital operation supplies the medium. Search supplies the domain. DSEARCH supplies the architecture through which that domain can function as an interoperable commons.
Each level adds specificity while retaining what lies beneath it. The commons supplies the social and institutional form. Digital operation supplies the medium. Search supplies the domain. Overarching **DSEARCH** contains the architecture for interoperability. The Base Layer is the common meanings through which participation becomes intelligible across systems, while operator mappings connect those meanings to the diverse terminology and structures used by independently developed implementations.
This arrangement allows the digital search commons (**DSEARCH**) to obtain shared meaning without requiring uniformity from operators of incoming translatable individual terms. Operators remain free to describe and implement their own systems in forms suited to their purposes; **DSEARCH** provides the semantic meeting place within which those different systems can understand one another.
## A Digital Search Commons
The digital search commons described above is the intended environment; DSEARCH is one means of making that environment possible. The commons consists of its participating community, the shared search-related resources and evidence available to that community, and the relationships through which participants contribute, retrieve, preserve, compare, and use them. DSEARCH supplies common structures that allow those activities to occur across independently designed systems.
The digital search commons described above is the intended environment; **DSEARCH** is a projected means of making that environment possible. The commons consists of its participating community, the shared search-related resources and evidence available to that community, and the relationships through which participants contribute, retrieve, preserve, compare, and use these things. DSEARCH supplies common structures that allow these activities to occur across independently designed systems.
Its purpose is to make search-related information usable across administrative, technical, and geographic boundaries while preserving the operational independence of participants. Crawlers, archives, search engines, applications, AI systems, specialist indexes, researchers, and infrastructure operators can therefore contribute different capabilities to the same environment while retaining control of the machinery and methods through which they participate.
Its purpose is to make search-related information usable across administrative, technical, and geographic boundaries while preserving the operational independence of participants. Crawlers, archives, search engines, applications, AI systems, specialist indexes, researchers, and infrastructure operators can therefore contribute different capabilities to the same environment while retaining control of the machinery and methods through which they discretely participate.
The commons is consequently larger than any particular search engine, index, database, crawler, protocol, company, network, or DSEARCH implementation. These provide capabilities within the commons. The commons itself is the larger socio-technical environment formed by the participants, the shared information and evidence available among them, and the interoperable relationships that allow those contributions to become useful beyond the system that originally produced them.
The commons is consequently larger than any particular search engine, index, database, crawler, protocol, company, network, or particular **DSEARCH** implementation. These are parts of **DSEARCH** and contribute capabilities within the commons. The commons itself is the larger socio-technical environment formed by the participants, the shared information and evidence available among them, and the interoperable relationships that allow those contributions to become useful beyond the system that originally produced them.
DSEARCH is concerned especially with those interoperable relationships. It seeks to provide enough shared meaning that independently built systems can understand what another participant is asserting, while leaving substantial freedom over how each participant performs its own work.
(Note: For purposes of utility the mechanical parts of DSEARCH are to be accepted as part of the commons.)
At the foundation of search evidence are several concepts that need to remain distinguishable. A **resource** is something identifiable that can become the subject of discovery, retrieval, or observation. A **resource identity** identifies that subject across relevant acts of search or observation. **Content** is material obtained from, associated with, or presented by a resource under particular conditions. A **content identity** identifies particular content independently of the resource through which it was obtained.
**DSEARCH** is concerned especially with those interoperable relationships. It seeks to provide enough shared meaning that independently built systems can understand what another independent participant is asserting, while leaving substantial freedom over how each participant performs its own work.
An **observation** is a record of an act in which an observer examined a resource at a particular time and under describable conditions. The observation can identify the resource examined, the content obtained, the observer responsible for the act, the time at which it occurred, and other information needed to interpret the resulting evidence. An **observation identity** identifies that particular evidentiary act or record.
At the foundation of search-evidence are several concepts that need to remain distinguishable. A **resource** is something identifiable that can become the subject of discovery, retrieval, or observation. A **resource identity** identifies that subject **resource** across relevant acts of search or observation. **Content** is material obtained from, associated with, or presented by a resource and under particular conditions. A **content identity** identifies particular content independently of the resource through which it was obtained.
These distinctions are semantic rather than merely terminological. A resource can present changing content. Identical content can appear through several resources. Several observers can examine the same resource and obtain the same content while still producing several observations. DSEARCH therefore preserves the identities of resource, content, and observation separately because each answers a different question.
An **observation** is a record of an act in which an observer examined a resource at a particular time and under described conditions. The observation can identify the resource examined, the content obtained, the observer responsible for the act, the time at which it occurred, and other information needed to interpret the resulting evidence for search use. An **observation identity** identifies that particular evidentiary act or record.
Other records and relationships can provide provenance, replication information, coverage claims, timing, corroboration, contradiction, reputation, operator relationships, and other evidence useful in interpreting what has been observed. Some evaluation will occur automatically inside crawlers, indexes, ranking systems, AI systems, and other mechanisms; some will occur through decisions made by people or organizations. The Base Layer should preserve the evidence required for these different forms of evaluation while allowing each consumer to apply methods suited to its own purpose.
These distinctions are semantic as well as terminological. A resource can present fluidly-changing content. Identical content can appear through several resources. Several observers can examine the same resource and obtain the same content while still producing several discrete observations. **DSEARCH** therefore preserves diverse identities of resource, content, and observation separately because each of them answers a different question.
The shared structures created through DSEARCH allow participants to contribute capabilities of different kinds and at different levels of specialization. One participant may operate a broad crawler. Another may concentrate on historical preservation. Another may maintain an index for a particular language, geography, subject, or collection. Another may contribute storage, ranking, analysis, provenance assessment, or an interface through which people search the accumulated evidence.
Other **DSEARCH** records and relationships can provide provenance, replication information, coverage claims, timing, corroboration, contradiction, reputation, operator relationships, and other evidence useful in interpreting what has been observed. Some evaluation will occur automatically inside independent operators' crawlers, indexes, ranking systems, AI systems, and other mechanisms; some will occur through decisions made by people or organizations. The architecture carries the main bulk of evaluation. Its Base Layer should preserve the evidence required for these different forms of evaluation, while allowing each consumer to apply methods suited to its own purpose.
The shared structures created through **DSEARCH** allow participants to contribute capabilities of different kinds and at different levels of specialization. One participant may operate a broad crawler. Another may concentrate on historical preservation. Another may maintain an index for a particular language, geography, subject, or collection. Another may contribute storage, ranking, analysis, provenance assessment, or an interface through which a searcher searches the accumulated evidence.
A participant's value to the commons therefore comes from the useful capability or evidence it makes available and from the degree to which that contribution can be understood and used beyond its own implementation. Specialized participants need only perform the roles they are suited to perform. Their contributions become part of a larger search environment because shared contracts allow those specialized capabilities to interoperate.
The ideal can therefore be described as **diversity with interoperability**. DSEARCH creates the architectural ground on which independently chosen databases, programming languages, transports, storage systems, ranking methods, crawling strategies, analysis methods, and operational models can exchange compatible search information. The independence lies in how participants perform their work; the interoperability lies in the meanings and boundary contracts through which their work can become useful to others.
The ideal can therefore be described as **diversity with interoperability**. **DSEARCH** supplies the architectural ground on which independently chosen databases, programming languages, transports, storage systems, ranking methods, crawling strategies, analysis methods, and operational models can exchange mutually compatible search information. The independence lies in how participants are enabled to perform their work; the interoperability lies in the meanings and boundary contracts through which their work can become useful to others.
This distinction matters because a digital search commons gains much of its strength from specialization. A crawler can concentrate on producing reliable observations. An archive can concentrate on durable preservation. A ranking system can concentrate on evaluating available evidence according to a declared method. A transport can concentrate on moving records. The Base Layer provides the common meanings that allow these separate capabilities to form a larger whole.
@@ -131,6 +91,7 @@ Together they serve two directions at once. They help the commons remain useful
A single decision can affect all three forms of resilience. A convenient implementation choice may initially improve technical capability, later become embedded in the architecture, and eventually create a governance problem when independently operated participants have organized their systems around it. The purpose of distinguishing these forms is therefore analytical: it gives DSEARCH several perspectives from which to examine the consequences of decisions whose effects may unfold over many years.
## Architecture of the Commons
A durable digital search commons begins with **agreements about meaning**. Those agreements define what a resource is, what an observation represents, what replication accomplishes, what a claim of coverage establishes, how provenance is expressed, and which relationships among records participating systems are expected to understand. The technologies used to implement those agreements can change many times during the life of the commons, while the meanings need enough stability to remain intelligible across those changes.
@@ -351,8 +312,7 @@ These principles provide the context in which lower-level decisions are evaluate
### Base Layer Invariants
The next level converts the general principles into statements that should remain true across conforming implementations. Examples include `publication != replication`, `signed != true`, `replication != independent observation`, `resource identity != content identity != observation identity`, `retrieval success != currentness`, `reputation != truth authority`, and `deprecation != migration`.
These invariants are more precise than the vision while remaining independent of wire format or implementation technology. They establish boundaries that later contracts are expected to preserve.
remaining independent of wire format or implementation technology. They establish boundaries that later contracts are expected to preserve.
### Base Layer Contracts and Schemas