fix and finish the alertSearch query & results

This commit is contained in:
2026-10-16 09:14:44 -06:00
parent 0c67bfbc9b
commit e14a65d210
29 changed files with 662 additions and 81 deletions
+9
View File
@@ -9,9 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Optional `MESSAGE_ENV_LABEL` appends ` (<label>)` to every SMS and push body, so a recipient can tell which server sent it; unset or blank appends nothing. `NODE_ENV` cannot serve, since the Docker image sets it to `production` on every server.
- Both alert-authorization routes accept optional `endorserAfterCursor`, `partnerAfterCursor`, `planHandleIds` and `location`; the cursors replace the channel's stored cursors, and the other two are sent to alertSearch on every run.
- The SMS link and the push data carry `c`, the cursors the run counted from.
- `profilesSimilar` and `plansSimilar` count in the digest.
### Changed
- The SMS digest links to `https://giftopia.me/deep-link/new-activity` instead of the bare `https://giftopia.me`.
- A first run (no stored cursor) only stores the cursor and reports nothing, instead of reporting the server's most recent rows.
### Fixed
- The daily run sent no starred plans or area, so the tracked-plan and nearby lists were always empty.
- When one source completed and the other failed, the completed source's cursor moved anyway, and its rows were never counted on the retry; both cursors now move together or not at all.
- A batch uploaded after today's run installed an unused JWT for today, so the run (and the push) could repeat the same day; days with a consumed JWT are skipped.
### Changed in DB or environment
- `alert_search_cursors` and `sms_alert_search_cursors` gain `plan_handle_ids` and `location` columns, added on startup.
- `MESSAGE_ENV_LABEL` (see above) on test environment
+23 -7
View File
@@ -70,7 +70,22 @@ Authorization: Bearer <current-user-JWT>
`notifyHourUtc` and `notifyMinuteUtc` are **required** and `timezone` is optional; all three behave exactly as on [the SMS twin](#endpoints). Both channels gate on the stored hour ([Notification hour](#notification-hour)).
A successful call replaces that user's previous **unused** JWTs atomically. Passkey (`did:peer`) identities cannot mint this batch and receive `DELEGATED_JWT_UNSUPPORTED_IDENTITY`.
A successful call replaces that user's previous **unused** JWTs atomically. JWTs for a UTC day that already holds a consumed JWT are skipped, so a re-upload after today's run does not run (and notify) again today; `storedCount` reports what was inserted. Passkey (`did:peer`) identities cannot mint this batch and receive `DELEGATED_JWT_UNSUPPORTED_IDENTITY`.
The body may also carry the channel's alertSearch state, on both alert-authorization routes:
```json
{
"endorserAfterCursor": "48213",
"partnerAfterCursor": "912-48213",
"planHandleIds": ["https://endorser.ch/entity/01H..."],
"location": { "minLocLat": 40.1, "maxLocLat": 40.9, "minLocLon": -112.2, "maxLocLon": -111.4 }
}
```
- The cursors replace the channel's stored cursors, so the next run counts changes after them. The app sends the latest cursors when notifications are turned on and when the person opens New Activity. A cursor is at most 64 letters, digits, `_`, `:` or `-`.
- `planHandleIds` (at most 200) and `location` are sent to alertSearch on every run, for the starred-project and nearby lists. `null` clears either one.
- An absent field keeps what is stored. Any invalid field rejects the whole upload, and nothing is stored.
### Removing an authorization
@@ -86,13 +101,13 @@ The daily scheduler runs `retrieveAlertSearch` against:
- `{ENDORSER_URL}/api/v2/report/alertSearch`
- `{PARTNER_URL}/api/partner/alertSearch`
The **delegated** JWT is sent as `Authorization: Bearer`. Each source takes its own cursor, `endorserAfterCursor` / `partnerAfterCursor`: an opaque string the server returned as `nextAfterCursor` and receives back as `afterCursor` (endorser-ch CHANGE/PLAN-accurate-cursors.md). With a cursor, a source pages while `hasMore` (up to 20 pages; the rest comes on the next run). Without one (a user's first run), it makes one request without a cursor, which returns the server's most recent rows and the cursor to start from. A response without `nextAfterCursor` (endorser-ch through 4.4.6) is `malformed`. Nearby search uses the alertSearch bbox (`minLocLat`, `maxLocLat`, `minLocLon`, `maxLocLon`).
The **delegated** JWT is sent as `Authorization: Bearer`. Each source takes its own cursor, `endorserAfterCursor` / `partnerAfterCursor`: an opaque string the server returned as `nextAfterCursor` and receives back as `afterCursor` (endorser-ch CHANGE/PLAN-accurate-cursors.md). With a cursor, a source pages while `hasMore` (up to 20 pages; the rest comes on the next run). Without one (a user's first run), it makes one request without a cursor and keeps only the returned cursor: the rows predate the notifications, so a first run reports nothing (`seeded: true`, outcome `empty`). A response without `nextAfterCursor` (endorser-ch through 4.4.6) is `malformed`. Nearby search uses the stored `location` bbox (`minLocLat`, `maxLocLat`, `minLocLon`, `maxLocLon`), and the tracked-plan lists use the stored `planHandleIds`; without them those lists are empty.
`loadAlertSearchCursors` / `retrieveAlertSearch` / `advanceAlertSearchCursors` (or `runAlertSearchCycle`) persist those cursors per user DID in SQLite. A cursor moves to the last `nextAfterCursor` after a `success` or `empty` run, and stays after `pagination` (the server's userMessage says a list timed out) or an error.
`loadAlertSearchCursors` / `retrieveAlertSearch` / `advanceAlertSearchCursors` (or `runAlertSearchCycle`) persist those cursors per user DID in SQLite. Both cursors move to their last `nextAfterCursor` only when both sources had a `success` or `empty` run. After `pagination` (the server's userMessage says a list timed out) or an error on either side, both stay, so the retry searches the whole window again rather than skipping what the completed side found. Because the cursors move after every complete run, sent or not, each notification counts only changes after the previous run.
`runDailyAlertSearch(userId, now?)` picks the unused delegated JWT for the current **UTC** day, runs `runAlertSearchCycle` with that JWT, and marks that specific JWT consumed only when both Endorser and Partner complete (`success` or `empty`, including both empty). Pagination or source failures leave the JWT unused so the same day can be retried. The result reports the day it used as `utcDay`. After a retrieve, the result includes `digest` from `buildAlertSearchDigest` (six bucket records and counts). `digest` is `null` when there is no batch or no unused JWT for today. Consumption does not depend on `digest.hasUpdates`.
`runDailyAlertSearch(userId, now?)` picks the unused delegated JWT for the current **UTC** day, runs `runAlertSearchCycle` with that JWT, and marks that specific JWT consumed only when both Endorser and Partner complete (`success` or `empty`, including both empty). Pagination or source failures leave the JWT unused so the same day can be retried. The result reports the day it used as `utcDay`. After a retrieve, the result includes `digest` from `buildAlertSearchDigest` (eight bucket records and counts: the five endorser lists plus `profilesNearby`, `profilesSimilar` and `plansSimilar`), and `startCursors`, the cursors the run counted from (absent on a first run). `digest` is `null` when there is no batch or no unused JWT for today. Consumption does not depend on `digest.hasUpdates`.
`startAlertSearchScheduler()` (started from `src/index.ts` next to the FCM scheduler) is a **separate** user-level job. One query per pass asks for the users who hold an unused JWT for the current UTC day, flagged by whether their `notify_hour_min_utc` has arrived ([Scheduler selection](#scheduler-selection)); it calls `runDailyAlertSearch` on the due ones, up to `ALERT_SEARCH_USER_CONCURRENCY` at a time. After each run, if the digest is complete with updates **and** today's JWT was consumed, it sends a user-visible FCM message (`title: TimeSafari`, body `You have N new updates.`, data `type: alert_search`) to that user's registered tokens. It does not call `sendPushToDevice` or change `WAKEUP_PING`. Subsequent ticks the same UTC day see no unused JWT (`digest: null`) and do not resend. FCM send failures are logged and do not roll back cursors or JWT consumption. A process-local in-flight flag skips a tick if a pass is still running.
`startAlertSearchScheduler()` (started from `src/index.ts` next to the FCM scheduler) is a **separate** user-level job. One query per pass asks for the users who hold an unused JWT for the current UTC day, flagged by whether their `notify_hour_min_utc` has arrived ([Scheduler selection](#scheduler-selection)); it calls `runDailyAlertSearch` on the due ones, up to `ALERT_SEARCH_USER_CONCURRENCY` at a time. After each run, if the digest is complete with updates **and** today's JWT was consumed, it sends a user-visible FCM message (`title: TimeSafari`, body `You have N new updates.`, data `type: alert_search` and `c`, the start cursors joined by `.`) to that user's registered tokens. It does not call `sendPushToDevice` or change `WAKEUP_PING`. Subsequent ticks the same UTC day see no unused JWT (`digest: null`) and do not resend. FCM send failures are logged and do not roll back cursors or JWT consumption. A process-local in-flight flag skips a tick if a pass is still running.
`buildAlertSearchDigest` maps a retrieve result into structured payload data: per-bucket record arrays, counts, `totalCount`, `hasUpdates`, and Endorser/Partner completion status. It does not invent a notification string; FCM uses only `totalCount` for the short body. Incomplete outcomes (`pagination`, auth, network, etc.) yield `completed: false` and `hasUpdates: false`. Empty successful retrieves are complete with `hasUpdates: false` and do not send FCM.
@@ -405,9 +420,10 @@ second replica double-texts.
Eligibility is the same predicate the FCM path uses: the run consumed today's
JWT and the digest is complete with updates. Consumption is what makes later
ticks on the same UTC day no-ops. The message is
`Gift Economies: you have N new updates. https://giftopia.me/deep-link/new-activity Reply STOP to end.`,
`Gift Economies: you have N new updates. https://giftopia.me/deep-link/new-activity?c=<start cursors> Reply STOP to end.`,
kept inside one 160-character GSM-7 segment, since a second segment is a second
charge. Underneath the JWT rule, an `sms_phone_log` count caps sends at one per
charge. `c` is the endorser and partner cursors the run counted from, joined by
`.`; the app compares it with its own to tell whether it shows the same window. Underneath the JWT rule, an `sms_phone_log` count caps sends at one per
handset per identity per UTC calendar day. Send failures are logged and do not roll back
cursor advancement or JWT consumption.
+18 -2
View File
@@ -55,7 +55,7 @@ export function emptyEndorserData(): EndorserAlertSearchData {
}
export function emptyPartnerData(): PartnerAlertSearchData {
return { profilesNearby: [] };
return { profilesNearby: [], profilesSimilar: [], plansSimilar: [] };
}
/** The `error` text of an error response body, when it has one. */
@@ -250,10 +250,26 @@ export function parsePartnerAlertSearchResponse(
if (!isRecord(body) || !isRecord(body.data)) return undefined;
const profilesNearby = asObjectArray(body.data.profilesNearby);
if (profilesNearby === undefined) return undefined;
// optional: servers before the similarity matches send neither list
const profilesSimilar =
body.data.profilesSimilar === undefined
? []
: asObjectArray(body.data.profilesSimilar);
const plansSimilar =
body.data.plansSimilar === undefined
? []
: asObjectArray(body.data.plansSimilar);
if (profilesSimilar === undefined || plansSimilar === undefined) {
return undefined;
}
const parsed: PartnerAlertSearchResponse = {
data: {
profilesNearby:
profilesNearby as PartnerAlertSearchResponse["data"]["profilesNearby"],
profilesSimilar:
profilesSimilar as PartnerAlertSearchResponse["data"]["profilesSimilar"],
plansSimilar:
plansSimilar as PartnerAlertSearchResponse["data"]["plansSimilar"],
},
};
copyCursorFields(body, parsed);
@@ -304,7 +320,7 @@ export async function fetchPartnerAlertSearchPage(input: {
return {
ok: false,
reason: "malformed",
message: "Partner alertSearch response is missing profilesNearby.",
message: "Partner alertSearch response is missing profilesNearby or has malformed similar lists.",
status: raw.status,
url,
};
+40 -11
View File
@@ -7,7 +7,7 @@ import type {
AlertSearchSourceResult,
RetrieveAlertSearchResult,
} from "./retrieve.js";
import type { AlertSearchChannel } from "./types.js";
import type { AlertSearchChannel, AlertSearchLocationBBox } from "./types.js";
/**
* `endorserAfterCursor` / `partnerAfterCursor` are the servers' opaque cursors;
@@ -25,11 +25,24 @@ export type CursorAdvanceResult = {
partnerAfterCursor: string | null;
};
/** The search inputs an upload stored for a channel. Omits missing values. */
export type StoredAlertSearchInputs = {
planHandleIds?: string[];
location?: AlertSearchLocationBBox;
};
const CURSOR_STORES: Record<AlertSearchChannel, AlertSearchCursorsStore> = {
fcm: alertSearchCursorsDb,
sms: smsAlertSearchCursorsDb,
};
/** The cursor and search-input store for a channel. */
export function alertSearchCursorsStore(
channel: AlertSearchChannel
): AlertSearchCursorsStore {
return CURSOR_STORES[channel];
}
/**
* The cursor to store after a run, or undefined when it stays. It also moves
* on an empty run: it covers everything the server had, so holding it back
@@ -59,10 +72,28 @@ export async function loadAlertSearchCursors(
return out;
}
/** Load the starred handle IDs and area stored for a channel. */
export async function loadAlertSearchInputs(
userId: string,
channel: AlertSearchChannel = "fcm"
): Promise<StoredAlertSearchInputs> {
const row = await CURSOR_STORES[channel].get(userId);
const out: StoredAlertSearchInputs = {};
if (row?.planHandleIds && row.planHandleIds.length > 0) {
out.planHandleIds = row.planHandleIds;
}
if (row?.location) {
out.location = row.location;
}
return out;
}
/**
* Advance each cursor independently after retrieveAlertSearch, to the
* server's nextAfterCursor after a success or empty run. Does not persist on
* pagination or error outcomes.
* Advance both cursors after retrieveAlertSearch, to each server's
* nextAfterCursor, only when both sources had a success or empty run. A
* failure on either side leaves both in place: the run sends nothing and does
* not consume its JWT, so the retry must search the whole window again, and a
* cursor that had moved would skip the rows it covered.
*/
export async function advanceAlertSearchCursors(
userId: string,
@@ -72,18 +103,16 @@ export async function advanceAlertSearchCursors(
const store = CURSOR_STORES[channel];
const nextEndorser = nextCursorFromResult(result.endorser);
if (nextEndorser !== undefined) {
await store.setEndorserAfterCursor(userId, nextEndorser);
}
const nextPartner = nextCursorFromResult(result.partner);
if (nextPartner !== undefined) {
await store.setPartnerAfterCursor(userId, nextPartner);
const advance = nextEndorser !== undefined && nextPartner !== undefined;
if (advance) {
await store.setAfterCursors(userId, nextEndorser, nextPartner);
}
const stored = await store.get(userId);
return {
endorserAdvanced: nextEndorser !== undefined,
partnerAdvanced: nextPartner !== undefined,
endorserAdvanced: advance,
partnerAdvanced: advance,
endorserAfterCursor: stored?.endorserAfterCursor ?? null,
partnerAfterCursor: stored?.partnerAfterCursor ?? null,
};
+7 -2
View File
@@ -1,6 +1,7 @@
import {
advanceAlertSearchCursors,
loadAlertSearchCursors,
loadAlertSearchInputs,
type CursorAdvanceResult,
} from "./cursors.js";
import type { AlertSearchChannel } from "./types.js";
@@ -22,8 +23,10 @@ export type AlertSearchCycleResult = {
};
/**
* Load stored cursors, retrieve Endorser+Partner alertSearch, then advance
* only complete source cursors. Not invoked by FCM, register, or startup.
* Load stored cursors and search inputs, retrieve Endorser+Partner
* alertSearch, then advance the cursors when both sources completed. A
* location or planHandleIds in the input replaces the stored one. Not invoked
* by FCM, register, or startup.
*/
export async function runAlertSearchCycle(
userId: string,
@@ -31,7 +34,9 @@ export async function runAlertSearchCycle(
channel: AlertSearchChannel = "fcm"
): Promise<AlertSearchCycleResult> {
const loaded = await loadAlertSearchCursors(userId, channel);
const inputs = await loadAlertSearchInputs(userId, channel);
const retrieved = await retrieveAlertSearch({
...inputs,
...input,
endorserAfterCursor: loaded.endorserAfterCursor,
partnerAfterCursor: loaded.partnerAfterCursor,
+26
View File
@@ -39,6 +39,11 @@ export type DailyAlertSearchResult = {
consumed: boolean;
/** Phase 6A digest after a retrieve; null when no batch/JWT was used. */
digest: AlertSearchDigest | null;
/**
* The cursors this run counted from, for the notification link; absent when
* no run happened or a source had never run before.
*/
startCursors?: { endorser: string; partner: string };
};
function noJwtResult(
@@ -59,6 +64,18 @@ function noJwtResult(
};
}
/**
* The `c` query value for a notification link: the start cursors joined by
* ".", which appears in neither server's cursor format.
*/
export function alertSearchLinkCursorParam(
startCursors: DailyAlertSearchResult["startCursors"]
): string | undefined {
return startCursors === undefined
? undefined
: `${startCursors.endorser}.${startCursors.partner}`;
}
/**
* Select today's unused delegated JWT by UTC day, run the existing alertSearch
* cycle, and consume that JWT only when both required sources completed
@@ -112,5 +129,14 @@ export async function runDailyAlertSearch(
completed,
consumed,
digest,
...(cycle.loaded.endorserAfterCursor !== undefined &&
cycle.loaded.partnerAfterCursor !== undefined
? {
startCursors: {
endorser: cycle.loaded.endorserAfterCursor,
partner: cycle.loaded.partnerAfterCursor,
},
}
: {}),
};
}
+10 -4
View File
@@ -24,6 +24,8 @@ export const ALERT_SEARCH_DIGEST_BUCKETS = [
"trackedPlanClaims",
"plansNearby",
"profilesNearby",
"profilesSimilar",
"plansSimilar",
] as const;
export type AlertSearchDigestBucket =
@@ -66,10 +68,12 @@ function copyEndorserRecords(
};
}
function copyPartnerRecords(
data: PartnerAlertSearchData
): Pick<CombinedAlertSearchData, "profilesNearby"> {
return { profilesNearby: [...data.profilesNearby] };
function copyPartnerRecords(data: PartnerAlertSearchData): PartnerAlertSearchData {
return {
profilesNearby: [...data.profilesNearby],
profilesSimilar: [...data.profilesSimilar],
plansSimilar: [...data.plansSimilar],
};
}
function sourceStatus(
@@ -95,6 +99,8 @@ export function countAlertSearchDigestRecords(
trackedPlanClaims: records.trackedPlanClaims.length,
plansNearby: records.plansNearby.length,
profilesNearby: records.profilesNearby.length,
profilesSimilar: records.profilesSimilar.length,
plansSimilar: records.plansSimilar.length,
};
}
+2
View File
@@ -30,7 +30,9 @@ export { retrieveAlertSearch } from "./retrieve.js";
export type { RetrieveAlertSearchInput, RetrieveAlertSearchResult } from "./retrieve.js";
export {
advanceAlertSearchCursors,
alertSearchCursorsStore,
loadAlertSearchCursors,
loadAlertSearchInputs,
nextCursorFromResult,
} from "./cursors.js";
export type { CursorAdvanceResult, StoredAlertSearchCursors } from "./cursors.js";
+14 -3
View File
@@ -1,6 +1,9 @@
import { db } from "../db/fcmTokensSqlite.js";
import { errorMessage } from "../util/formatElapsed.js";
import type { DailyAlertSearchResult } from "./daily.js";
import {
alertSearchLinkCursorParam,
type DailyAlertSearchResult,
} from "./daily.js";
import { log } from "../util/log.js";
export const ALERT_SEARCH_NOTIFICATION_TITLE = "TimeSafari";
@@ -10,6 +13,8 @@ export type AlertSearchNotificationContent = {
title: string;
body: string;
type: typeof ALERT_SEARCH_FCM_TYPE;
/** The run's start cursors, as the `c` query value of the app link. */
cursorParam?: string;
};
export type AlertSearchTokenSender = (
@@ -34,12 +39,14 @@ export function alertSearchNotificationBody(totalCount: number): string {
}
export function buildAlertSearchNotificationContent(
totalCount: number
totalCount: number,
cursorParam?: string
): AlertSearchNotificationContent {
return {
title: ALERT_SEARCH_NOTIFICATION_TITLE,
body: alertSearchNotificationBody(totalCount),
type: ALERT_SEARCH_FCM_TYPE,
...(cursorParam !== undefined ? { cursorParam } : {}),
};
}
@@ -82,6 +89,7 @@ async function defaultSend(
return sendAlertSearchPushToDevice(fcmToken, {
title: content.title,
body: content.body,
cursorParam: content.cursorParam,
});
}
@@ -98,7 +106,10 @@ export async function deliverAlertSearchNotification(
return { eligible: false, sent: 0, failed: 0 };
}
const content = buildAlertSearchNotificationContent(digest.totalCount);
const content = buildAlertSearchNotificationContent(
digest.totalCount,
alertSearchLinkCursorParam(result.startCursors)
);
const listTokens = deps.listTokens ?? defaultListTokens;
const send = deps.send ?? defaultSend;
const tokens = await listTokens(result.userId);
+53 -4
View File
@@ -35,6 +35,8 @@ export type AlertSearchSourceResult<TData> = {
data: TData;
/** Cursor to store: the last page's nextAfterCursor. Absent on failure. */
nextAfterCursor?: AlertSearchCursor;
/** A first run: the rows were discarded and only the cursor is kept. */
seeded?: boolean;
};
export type RetrieveAlertSearchInput = {
@@ -119,9 +121,41 @@ function mergePartner(
"issuerDid",
"updatedAt",
]),
profilesSimilar: mergeNested(acc.profilesSimilar, page.profilesSimilar, (m) =>
m.profile === null || typeof m.profile !== "object"
? undefined
: recordKey(m.profile as Record<string, unknown>, ["rowId", "issuerDid"])
),
plansSimilar: mergeNested(acc.plansSimilar, page.plansSimilar, (m) =>
m.plan === null || typeof m.plan !== "object"
? undefined
: recordKey(m.plan as Record<string, unknown>, ["handleId"])
),
};
}
/**
* Merge similarity matches, keyed by the record inside each. A later page can
* repeat a subject whose vector changed again inside the window.
*/
function mergeNested<T>(
existing: T[],
incoming: T[],
keyOf: (match: T) => string | undefined
): T[] {
const seen = new Set<string>();
const out: T[] = [];
for (const match of [...existing, ...incoming]) {
const key = keyOf(match);
if (key !== undefined) {
if (seen.has(key)) continue;
seen.add(key);
}
out.push(match);
}
return out;
}
function endorserEmpty(data: EndorserAlertSearchData): boolean {
return (
data.claims.length === 0 &&
@@ -133,7 +167,11 @@ function endorserEmpty(data: EndorserAlertSearchData): boolean {
}
function partnerEmpty(data: PartnerAlertSearchData): boolean {
return data.profilesNearby.length === 0;
return (
data.profilesNearby.length === 0 &&
data.profilesSimilar.length === 0 &&
data.plansSimilar.length === 0
);
}
type PageResponse<TData> = {
@@ -149,8 +187,9 @@ type PageResponse<TData> = {
* usable cursor; the rest comes on the next run.
*
* Without afterCursor (a first run), the server returns its most recent rows
* and the cursor at which they were read; that one page is the whole run,
* since older rows are not news.
* and the cursor at which they were read. That one page is the whole run, and
* its rows are dropped: they happened before notifications were turned on, so
* a first run only seeds the cursor.
*/
async function retrieveCursorPages<TData>(input: {
afterCursor?: AlertSearchCursor;
@@ -209,7 +248,17 @@ async function retrieveCursorPages<TData>(input: {
data,
};
}
if (result.body.hasMore !== true || afterCursor === undefined) {
if (afterCursor === undefined) {
return {
outcome: "empty",
userMessage,
pageCount,
data: input.empty,
nextAfterCursor,
seeded: true,
};
}
if (result.body.hasMore !== true) {
return {
outcome: input.isEmpty(data) ? "empty" : "success",
userMessage,
+21 -4
View File
@@ -13,7 +13,10 @@ import { smsBlockedNumbersDb } from "../db/smsBlockedNumbersSqlite.js";
import { errorMessage } from "../util/formatElapsed.js";
import { maskPhoneNumber } from "../util/smsPhoneNumber.js";
import { hashPhoneNumber } from "../util/smsVerificationCode.js";
import type { DailyAlertSearchResult } from "./daily.js";
import {
alertSearchLinkCursorParam,
type DailyAlertSearchResult,
} from "./daily.js";
import { log } from "../util/log.js";
/**
@@ -48,11 +51,22 @@ export type AlertSearchSmsNotifyResult = {
blocked: number;
};
export function alertSearchSmsBody(totalCount: number): string {
/**
* @param cursorParam the run's start cursors (`alertSearchLinkCursorParam`),
* so the app can tell whether it shows the window this text counted
*/
export function alertSearchSmsBody(
totalCount: number,
cursorParam?: string
): string {
const noun = totalCount === 1 ? "update" : "updates";
const link =
cursorParam === undefined
? ALERT_SEARCH_SMS_LINK
: `${ALERT_SEARCH_SMS_LINK}?c=${encodeURIComponent(cursorParam)}`;
return (
`Gift Economies: you have ${totalCount} new ${noun}. ` +
`${ALERT_SEARCH_SMS_LINK} Reply STOP to end.`
`${link} Reply STOP to end.`
);
}
@@ -133,7 +147,10 @@ export async function deliverAlertSearchSms(
return { eligible: false, sent: 0, failed: 0, blocked: 0 };
}
const body = alertSearchSmsBody(digest.totalCount);
const body = alertSearchSmsBody(
digest.totalCount,
alertSearchLinkCursorParam(result.startCursors)
);
const listPhones = deps.listPhones ?? defaultListPhones;
const send = deps.send ?? sendSms;
const now = deps.now ?? new Date();
+19
View File
@@ -86,8 +86,27 @@ export type EndorserAlertSearchData = {
plansNearby: AlertSearchPlanRecord[];
};
/**
* A profile or plan close in meaning to the requester's profile, from partner
* `profilesSimilar` / `plansSimilar` (best first, at most 50, not paged).
*/
export type AlertSearchSimilarProfile = {
similarity: number;
profile: AlertSearchProfileRecord;
[key: string]: unknown;
};
export type AlertSearchSimilarPlan = {
similarity: number;
plan: AlertSearchPlanRecord;
[key: string]: unknown;
};
/** The similar lists are empty from partner servers that predate them. */
export type PartnerAlertSearchData = {
profilesNearby: AlertSearchProfileRecord[];
profilesSimilar: AlertSearchSimilarProfile[];
plansSimilar: AlertSearchSimilarPlan[];
};
export type CombinedAlertSearchData = EndorserAlertSearchData &
+20 -2
View File
@@ -163,7 +163,9 @@ export function createAlertAuthorizationStore(tables: AlertAuthorizationTables)
return {
/**
* Atomically drop this user's unused JWTs (and empty batch rows), then install
* a new batch. Consumed JWTs from prior batches are left in place.
* a new batch. Consumed JWTs from prior batches are left in place, and the
* new batch's JWTs for those days are skipped: a day that already ran must
* not run (and notify) again because the app uploaded after it.
*/
async replaceUnusedBatch(input: {
userId: string;
@@ -181,6 +183,7 @@ export function createAlertAuthorizationStore(tables: AlertAuthorizationTables)
const now = new Date().toISOString();
const batchPk = randomUUID();
let insertedCount = 0;
const run = connection.transaction(() => {
connection
.prepare(
@@ -191,6 +194,19 @@ export function createAlertAuthorizationStore(tables: AlertAuthorizationTables)
)
.run(input.userId, ALERT_JWT_STATUS_UNUSED);
const consumedDays = new Set(
(
connection
.prepare(
`
SELECT DISTINCT day FROM ${tables.jwts}
WHERE user_id = ? AND status = ?
`
)
.all(input.userId, ALERT_JWT_STATUS_CONSUMED) as { day: string }[]
).map((row) => row.day)
);
connection
.prepare(
`
@@ -234,6 +250,8 @@ export function createAlertAuthorizationStore(tables: AlertAuthorizationTables)
);
for (const item of input.jwts) {
if (consumedDays.has(item.day)) continue;
insertedCount += 1;
insertJwt.run({
id: randomUUID(),
batch_pk: batchPk,
@@ -264,7 +282,7 @@ export function createAlertAuthorizationStore(tables: AlertAuthorizationTables)
timezone: input.timezone,
createdAt: now,
},
storedCount: input.jwts.length,
storedCount: insertedCount,
unusedCount,
};
},
+73 -7
View File
@@ -1,3 +1,4 @@
import type { AlertSearchLocationBBox } from "../alertSearch/types.js";
import { getDatabase } from "./sqlite.js";
export type AlertSearchCursorRecord = {
@@ -6,25 +7,52 @@ export type AlertSearchCursorRecord = {
endorserAfterCursor: string | null;
/** Opaque Partner cursor (alertSearch nextAfterCursor). */
partnerAfterCursor: string | null;
/** Starred plan handle IDs to track; null when none were uploaded. */
planHandleIds: string[] | null;
/** Area for the nearby lists; null when none was uploaded. */
location: AlertSearchLocationBBox | null;
createdAt: string;
updatedAt: string;
};
/**
* What an upload sets. A field left undefined keeps its stored value; null
* clears planHandleIds or location.
*/
export type AlertSearchStateUpdate = {
endorserAfterCursor?: string;
partnerAfterCursor?: string;
planHandleIds?: string[] | null;
location?: AlertSearchLocationBBox | null;
};
type CursorDbRow = {
user_id: string;
endorser_after_cursor: string | null;
partner_after_cursor: string | null;
plan_handle_ids: string | null;
location: string | null;
created_at: string;
updated_at: string;
};
type CursorColumn = "endorser_after_cursor" | "partner_after_cursor";
/** A stored JSON column, or null when it is empty or unreadable. */
function parseJsonColumn<T>(value: string | null): T | null {
if (value === null) return null;
try {
return JSON.parse(value) as T;
} catch {
return null;
}
}
function toRecord(row: CursorDbRow): AlertSearchCursorRecord {
return {
userId: row.user_id,
endorserAfterCursor: row.endorser_after_cursor,
partnerAfterCursor: row.partner_after_cursor,
planHandleIds: parseJsonColumn<string[]>(row.plan_handle_ids),
location: parseJsonColumn<AlertSearchLocationBBox>(row.location),
createdAt: row.created_at,
updatedAt: row.updated_at,
};
@@ -55,7 +83,10 @@ function ensureRow(table: string, userId: string, now: string): void {
* delta and leave the other reporting nothing.
*/
export function createAlertSearchCursorsStore(table: string) {
function setColumn(column: CursorColumn, userId: string, value: string): void {
/** Column name to value; names come only from this module. */
function setColumns(userId: string, values: Record<string, string | null>): void {
const names = Object.keys(values);
if (names.length === 0) return;
const now = new Date().toISOString();
const connection = getDatabase();
const run = connection.transaction(() => {
@@ -64,11 +95,11 @@ export function createAlertSearchCursorsStore(table: string) {
.prepare(
`
UPDATE ${table}
SET ${column} = ?, updated_at = ?
SET ${names.map((name) => `${name} = ?`).join(", ")}, updated_at = ?
WHERE user_id = ?
`
)
.run(value, now, userId);
.run(...names.map((name) => values[name]), now, userId);
});
run();
}
@@ -81,7 +112,7 @@ export function createAlertSearchCursorsStore(table: string) {
.prepare(
`
SELECT user_id, endorser_after_cursor, partner_after_cursor,
created_at, updated_at
plan_handle_ids, location, created_at, updated_at
FROM ${table}
WHERE user_id = ?
`
@@ -91,11 +122,46 @@ export function createAlertSearchCursorsStore(table: string) {
},
async setEndorserAfterCursor(userId: string, after: string): Promise<void> {
setColumn("endorser_after_cursor", userId, after);
setColumns(userId, { endorser_after_cursor: after });
},
async setPartnerAfterCursor(userId: string, after: string): Promise<void> {
setColumn("partner_after_cursor", userId, after);
setColumns(userId, { partner_after_cursor: after });
},
/** Both cursors in one write, so neither moves without the other. */
async setAfterCursors(
userId: string,
endorserAfter: string,
partnerAfter: string
): Promise<void> {
setColumns(userId, {
endorser_after_cursor: endorserAfter,
partner_after_cursor: partnerAfter,
});
},
/** Apply what an alert-authorization upload sent, in one write. */
async setSearchState(
userId: string,
update: AlertSearchStateUpdate
): Promise<void> {
const values: Record<string, string | null> = {};
if (update.endorserAfterCursor !== undefined) {
values.endorser_after_cursor = update.endorserAfterCursor;
}
if (update.partnerAfterCursor !== undefined) {
values.partner_after_cursor = update.partnerAfterCursor;
}
if (update.planHandleIds !== undefined) {
values.plan_handle_ids =
update.planHandleIds === null ? null : JSON.stringify(update.planHandleIds);
}
if (update.location !== undefined) {
values.location =
update.location === null ? null : JSON.stringify(update.location);
}
setColumns(userId, values);
},
};
}
+27
View File
@@ -102,6 +102,14 @@ CREATE TABLE IF NOT EXISTS alert_search_cursors (
user_id TEXT PRIMARY KEY NOT NULL,
endorser_after_cursor TEXT,
partner_after_cursor TEXT,
-- JSON array of the starred plan handle IDs to track, or NULL for none.
plan_handle_ids TEXT,
-- JSON {minLocLat, maxLocLat, minLocLon, maxLocLon} for the nearby lists,
-- or NULL for no area.
location TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -259,6 +267,8 @@ CREATE TABLE IF NOT EXISTS sms_alert_search_cursors (
user_id TEXT PRIMARY KEY NOT NULL,
endorser_after_cursor TEXT,
partner_after_cursor TEXT,
plan_handle_ids TEXT,
location TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@@ -285,9 +295,26 @@ function dropIdPathCursorTables(connection: Database.Database): void {
}
}
/** Search-input columns added to the cursor tables after 0.2.5. */
const CURSOR_TABLE_ADDED_COLUMNS = ["plan_handle_ids", "location"];
function addCursorSearchInputColumns(connection: Database.Database): void {
for (const table of CURSOR_TABLES) {
const columns = connection
.prepare(`PRAGMA table_info(${table})`)
.all() as Array<{ name: string }>;
for (const column of CURSOR_TABLE_ADDED_COLUMNS) {
if (!columns.some((c) => c.name === column)) {
connection.exec(`ALTER TABLE ${table} ADD COLUMN ${column} TEXT`);
}
}
}
}
function ensureSchema(connection: Database.Database): void {
dropIdPathCursorTables(connection);
connection.exec(SCHEMA_SQL);
addCursorSearchInputColumns(connection);
}
/**
+2
View File
@@ -1,5 +1,6 @@
import express, { Router } from "express";
import { alertAuthorizationDb } from "../db/alertAuthorizationSqlite.js";
import { alertSearchCursorsDb } from "../db/alertSearchCursorsSqlite.js";
import { db } from "../db/fcmTokensSqlite.js";
import {
requireAuth,
@@ -80,6 +81,7 @@ notificationsRouter.post(
timezone: validated.timezone,
jwts: validated.jwts,
});
await alertSearchCursorsDb.setSearchState(userId, validated.searchState);
res.status(200).json({
success: true,
batchId: stored.batch.batchId,
+5
View File
@@ -1,5 +1,6 @@
import express, { Router, type RequestHandler } from "express";
import { smsAlertAuthorizationDb } from "../db/smsAlertAuthorizationSqlite.js";
import { smsAlertSearchCursorsDb } from "../db/alertSearchCursorsSqlite.js";
import { smsPhoneLogDb } from "../db/smsPhoneLogSqlite.js";
import { smsRegistrationsDb } from "../db/smsRegistrationsSqlite.js";
import { smsConfig } from "../env.js";
@@ -716,6 +717,10 @@ export function createNotifySmsRouter(deps: NotifySmsDeps = {}): express.Router
timezone: validated.timezone,
jwts: validated.jwts,
});
await smsAlertSearchCursorsDb.setSearchState(
userId,
validated.searchState
);
await recordPhoneAction({
userId,
phoneE164: verified[0].phoneE164,
+102
View File
@@ -1,4 +1,5 @@
import { EXPECTED_ALERT_JWT_BATCH_SIZE } from "../db/alertAuthorizationSqlite.js";
import type { AlertSearchStateUpdate } from "../db/alertSearchCursorsSqlite.js";
import {
DELEGATED_JWT_UNSUPPORTED_IDENTITY_CODE,
decodeAndVerifyDelegatedJwt,
@@ -16,8 +17,26 @@ export type AlertAuthorizationRequestBody = {
notifyMinuteUtc?: unknown;
timezone?: unknown;
jwts?: unknown;
/**
* Optional alertSearch state. Cursors replace the channel's stored cursors,
* so the next run counts from them; planHandleIds and location replace the
* stored search inputs, and null clears them. Absent keeps what is stored.
*/
endorserAfterCursor?: unknown;
partnerAfterCursor?: unknown;
planHandleIds?: unknown;
location?: unknown;
};
/**
* Cursors are opaque, but they travel in a URL query and are joined with "."
* in notification links, so only these characters are accepted.
*/
const CURSOR_RE = /^[0-9A-Za-z_:-]{1,64}$/;
/** The daily GET sends these as JSON in the query string. */
const MAX_PLAN_HANDLE_IDS = 200;
const MAX_PLAN_HANDLE_ID_LENGTH = 200;
export type ValidatedAlertJwt = {
sequence: number;
day: string;
@@ -39,6 +58,8 @@ export type BatchValidationSuccess = {
*/
timezone: string | undefined;
jwts: ValidatedAlertJwt[];
/** Apply with the channel's cursor store after the batch is stored. */
searchState: AlertSearchStateUpdate;
};
export type BatchValidationFailure = {
@@ -222,6 +243,8 @@ export async function validateAlertAuthorizationBatch(
}
}
const searchState = validateAlertSearchState(body, details);
if (!Array.isArray(body.jwts)) {
details.push("jwts must be an array of 100 delegated JWT entries");
return fail(details);
@@ -384,9 +407,88 @@ export async function validateAlertAuthorizationBatch(
notifyMinuteUtc,
timezone,
jwts: validated,
searchState,
};
}
function isLatitude(value: unknown): boolean {
return typeof value === "number" && Number.isFinite(value) && value >= -90 && value <= 90;
}
function isLongitude(value: unknown): boolean {
return typeof value === "number" && Number.isFinite(value) && value >= -180 && value <= 180;
}
/** The optional alertSearch fields of an upload; problems go into details. */
function validateAlertSearchState(
body: AlertAuthorizationRequestBody,
details: string[]
): AlertSearchStateUpdate {
const state: AlertSearchStateUpdate = {};
for (const field of ["endorserAfterCursor", "partnerAfterCursor"] as const) {
const value = body[field];
if (value === undefined || value === null) continue;
if (typeof value !== "string" || !CURSOR_RE.test(value)) {
details.push(
`${field} must be a cursor from an alertSearch response (letters, digits, '_', ':', '-'; at most 64)`
);
} else {
state[field] = value;
}
}
if (body.planHandleIds === null) {
state.planHandleIds = null;
} else if (body.planHandleIds !== undefined) {
const ids = body.planHandleIds;
if (
!Array.isArray(ids) ||
ids.length > MAX_PLAN_HANDLE_IDS ||
!ids.every(
(id) =>
typeof id === "string" &&
id.length > 0 &&
id.length <= MAX_PLAN_HANDLE_ID_LENGTH
)
) {
details.push(
`planHandleIds must be an array of at most ${MAX_PLAN_HANDLE_IDS} non-empty strings`
);
} else {
state.planHandleIds = ids as string[];
}
}
if (body.location === null) {
state.location = null;
} else if (body.location !== undefined) {
const box = body.location as Record<string, unknown>;
if (
typeof box !== "object" ||
Array.isArray(box) ||
!isLatitude(box.minLocLat) ||
!isLatitude(box.maxLocLat) ||
!isLongitude(box.minLocLon) ||
!isLongitude(box.maxLocLon) ||
(box.minLocLat as number) > (box.maxLocLat as number)
) {
details.push(
"location must be { minLocLat, maxLocLat, minLocLon, maxLocLon } with valid coordinates"
);
} else {
state.location = {
minLocLat: box.minLocLat as number,
maxLocLat: box.maxLocLat as number,
minLocLon: box.minLocLon as number,
maxLocLon: box.maxLocLon as number,
};
}
}
return state;
}
function fail(details: string[]): BatchValidationFailure {
const reported = details.slice(0, MAX_REPORTED_ERRORS);
if (details.length > MAX_REPORTED_ERRORS) {
+5 -1
View File
@@ -11,7 +11,7 @@ export const ALERT_SEARCH_FCM_TYPE = "alert_search";
*/
export async function sendAlertSearchPushToDevice(
fcmToken: string,
content: { title: string; body: string }
content: { title: string; body: string; cursorParam?: string }
): Promise<"sent" | "failed"> {
const suffix = maskToken(fcmToken);
const sendStarted = Date.now();
@@ -26,6 +26,10 @@ export async function sendAlertSearchPushToDevice(
},
data: {
type: ALERT_SEARCH_FCM_TYPE,
// the app opens /new-activity?c=<this> on a tap
...(content.cursorParam !== undefined
? { c: content.cursorParam }
: {}),
},
});
log.info(
+15 -11
View File
@@ -29,7 +29,7 @@ function emptyEndorser(): EndorserAlertSearchData {
}
function emptyPartner(): PartnerAlertSearchData {
return { profilesNearby: [] };
return { profilesNearby: [], profilesSimilar: [], plansSimilar: [] };
}
function endorserResult(
@@ -143,15 +143,16 @@ describe("alertSearch cursor persistence", () => {
assert.equal(advanced.partnerAfterCursor, "912-48213");
});
it("empty run → cursor advances", async () => {
it("empty run → cursors advance", async () => {
await alertSearchCursorsDb.setEndorserAfterCursor(USER, "10");
const advanced = await advanceAlertSearchCursors(
USER,
combined(endorserResult("empty", "25"), partnerResult("empty"))
combined(endorserResult("empty", "25"), partnerResult("empty", "2-25"))
);
assert.equal(advanced.endorserAdvanced, true);
assert.equal(advanced.endorserAfterCursor, "25");
assert.equal(advanced.partnerAdvanced, false);
assert.equal(advanced.partnerAdvanced, true);
assert.equal(advanced.partnerAfterCursor, "2-25");
});
it("pagination or error → cursor unchanged", async () => {
@@ -169,26 +170,29 @@ describe("alertSearch cursor persistence", () => {
}
});
it("Endorser success + Partner failure → only Endorser advances", async () => {
// the retry must search the same window, or the rows the completed source
// found would never be counted
it("Endorser success + Partner failure → neither advances", async () => {
await alertSearchCursorsDb.setEndorserAfterCursor(USER, "30");
const advanced = await advanceAlertSearchCursors(
USER,
combined(endorserResult("success", "40"), partnerResult("auth"))
);
assert.equal(advanced.endorserAdvanced, true);
assert.equal(advanced.endorserAfterCursor, "40");
assert.equal(advanced.endorserAdvanced, false);
assert.equal(advanced.endorserAfterCursor, "30");
assert.equal(advanced.partnerAdvanced, false);
assert.equal(advanced.partnerAfterCursor, null);
});
it("Endorser failure + Partner success → only Partner advances", async () => {
it("Endorser failure + Partner success → neither advances", async () => {
const advanced = await advanceAlertSearchCursors(
USER,
combined(endorserResult("timeout"), partnerResult("success", "5-41"))
);
assert.equal(advanced.endorserAdvanced, false);
assert.equal(advanced.endorserAfterCursor, null);
assert.equal(advanced.partnerAdvanced, true);
assert.equal(advanced.partnerAfterCursor, "5-41");
assert.equal(advanced.partnerAdvanced, false);
assert.equal(advanced.partnerAfterCursor, null);
});
it("survives close and reopen of the SQLite connection", async () => {
@@ -253,7 +257,7 @@ describe("alertSearch cursor persistence", () => {
).map((c) => c.name);
assert.deepEqual(
columns,
["user_id", "endorser_after_cursor", "partner_after_cursor", "created_at", "updated_at"],
["user_id", "endorser_after_cursor", "partner_after_cursor", "plan_handle_ids", "location", "created_at", "updated_at"],
table
);
}
+13 -10
View File
@@ -105,7 +105,7 @@ describe("alertSearch cycle integration", () => {
await rm(dir, { recursive: true, force: true });
});
it("first run sends no cursor and stores the server's", async () => {
it("first run sends no cursor, drops the rows, and stores the server's cursor", async () => {
const cap = captureFetch((url, params) => {
assert.equal(params.has("afterCursor"), false);
if (isPartner(url)) return jsonResponse(partnerBody("3-15", ONE_PROFILE));
@@ -116,8 +116,11 @@ describe("alertSearch cycle integration", () => {
config: { fetch: cap.fetch },
});
assert.deepEqual(cycle.loaded, {});
assert.equal(cycle.retrieved.endorser.outcome, "success");
assert.equal(cycle.retrieved.partner.outcome, "success");
assert.equal(cycle.retrieved.endorser.outcome, "empty");
assert.equal(cycle.retrieved.endorser.seeded, true);
assert.equal(cycle.retrieved.partner.outcome, "empty");
assert.equal(cycle.retrieved.data.claims.length, 0);
assert.equal(cycle.retrieved.data.profilesNearby.length, 0);
assert.equal(cycle.advanced.endorserAfterCursor, "15");
assert.equal(cycle.advanced.partnerAfterCursor, "3-15");
});
@@ -176,8 +179,8 @@ describe("alertSearch cycle integration", () => {
assert.equal(cycle.advanced.endorserAfterCursor, null);
});
it("does not advance a source whose server timed out a list", async () => {
await alertSearchCursorsDb.setPartnerAfterCursor(USER, "1-100");
it("does not advance either source when one server timed out a list", async () => {
await alertSearchCursorsDb.setAfterCursors(USER, "100", "1-100");
const cap = captureFetch((url) => {
if (isPartner(url)) {
return jsonResponse({
@@ -194,10 +197,10 @@ describe("alertSearch cycle integration", () => {
assert.equal(cycle.retrieved.partner.outcome, "pagination");
assert.equal(cycle.advanced.partnerAdvanced, false);
assert.equal(cycle.advanced.partnerAfterCursor, "1-100");
assert.equal(cycle.advanced.endorserAfterCursor, "120");
assert.equal(cycle.advanced.endorserAfterCursor, "100");
});
it("Endorser success + Partner failure → Endorser only", async () => {
it("Endorser success + Partner failure → neither advances", async () => {
const cap = captureFetch((url) => {
if (isPartner(url)) return jsonResponse({ error: "Unauthorized" }, 401);
return jsonResponse(endorserBody("21", { claims: claimRows(21, 1) }));
@@ -207,12 +210,12 @@ describe("alertSearch cycle integration", () => {
config: { fetch: cap.fetch },
});
assert.equal(cycle.retrieved.partner.outcome, "auth");
assert.equal(cycle.advanced.endorserAfterCursor, "21");
assert.equal(cycle.advanced.endorserAfterCursor, null);
assert.equal(cycle.advanced.partnerAdvanced, false);
assert.equal(cycle.advanced.partnerAfterCursor, null);
});
it("Endorser failure + Partner success → Partner only", async () => {
it("Endorser failure + Partner success → neither advances", async () => {
const cap = captureFetch((url) => {
if (isPartner(url)) return jsonResponse(partnerBody("6-30", ONE_PROFILE));
return jsonResponse({ error: "fail" }, 500);
@@ -224,7 +227,7 @@ describe("alertSearch cycle integration", () => {
assert.equal(cycle.retrieved.endorser.outcome, "http");
assert.equal(cycle.advanced.endorserAdvanced, false);
assert.equal(cycle.advanced.endorserAfterCursor, null);
assert.equal(cycle.advanced.partnerAfterCursor, "6-30");
assert.equal(cycle.advanced.partnerAfterCursor, null);
});
it("both fail → neither advances", async () => {
+50 -4
View File
@@ -154,6 +154,8 @@ describe("runDailyAlertSearch", () => {
dir = await mkdtemp(path.join(tmpdir(), "alert-search-daily-"));
process.env.NOTIFY_DATA_DIR = dir;
closeDatabase();
// a run with no stored cursor only seeds it, so start from one
await alertSearchCursorsDb.setAfterCursors(USER, "1", "1-1");
});
afterEach(async () => {
@@ -263,6 +265,7 @@ describe("runDailyAlertSearch", () => {
assert.ok(result.digest);
assert.equal(result.digest.hasUpdates, true);
assert.equal(result.jwtSequence, 1);
assert.deepEqual(result.startCursors, { endorser: "1", partner: "1-1" });
});
it("consumes today's JWT when both sources succeed", async () => {
@@ -326,9 +329,10 @@ describe("runDailyAlertSearch", () => {
assert.equal(result.digest.records.claims[0].id, ulid(10));
const unused = await alertAuthorizationDb.getUnusedForDay(USER, DAY_UTC);
assert.equal(unused?.jwt, JWT_DAY_UTC);
// the retry searches the same window, so the claim is not lost
const stored = await alertSearchCursorsDb.get(USER);
assert.equal(stored?.endorserAfterCursor, ENDORSER_CURSOR);
assert.equal(stored?.partnerAfterCursor, null);
assert.equal(stored?.endorserAfterCursor, "1");
assert.equal(stored?.partnerAfterCursor, "1-1");
});
it("does not consume when Partner succeeds and Endorser fails", async () => {
@@ -362,8 +366,8 @@ describe("runDailyAlertSearch", () => {
const unused = await alertAuthorizationDb.getUnusedForDay(USER, DAY_UTC);
assert.equal(unused?.status, ALERT_JWT_STATUS_UNUSED);
const stored = await alertSearchCursorsDb.get(USER);
assert.equal(stored?.endorserAfterCursor, null);
assert.equal(stored?.partnerAfterCursor, PARTNER_CURSOR);
assert.equal(stored?.endorserAfterCursor, "1");
assert.equal(stored?.partnerAfterCursor, "1-1");
});
it("does not consume on Endorser pagination", async () => {
@@ -508,6 +512,48 @@ describe("runDailyAlertSearch", () => {
assert.equal(await alertAuthorizationDb.countUnused(USER), 0);
});
it("a first run seeds the cursors, counts nothing, and has no start cursors", async () => {
closeDatabase();
await rm(dir, { recursive: true, force: true });
dir = await mkdtemp(path.join(tmpdir(), "alert-search-daily-"));
process.env.NOTIFY_DATA_DIR = dir;
await seedBatch([jwtInput(1, DAY_UTC, JWT_DAY_UTC)]);
const result = await runDailyAlertSearch(
USER,
NOW_SPLIT,
cycleOpts(bothSuccessFetch().fetch)
);
assert.equal(result.completed, true);
assert.equal(result.consumed, true);
assert.equal(result.digest?.hasUpdates, false);
assert.equal(result.digest?.totalCount, 0);
assert.equal(result.startCursors, undefined);
const stored = await alertSearchCursorsDb.get(USER);
assert.equal(stored?.endorserAfterCursor, ENDORSER_CURSOR);
assert.equal(stored?.partnerAfterCursor, PARTNER_CURSOR);
});
it("sends the stored starred handle IDs and area", async () => {
await alertSearchCursorsDb.setSearchState(USER, {
planHandleIds: ["https://endorser.ch/entity/01HPLAN"],
location: { minLocLat: 40, maxLocLat: 41, minLocLon: -112, maxLocLon: -111 },
});
await seedBatch([jwtInput(1, DAY_UTC, JWT_DAY_UTC)]);
const cap = bothSuccessFetch();
await runDailyAlertSearch(USER, NOW_SPLIT, cycleOpts(cap.fetch));
const endorserUrl = new URL(
cap.urls.find((u) => u.includes("/api/v2/report/alertSearch"))!
);
assert.deepEqual(JSON.parse(endorserUrl.searchParams.get("planHandleIds")!), [
"https://endorser.ch/entity/01HPLAN",
]);
assert.equal(endorserUrl.searchParams.get("minLocLat"), "40");
const partnerUrl = new URL(
cap.urls.find((u) => u.includes("/api/partner/alertSearch"))!
);
assert.equal(partnerUrl.searchParams.get("maxLocLon"), "-111");
});
it("advances the cursors after an empty run and after a successful one", async () => {
await seedBatch([jwtInput(1, DAY_UTC, JWT_DAY_UTC)]);
await runDailyAlertSearch(
+9 -5
View File
@@ -86,10 +86,12 @@ function populatedEndorser(): EndorserAlertSearchData {
}
describe("buildAlertSearchDigest", () => {
it("preserves all six buckets and their records when populated", () => {
it("preserves all eight buckets and their records when populated", () => {
const endorserData = populatedEndorser();
const partnerData = {
profilesNearby: [profile(1), profile(2)],
profilesSimilar: [{ similarity: 0.9, profile: profile(3) }],
plansSimilar: [{ similarity: 0.8, plan: plan(50) }],
};
const digest = buildAlertSearchDigest(
retrieved(
@@ -105,7 +107,9 @@ describe("buildAlertSearchDigest", () => {
assert.equal(digest.counts.trackedPlanClaims, 2);
assert.equal(digest.counts.plansNearby, 1);
assert.equal(digest.counts.profilesNearby, 2);
assert.equal(digest.totalCount, 10);
assert.equal(digest.counts.profilesSimilar, 1);
assert.equal(digest.counts.plansSimilar, 1);
assert.equal(digest.totalCount, 12);
assert.deepEqual(digest.records.claims, endorserData.claims);
assert.deepEqual(
digest.records.personalPlanContributions,
@@ -146,7 +150,7 @@ describe("buildAlertSearchDigest", () => {
claims: [claim(1), claim(2), claim(3)],
trackedPlanUpdates: [plan(20)],
}),
partnerResult("success", { profilesNearby: [profile(1), profile(2)] })
partnerResult("success", { ...emptyPartnerData(), profilesNearby: [profile(1), profile(2)] })
)
);
assert.equal(digest.completed, true);
@@ -205,7 +209,7 @@ describe("buildAlertSearchDigest", () => {
reason: "network",
message: "fetch failed",
}),
partnerResult("success", { profilesNearby: [profile(1)] })
partnerResult("success", { ...emptyPartnerData(), profilesNearby: [profile(1)] })
)
);
assert.equal(digest.completed, false);
@@ -241,7 +245,7 @@ describe("buildAlertSearchDigest", () => {
const digest = buildAlertSearchDigest(
retrieved(
endorserResult("success", populatedEndorser()),
partnerResult("success", { profilesNearby: [profile(1), profile(2)] })
partnerResult("success", { ...emptyPartnerData(), profilesNearby: [profile(1), profile(2)] })
)
);
const sum = ALERT_SEARCH_DIGEST_BUCKETS.reduce(
+10
View File
@@ -32,6 +32,8 @@ function emptyDigest(
trackedPlanClaims: 0,
plansNearby: 0,
profilesNearby: 0,
profilesSimilar: 0,
plansSimilar: 0,
},
records: {
claims: [],
@@ -40,6 +42,8 @@ function emptyDigest(
trackedPlanClaims: [],
plansNearby: [],
profilesNearby: [],
profilesSimilar: [],
plansSimilar: [],
},
endorser: { outcome: "empty", completed: true },
partner: { outcome: "empty", completed: true },
@@ -59,6 +63,8 @@ function updatesDigest(totalCount: number): AlertSearchDigest {
trackedPlanClaims: 0,
plansNearby: 0,
profilesNearby: 0,
profilesSimilar: 0,
plansSimilar: 0,
},
records: {
claims: Array.from({ length: totalCount }, (_, i) => ({
@@ -71,6 +77,8 @@ function updatesDigest(totalCount: number): AlertSearchDigest {
trackedPlanClaims: [],
plansNearby: [],
profilesNearby: [],
profilesSimilar: [],
plansSimilar: [],
},
endorser: { outcome: "success", completed: true },
partner: { outcome: "empty", completed: true },
@@ -152,6 +160,8 @@ describe("alertSearch notification gate", () => {
assert.equal(content.body.includes(USER), false);
assert.equal(content.body.includes("01H"), false);
assert.equal(alertSearchNotificationBody(1), "You have 1 new update.");
assert.equal(content.cursorParam, undefined);
assert.equal(buildAlertSearchNotificationContent(7, "5.2-5").cursorParam, "5.2-5");
});
});
+13 -2
View File
@@ -202,7 +202,10 @@ describe("retrieveAlertSearch", () => {
for (const url of cap.urls) {
assert.equal(new URL(url).search, "");
}
assert.equal(result.endorser.outcome, "success");
// a first run only seeds the cursor: those rows predate notifications
assert.equal(result.endorser.outcome, "empty");
assert.equal(result.endorser.seeded, true);
assert.equal(result.data.claims.length, 0);
assert.equal(result.endorser.nextAfterCursor, "40");
assert.equal(result.partner.outcome, "empty");
assert.equal(result.partner.nextAfterCursor, "3-40");
@@ -247,7 +250,7 @@ describe("retrieveAlertSearch", () => {
assert.ok("profilesNearby" in result.data);
});
it("preserves all six result categories", async () => {
it("preserves all eight result categories", async () => {
const cap = captureFetch((url) => {
if (url.includes("/api/partner/")) {
return jsonResponse({
@@ -255,6 +258,10 @@ describe("retrieveAlertSearch", () => {
profilesNearby: [
{ issuerDid: "did:ethr:0x2", description: "near", extra: true },
],
profilesSimilar: [
{ similarity: 0.9, profile: { rowId: 7, issuerDid: "x", description: "d" } },
],
plansSimilar: [{ similarity: 0.8, plan: { handleId: "q", jwtId: ulid(6) } }],
}),
nextAfterCursor: "1-5",
});
@@ -276,12 +283,16 @@ describe("retrieveAlertSearch", () => {
});
const result = await retrieveAlertSearch({
jwt: JWT,
endorserAfterCursor: "1",
partnerAfterCursor: "1-1",
endorserBaseUrl: ENDORSER_BASE,
partnerBaseUrl: PARTNER_BASE,
config: { fetch: cap.fetch },
});
assert.equal(result.endorser.outcome, "success");
assert.equal(result.partner.outcome, "success");
assert.equal(result.data.profilesSimilar.length, 1);
assert.equal(result.data.plansSimilar[0]?.plan.handleId, "q");
assert.equal(result.data.claims[0]?.extra, 1);
assert.equal(result.data.profilesNearby[0]?.extra, true);
assert.equal(result.data.personalPlanContributions.length, 1);
+4
View File
@@ -46,6 +46,8 @@ function eligibleDaily(userId: string): DailyAlertSearchResult {
trackedPlanClaims: 0,
plansNearby: 0,
profilesNearby: 0,
profilesSimilar: 0,
plansSimilar: 0,
},
records: {
claims: [],
@@ -54,6 +56,8 @@ function eligibleDaily(userId: string): DailyAlertSearchResult {
trackedPlanClaims: [],
plansNearby: [],
profilesNearby: [],
profilesSimilar: [],
plansSimilar: [],
},
endorser: { outcome: "success", completed: true },
partner: { outcome: "empty", completed: true },
+1 -1
View File
@@ -51,7 +51,7 @@ function retrieved(endorserCursor: string, partnerCursor: string): RetrieveAlert
outcome: "success",
pageCount: 1,
nextAfterCursor: partnerCursor,
data: { profilesNearby: [{ rowId: "r1" } as never] },
data: { profilesNearby: [{ rowId: "r1" } as never], profilesSimilar: [], plansSimilar: [] },
};
return {
data: { ...endorser.data, ...partner.data },
+14
View File
@@ -64,6 +64,8 @@ function digest(overrides: Partial<AlertSearchDigest> = {}): AlertSearchDigest {
trackedPlanClaims: 0,
plansNearby: 0,
profilesNearby: 0,
profilesSimilar: 0,
plansSimilar: 0,
};
return {
completed: true,
@@ -77,6 +79,8 @@ function digest(overrides: Partial<AlertSearchDigest> = {}): AlertSearchDigest {
trackedPlanClaims: [],
plansNearby: [],
profilesNearby: [],
profilesSimilar: [],
plansSimilar: [],
},
endorser: { outcome: "success", completed: true },
partner: { outcome: "empty", completed: true },
@@ -131,6 +135,16 @@ describe("alertSearchSmsBody", () => {
assert.ok(body.includes("https://giftopia.me/deep-link/new-activity "));
assert.equal(body.includes("giftopia.tech"), false);
});
it("carries the start cursors as 'c' and still fits one segment", () => {
const cursorParam = "123456789.12345678-123456789";
const body = alertSearchSmsBody(9999, cursorParam);
assert.ok(
body.includes(`${ALERT_SEARCH_SMS_LINK}?c=${cursorParam} Reply STOP`),
body
);
assert.ok(body.length <= SMS_SINGLE_SEGMENT_LIMIT, `length ${body.length}`);
});
});
describe("SMS notification gate", () => {
+57 -1
View File
@@ -6,6 +6,7 @@ import path from "node:path";
import { afterEach, beforeEach, describe, it } from "node:test";
import express, { type RequestHandler } from "express";
import { smsAlertAuthorizationDb } from "../../src/db/smsAlertAuthorizationSqlite.js";
import { smsAlertSearchCursorsDb } from "../../src/db/alertSearchCursorsSqlite.js";
import { smsPhoneLogDb } from "../../src/db/smsPhoneLogSqlite.js";
import { smsBlockedNumbersDb } from "../../src/db/smsBlockedNumbersSqlite.js";
import { smsRegistrationsDb } from "../../src/db/smsRegistrationsSqlite.js";
@@ -755,11 +756,66 @@ describe("POST /notify-sms/alert-authorization", () => {
auth: { claim: authorizeClaim },
});
assert.equal(again.status, 200);
assert.equal(again.body.storedCount, 100);
// today already ran, so the new batch's JWT for today is skipped
assert.equal(again.body.storedCount, 99);
assert.equal(
(await smsAlertAuthorizationDb.getJwtById(first.id))?.status,
"consumed"
);
assert.equal(
await smsAlertAuthorizationDb.getUnusedForDay(USER, today),
undefined
);
});
it("stores the cursors, starred handle IDs and area sent with a batch", async () => {
await registerAndVerify();
const location = { minLocLat: 40, maxLocLat: 41, minLocLon: -112, maxLocLon: -111 };
const result = await call({
path: "/notify-sms/alert-authorization",
body: {
...batch(),
endorserAfterCursor: "4821",
partnerAfterCursor: "91-4821",
planHandleIds: ["https://endorser.ch/entity/01HPLAN"],
location,
},
auth: { claim: authorizeClaim },
});
assert.equal(result.status, 200, JSON.stringify(result.body));
const stored = await smsAlertSearchCursorsDb.get(USER);
assert.equal(stored?.endorserAfterCursor, "4821");
assert.equal(stored?.partnerAfterCursor, "91-4821");
assert.deepEqual(stored?.planHandleIds, ["https://endorser.ch/entity/01HPLAN"]);
assert.deepEqual(stored?.location, location);
// absent fields keep what is stored; null clears
await call({
path: "/notify-sms/alert-authorization",
body: { ...batch(), location: null },
auth: { claim: authorizeClaim },
});
const after = await smsAlertSearchCursorsDb.get(USER);
assert.equal(after?.endorserAfterCursor, "4821");
assert.deepEqual(after?.planHandleIds, ["https://endorser.ch/entity/01HPLAN"]);
assert.equal(after?.location, null);
});
it("rejects a cursor with a '.' or a malformed area, storing nothing", async () => {
await registerAndVerify();
const result = await call({
path: "/notify-sms/alert-authorization",
body: {
...batch(),
endorserAfterCursor: "1.2",
location: { minLocLat: 95, maxLocLat: 41, minLocLon: -112, maxLocLon: -111 },
},
auth: { claim: authorizeClaim },
});
assert.equal(result.status, 400);
assert.equal((result.body.details as string[]).length, 2);
assert.equal(await smsAlertSearchCursorsDb.get(USER), undefined);
assert.equal(await smsAlertAuthorizationDb.getLatestBatch(USER), undefined);
});
it("stores the hour and minute and echoes both back", async () => {