diff --git a/.cursor/rules/endorser-service.mdc b/.cursor/rules/endorser-service.mdc new file mode 100644 index 0000000..2854330 --- /dev/null +++ b/.cursor/rules/endorser-service.mdc @@ -0,0 +1,260 @@ +--- +description: Endorser-ch API pagination conventions and database migration documentation +globs: repos/endorser-ch/src/** +alwaysApply: true +--- + +# Endorser-ch Development Conventions + +# Pagination Conventions + +### Standard Pagination Parameters + +### Required Pattern +- **`beforeId`**: JWT ID to end before (exclusive) - primary pagination parameter +- **`afterId`**: JWT ID to start after (exclusive) - used for feed-style results where newer items are needed + +### Parameter Usage +- **Routes**: Accept `beforeId` and `afterId` as query parameters +- **Database methods**: Pass these parameters through to database layer +- **Never use**: `limit`, `offset`, `page`, `skip`, or similar traditional pagination + +### Limit +- The limit is managed by the DB service, currently always set to DEFAULT_LIMIT + +## API Route Implementation + +### Query Parameter Handling +```javascript +// ✅ CORRECT - Use beforeId/afterId pattern +const beforeId = req.query.beforeId +const afterId = req.query.afterId +dbService.methodByParamsPaged(params, afterId, beforeId) + +// ❌ WRONG - Never use limit/offset +const limit = parseInt(req.query.limit) || 50 // DON'T DO THIS +const offset = parseInt(req.query.offset) || 0 // DON'T DO THIS +``` + +### JSDoc Documentation Pattern +```javascript +/** + * @param {string} beforeId.query.optional - the JWT ID of the entry before which to look (exclusive); by default, the last one is included + * @param {string} afterId.query.optional - the JWT ID of the entry after which to look (exclusive); by default, the first one is included + */ +``` + +## Database Service Implementation + +### Method Signature Pattern +```javascript +// ✅ CORRECT - Standard pagination signature +methodByParamsPaged(params, afterIdInput, beforeIdInput) { + // Implementation using JWT ID comparisons +} + +// ❌ WRONG - Never use limit/offset in method signatures +methodByParamsPaged(params, limit, offset) { // DON'T DO THIS +``` + +### SQL Implementation +```javascript +// ✅ CORRECT - Use JWT ID comparisons +let sql = "SELECT * FROM table WHERE conditions" +if (afterIdInput) { + params.push(afterIdInput) + sql += " AND jwtId > ?" +} +if (beforeIdInput) { + params.push(beforeIdInput) + sql += " AND jwtId < ?" +} +sql += " ORDER BY jwtId DESC LIMIT " + DEFAULT_LIMIT + +// ❌ WRONG - Never use OFFSET +sql += " LIMIT ? OFFSET ?" // DON'T DO THIS +``` + +### Response Format +```javascript +// ✅ CORRECT - Standard response format +const result = { + data: processedResults, + hitLimit: results.length === DEFAULT_LIMIT +} + +// ❌ WRONG - Don't include limit/offset metadata +const result = { + data: results, + limit: 50, // DON'T DO THIS + offset: 100 // DON'T DO THIS +} +``` + +## Consistent Patterns Across Codebase + +### Why This Pattern? +1. **Consistency**: JWT IDs provide stable, chronological ordering +2. **Performance**: Index-based lookups are faster than OFFSET +3. **Reliability**: Results don't shift when new records are added + +## Implementation Checklist + +When creating new paginated endpoints: + +- [ ] Use `beforeId`/`afterId` query parameters in routes +- [ ] Pass these parameters to database methods as `afterIdInput`/`beforeIdInput` +- [ ] Use JWT ID comparisons (`jwtId > ?`, `jwtId < ?`) in SQL +- [ ] Return `{ data: [], hitLimit: boolean }` response format +- [ ] Document parameters with standard JSDoc format +- [ ] Never use `limit`, `offset`, `page`, or similar parameters + +## Error Prevention + +**Red Flags** - These patterns should be avoided: +- Any mention of `limit` or `offset` in parameter names +- `LIMIT ? OFFSET ?` in SQL queries +- `parseInt(req.query.limit)` in route handlers +- Response objects with `page`, `totalPages`, `totalCount` fields + +**Green Flags** - These patterns should be used: +- `beforeId`/`afterId` parameter naming +- `jwtId > ?` and `jwtId < ?` SQL conditions +- `hitLimit` boolean in responses +- `DEFAULT_LIMIT` constant for consistent limits + +# Database Migration Documentation + +## CRITICAL: Always Update sql/README.md + +**Every database migration MUST be documented in `sql/README.md`** because migration files cannot be edited after flyway has run. + +## Migration Documentation Process + +When creating a new migration file (e.g., `V18__new_feature.sqlite3`): + +1. **Create the migration file** with proper SQL schema changes +2. **Immediately update `sql/README.md`** with the new schema +3. **Add explanatory comments** for complex fields or relationships +4. **Update indexes** documentation if new indexes are created + +## Schema Update Checklist + +When creating a migration: + +- [ ] Create migration file with proper version number (VXX__description.sqlite3) +- [ ] Add comprehensive SQL comments in migration file +- [ ] Update sql/README.md with complete table schema +- [ ] Document all new indexes in README.md +- [ ] Explain field purposes and constraints +- [ ] Verify README.md reflects current schema state +- [ ] Test migration on development database + +## Why This Documentation Matters + +1. **Flyway Immutability**: Migration files cannot be edited after deployment +2. **Schema Clarity**: README.md provides the definitive current schema +3. **Developer Onboarding**: New developers need clear schema documentation +4. **Maintenance**: Future changes require understanding current structure + +# Data Privacy and DID Hiding + +## hideDidsAndAddLinksToNetwork Function + +The `hideDidsAndAddLinksToNetwork` function is a critical security component that ensures searches for DIDs (Decentralized Identifiers) only show results that are visible to users who have proper permissions to see them. + +### Purpose + +This function processes API response data to: +1. **Hide DIDs** that the requesting user doesn't have permission to see +2. **Add network links** showing which DIDs in the user's network can see hidden data +3. **Include public URLs** for DIDs that have published public profiles +4. **Validate search terms** to prevent data leakage through search queries + +### Function Signatures + +```javascript +// Primary function +async function hideDidsAndAddLinksToNetwork(requesterDid, inputData, searchTerms) + +// Wrapper for responses with a "data" key +async function hideDidsAndAddLinksToNetworkInDataKey(requesterDid, input, searchTerms) +``` + +### Critical searchTerms Parameter + +**SECURITY REQUIREMENT**: The `searchTerms` parameter MUST contain any user-provided query data that might include DIDs or parts of DIDs. + +#### Why searchTerms is Required + +The function validates that after hiding DIDs, all search terms are still visible in the result. This prevents users from: +- Searching for a DID they can't see and getting results that reveal its existence +- Using partial DID strings to probe for hidden data +- Discovering private information through search inference + +#### Implementation Pattern + +```javascript +// ✅ CORRECT - Include all user query parameters that might contain DIDs +const searchTermMayBeDIDs = [ + req.query.claimContents, + req.query.issuer, + req.query.subject, + req.query.handleId +] +dbService.someMethod(params) + .then(results => hideDidsAndAddLinksToNetwork( + res.locals.authTokenIssuer, + results, + searchTermMayBeDIDs + )) + +// ❌ WRONG - Empty array when user provided search terms +.then(results => hideDidsAndAddLinksToNetwork( + res.locals.authTokenIssuer, + results, + [] // DON'T DO THIS if user provided searchable parameters +)) +``` + +### Usage Guidelines + +#### When to Use Empty searchTerms Array + +Use `[]` only when: +- No user input was used to filter the results +- The endpoint doesn't accept search parameters +- You're certain no DIDs could be in the query parameters + +#### When to Include Search Terms + +Always include search terms when: +- User provided `claimContents`, `issuer`, `subject`, `handleId` parameters +- Any text search functionality is involved +- User input could contain DIDs or partial DIDs +- The query filters data based on user-provided strings + +### Security Checklist + +When implementing endpoints that use `hideDidsAndAddLinksToNetwork`: + +- [ ] Identify all user-provided query parameters +- [ ] Include any parameter that could contain DIDs in `searchTerms` +- [ ] Include any text search parameters in `searchTerms` +- [ ] Use empty array `[]` only when no user input affects the query +- [ ] Test that search results don't leak hidden DID information +- [ ] Verify that partial DID searches are properly filtered + +### Error Prevention + +**Red Flags** - These patterns indicate potential security issues: +- Using `[]` for searchTerms when user provided search parameters +- Not including text search fields in searchTerms +- Forgetting to include DID-related query parameters +- Bypassing the function for "simple" endpoints that still return DID data + +**Green Flags** - These patterns indicate proper usage: +- Including all user query parameters that might contain DIDs +- Using `[]` only for endpoints with no user search input +- Consistent application across all endpoints returning DID data +- Proper testing of search result filtering \ No newline at end of file diff --git a/.cursor/rules/node-express.mdc b/.cursor/rules/node-express.mdc index 975e8c2..fa2a129 100644 --- a/.cursor/rules/node-express.mdc +++ b/.cursor/rules/node-express.mdc @@ -30,6 +30,8 @@ alwaysApply: false - Implement proper response formats - Document APIs properly +- Use JWT ID-based cursor pagination (beforeId/afterId) - NEVER use limit/offset pagination (see endorser-pagination.mdc) + ## Database Integration - Use proper ORM/ODM - Implement proper migrations @@ -38,6 +40,9 @@ alwaysApply: false - Use proper query optimization - Handle database errors properly +- Implement JWT ID-based cursor pagination in database methods - NEVER use LIMIT/OFFSET patterns +- Always update sql/README.md when creating migrations (see endorser-pagination.mdc) + ## Authentication - Implement proper JWT handling - Use proper password hashing diff --git a/progress/TASK-emojis.md b/progress/TASK-emojis.md index 0977d52..62c036c 100644 --- a/progress/TASK-emojis.md +++ b/progress/TASK-emojis.md @@ -7,10 +7,10 @@ Allow people to attach an emoji onto a record. The main entity to which emojis w ### 1. Data Structure Design **Emoji Claim Structure:** -- **Context**: `"@context": "https://endorser.ch"` +- **Context**: Optional, with default of `"@context": "https://endorser.ch"` - **Type**: `"Emoji"` - **Text**: Contains one emoji - e.g., `"👍"`, `"❤️"`, or `"🚀"` -- **Parent Item**: Object with `lastClaimId` field containing the handleId of the target action (e.g., GiveAction) +- **ParentItem**: Object with `lastClaimId` field containing the handleId of the target action (e.g., GiveAction) - Why not multiple emojis (eg. to optimize bandwidth & storage when using many)? Because there's a possibility of removing an emoji, and then the communication and logic (on both client & server) for determining which is off and which is on becomes more complicated. It's also not a very typical action: people usually attach one at a time. It's possible, so it's an optimization worth considering someday. @@ -56,6 +56,9 @@ Allow people to attach an emoji onto a record. The main entity to which emojis w - Any authenticated user can add emojis to any public GiveAction - Users can retrieve all emoji taggers on a particular GiveAction, though DIDs are subject to visibility constraints +**Test** +- [ ] Write tests for each case on the back end (multiple emojis, removal, etc) in a new test file + ### 5. Client-Side **Add to UI** - [ ] Add the button for adding emojis, and a click sends it @@ -84,7 +87,6 @@ export interface EmojiSummaryRecord { - [ ] Add `emojiCount` field (map of emoji to count) to `GiveSummaryRecord` interface ### 6. Test -- [ ] Write tests for each case on the back end (multiple emojis, removal, etc) - [ ] Write tests for the front end ## Implementation Notes