---
description: Endorser-ch API pagination conventions and database migration documentation
globs: repos/endorser-ch/src/**
alwaysApply: true
---

# Endorser-ch Development Conventions

# Pagination Conventions

### Standard Pagination Parameters

### Required Pattern
- **`beforeId`**: JWT ID to end before (exclusive) - primary pagination parameter
- **`afterId`**: JWT ID to start after (exclusive) - used for feed-style results where newer items are needed

### Parameter Usage
- **Routes**: Accept `beforeId` and `afterId` as query parameters
- **Database methods**: Pass these parameters through to database layer
- **Never use**: `limit`, `offset`, `page`, `skip`, or similar traditional pagination

### Limit
- The limit is managed by the DB service, currently always set to DEFAULT_LIMIT

## API Route Implementation

### Query Parameter Handling
```javascript
// ✅ CORRECT - Use beforeId/afterId pattern
const beforeId = req.query.beforeId
const afterId = req.query.afterId
dbService.methodByParamsPaged(params, afterId, beforeId)

// ❌ WRONG - Never use limit/offset
const limit = parseInt(req.query.limit) || 50  // DON'T DO THIS
const offset = parseInt(req.query.offset) || 0 // DON'T DO THIS
```

### JSDoc Documentation Pattern
```javascript
/**
 * @param {string} beforeId.query.optional - the JWT ID of the entry before which to look (exclusive); by default, the last one is included
 * @param {string} afterId.query.optional - the JWT ID of the entry after which to look (exclusive); by default, the first one is included
 */
```

## Database Service Implementation

### Method Signature Pattern
```javascript
// ✅ CORRECT - Standard pagination signature
methodByParamsPaged(params, afterIdInput, beforeIdInput) {
  // Implementation using JWT ID comparisons
}

// ❌ WRONG - Never use limit/offset in method signatures
methodByParamsPaged(params, limit, offset) { // DON'T DO THIS
```

### SQL Implementation
```javascript
// ✅ CORRECT - Use JWT ID comparisons
let sql = "SELECT * FROM table WHERE conditions"
if (afterIdInput) {
  params.push(afterIdInput)
  sql += " AND jwtId > ?"
}
if (beforeIdInput) {
  params.push(beforeIdInput)
  sql += " AND jwtId < ?"
}
sql += " ORDER BY jwtId DESC LIMIT " + DEFAULT_LIMIT

// ❌ WRONG - Never use OFFSET
sql += " LIMIT ? OFFSET ?" // DON'T DO THIS
```

### Response Format
```javascript
// ✅ CORRECT - Standard response format
const result = { 
  data: processedResults,
  hitLimit: results.length === DEFAULT_LIMIT 
}

// ❌ WRONG - Don't include limit/offset metadata
const result = { 
  data: results,
  limit: 50,    // DON'T DO THIS
  offset: 100   // DON'T DO THIS
}
```

## Consistent Patterns Across Codebase

### Why This Pattern?
1. **Consistency**: JWT IDs provide stable, chronological ordering
2. **Performance**: Index-based lookups are faster than OFFSET
3. **Reliability**: Results don't shift when new records are added

## Implementation Checklist

When creating new paginated endpoints:

- [ ] Use `beforeId`/`afterId` query parameters in routes
- [ ] Pass these parameters to database methods as `afterIdInput`/`beforeIdInput`
- [ ] Use JWT ID comparisons (`jwtId > ?`, `jwtId < ?`) in SQL
- [ ] Return `{ data: [], hitLimit: boolean }` response format
- [ ] Document parameters with standard JSDoc format
- [ ] Never use `limit`, `offset`, `page`, or similar parameters

## Error Prevention

**Red Flags** - These patterns should be avoided:
- Any mention of `limit` or `offset` in parameter names
- `LIMIT ? OFFSET ?` in SQL queries
- `parseInt(req.query.limit)` in route handlers
- Response objects with `page`, `totalPages`, `totalCount` fields

**Green Flags** - These patterns should be used:
- `beforeId`/`afterId` parameter naming
- `jwtId > ?` and `jwtId < ?` SQL conditions
- `hitLimit` boolean in responses
- `DEFAULT_LIMIT` constant for consistent limits

# Database Migration Documentation

## CRITICAL: Always Update sql/README.md

**Every database migration MUST be documented in `sql/README.md`** because migration files cannot be edited after flyway has run.

## Migration Documentation Process

When creating a new migration file (e.g., `V18__new_feature.sqlite3`):

1. **Create the migration file** with proper SQL schema changes
2. **Immediately update `sql/README.md`** with the new schema
3. **Add explanatory comments** for complex fields or relationships
4. **Update indexes** documentation if new indexes are created

## Schema Update Checklist

When creating a migration:

- [ ] Create migration file with proper version number (VXX__description.sqlite3)
- [ ] Add comprehensive SQL comments in migration file
- [ ] Update sql/README.md with complete table schema
- [ ] Document all new indexes in README.md
- [ ] Explain field purposes and constraints
- [ ] Verify README.md reflects current schema state
- [ ] Test migration on development database

## Why This Documentation Matters

1. **Flyway Immutability**: Migration files cannot be edited after deployment
2. **Schema Clarity**: README.md provides the definitive current schema
3. **Developer Onboarding**: New developers need clear schema documentation
4. **Maintenance**: Future changes require understanding current structure

# Data Privacy and DID Hiding

## hideDidsAndAddLinksToNetwork Function

The `hideDidsAndAddLinksToNetwork` function is a critical security component that ensures searches for DIDs (Decentralized Identifiers) only show results that are visible to users who have proper permissions to see them.

### Purpose

This function processes API response data to:
1. **Hide DIDs** that the requesting user doesn't have permission to see
2. **Add network links** showing which DIDs in the user's network can see hidden data
3. **Include public URLs** for DIDs that have published public profiles
4. **Validate search terms** to prevent data leakage through search queries

### Function Signatures

```javascript
// Primary function
async function hideDidsAndAddLinksToNetwork(requesterDid, inputData, searchTerms)

// Wrapper for responses with a "data" key
async function hideDidsAndAddLinksToNetworkInDataKey(requesterDid, input, searchTerms)
```

### Critical searchTerms Parameter

**SECURITY REQUIREMENT**: The `searchTerms` parameter MUST contain any user-provided query data that might include DIDs or parts of DIDs.

#### Why searchTerms is Required

The function validates that after hiding DIDs, all search terms are still visible in the result. This prevents users from:
- Searching for a DID they can't see and getting results that reveal its existence
- Using partial DID strings to probe for hidden data
- Discovering private information through search inference

#### Implementation Pattern

```javascript
// ✅ CORRECT - Include all user query parameters that might contain DIDs
const searchTermMayBeDIDs = [
  req.query.claimContents,
  req.query.issuer,
  req.query.subject,
  req.query.handleId
]
dbService.someMethod(params)
  .then(results => hideDidsAndAddLinksToNetwork(
    res.locals.authTokenIssuer, 
    results, 
    searchTermMayBeDIDs
  ))

// ❌ WRONG - Empty array when user provided search terms
.then(results => hideDidsAndAddLinksToNetwork(
  res.locals.authTokenIssuer, 
  results, 
  [] // DON'T DO THIS if user provided searchable parameters
))
```

### Usage Guidelines

#### When to Use Empty searchTerms Array

Use `[]` only when:
- No user input was used to filter the results
- The endpoint doesn't accept search parameters
- You're certain no DIDs could be in the query parameters

#### When to Include Search Terms

Always include search terms when:
- User provided `claimContents`, `issuer`, `subject`, `handleId` parameters
- Any text search functionality is involved
- User input could contain DIDs or partial DIDs
- The query filters data based on user-provided strings

### Security Checklist

When implementing endpoints that use `hideDidsAndAddLinksToNetwork`:

- [ ] Identify all user-provided query parameters
- [ ] Include any parameter that could contain DIDs in `searchTerms`
- [ ] Include any text search parameters in `searchTerms`
- [ ] Use empty array `[]` only when no user input affects the query
- [ ] Test that search results don't leak hidden DID information
- [ ] Verify that partial DID searches are properly filtered

### Error Prevention

**Red Flags** - These patterns indicate potential security issues:
- Using `[]` for searchTerms when user provided search parameters
- Not including text search fields in searchTerms
- Forgetting to include DID-related query parameters
- Bypassing the function for "simple" endpoints that still return DID data

**Green Flags** - These patterns indicate proper usage:
- Including all user query parameters that might contain DIDs
- Using `[]` only for endpoints with no user search input
- Consistent application across all endpoints returning DID data
- Proper testing of search result filtering