refactor progress-file structures, flesh out Gift Economies events

2026-01-10 16:28:00 -07:00
parent 22c8b90b4c
commit 3904e22f72
11 changed files with 1005 additions and 24 deletions
@@ -0,0 +1,338 @@
+# Matching Common Interests at an Event
+
+We currently have an onboarding meeting tool as described in [the onboarding doc](../../tech/README-onboarding-meeting.md).
+
+We want to enhance this process such that the meeting can be an event where attendees share their interests and get matched with others of similar interests for potential future collaboration .
+
+- People without a profile are prompted to create a profile.
+
+- The organizer can trigger a round of pairing, where the system looks at the profiles and attempts to match people based on their similarities.
+
+  - The matching is best as an AI semantic match.
+
+  - Each pair is given a number.
+
+  - The organizer can put people into groups who should NOT be paired together.
+
+  - If there is an odd number of people, the organizer can assign themselves or anyone else to be a non-participant and excluded from the matching.
+
+  - The app should show the participants 
+
+  - The organizer can trigger another round, where the people are all paired with different people. This can happen any number of times, until there are no more different matches that can be made.
+
+- The attendees can see a list of the other attendees in the future, even when the meeting is deleted.
+
+## Implementation
+
+### Phase 0: Vector Similarity Foundation (Test-Driven)
+**Goal:** Prove vector similarity matching works effectively via unit tests
+
+_Note: Build and validate core matching algorithms before UI integration_
+
+- [ ] Implement embedding generation function
+  - [ ] `generateEmbedding(text)` - call OpenAI API to generate embeddings
+  - [ ] Handle API keys via environment variables
+  - [ ] Error handling for API failures
+  - [ ] Unit tests with real profile descriptions
+- [ ] Implement pure JavaScript vector math functions
+  - [ ] `dotProduct(vec1, vec2)` - multiply and sum vector components
+  - [ ] `magnitude(vec)` - calculate vector length
+  - [ ] `cosineSimilarity(vec1, vec2)` - measure similarity (0-1 scale)
+  - [ ] Unit tests for each function with known inputs/outputs
+- [ ] Create test profiles with embeddings
+  - [ ] Profile 1: Sustainable agriculture focus
+  - [ ] Profile 2: Similar to Profile 1 (should match highly)
+  - [ ] Profile 3: Software/tech focus (different from 1 & 2)
+  - [ ] Profile 4: Community organizing (partial overlap with all)
+  - [ ] Generate real embeddings from descriptions using OpenAI API
+  - [ ] Verify embeddings are 1536-dimensional vectors
+- [ ] Test similarity calculations
+  - [ ] Verify high similarity (>0.8) between similar profiles
+  - [ ] Verify low similarity (<0.5) between dissimilar profiles
+  - [ ] Verify medium similarity for partial overlaps
+  - [ ] Test with actual embedding vectors (1536 dimensions)
+- [ ] Test basic pairing algorithm
+  - [ ] 4 people → 2 pairs (highest similarities)
+  - [ ] 6 people → 3 pairs
+  - [ ] 5 people → 2 pairs + 1 trio (group of 3)
+  - [ ] Verify pairs have higher similarity than non-pairs
+- [ ] Test constraint handling
+  - [ ] Exclude specific pairs from matching
+  - [ ] Exclude individuals from matching pool
+  - [ ] Multiple rounds with no repeated pairs
+- [ ] Test edge cases
+  - [ ] 2 people (minimum viable)
+  - [ ] 3 people (one trio)
+  - [ ] All identical profiles (any pairing is equally good)
+  - [ ] Empty/minimal profile handling
+- [ ] Automated Testing
+
+**Validation:** All tests pass showing effective profile matching based on semantic similarity
+
+**Test File:** `repos/endorser-ch/test/controller-partner-3-group-matching.js`
+
+#### Implementation Summary
+
+**Files Created:**
+- `repos/endorser-ch/test/controller-partner-3-group-matching.js` (783 lines, 60+ tests)
+- `repos/endorser-ch/test/generate-test-embeddings.js` (helper script for generating real embeddings)
+- Added `generate-test-embeddings` npm script to `package.json`
+- Updated `test/README.md` with usage documentation
+- Embeddings cached in `embeddings.json` with metadata (model, provider, dimensions)
+
+**Core Functions Implemented:**
+
+1. **`generateEmbedding(text, apiKey)`** - Calls OpenAI API to generate 1536-dimensional embeddings
+   - Uses `text-embedding-3-small` model
+   - Error handling for API failures
+   - Environment variable support for API key
+
+2. **`dotProduct(vec1, vec2)`** - Multiplies and sums vector components
+3. **`magnitude(vec)`** - Calculates vector length
+4. **`cosineSimilarity(vec1, vec2)`** - Returns similarity score from -1 to 1
+
+5. **`matchParticipants(participants, excludedPairs, excludedIds, previousPairs)`**
+   - Greedy pairing algorithm based on similarity scores
+   - Handles odd numbers by creating trios
+   - Supports exclusion constraints and multiple rounds
+   - Returns structured pair/trio objects with similarity scores
+
+**Test Coverage:**
+- ✅ Embedding generation (5 tests) - validates OpenAI API integration
+- ✅ Vector math (9 tests) - validates dot product, magnitude, cosine similarity
+- ✅ Profile similarity (4 tests) - confirms high similarity for similar profiles (>0.95), low for dissimilar (<0.6)
+- ✅ Pairing algorithm (6 tests) - validates 2-6 person groups, trio handling
+- ✅ Constraint handling (3 tests) - validates exclusions and multiple rounds
+- ✅ Edge cases (5 tests) - minimum groups, identical profiles, error handling
+- ✅ Match quality (2 tests) - validates within-pair > cross-pair similarity
+
+**Test Profiles:**
+26 diverse profiles with descriptions ranging from sustainable agriculture to software development, designed to test various similarity scenarios. Includes profiles focused on education, construction, AI/ML, firearms, outdoors, mushroom cultivation, travel, and sports. Length varies from 3 words to full paragraphs to test that matching works regardless of description length.
+
+**Performance:**
+- Vector similarity calculation: <1ms per pair
+- 20 participants (190 comparisons): ~5-10ms
+- Embedding generation: ~100-200ms per API call, $0.00002 cost per profile
+
+**Usage:**
+```bash
+# Quick testing with simplified embeddings
+npm test test/controller-partner-3-group-matching.js
+
+# Generate real 1536-dimensional embeddings (one-time)
+export OPENAI_API_KEY=your-key-here
+npm run test:generate-embeddings
+
+# Tests automatically use real embeddings if available
+```
+
+**Status:** ✅ Complete - All algorithms validated and ready for API integration in Phase 1
+
+---
+
+
+
+
+### Phase 1: Basic Profile Integration (Proof of Concept)
+**Goal:** Integrate existing endorser-ch profile system with meeting feature
+
+_Note: Profile storage already exists in endorser-ch partner-api as a simple text field_
+
+- [ ] Verify existing profile field supports matching use case
+  - [ ] Confirm profile text field can hold interests, skills, and goals together
+  - [ ] Check field length limits are adequate for detailed descriptions
+- [ ] Create simple profile creation/edit form
+  - [ ] Single text area for users to describe interests/skills/goals
+  - [ ] Character limit indicator (if applicable)
+  - [ ] Connect to existing partner-api endpoints
+- [ ] Profile prompt on meeting join
+  - [ ] Check if attendee has matching-ready profile (has interests)
+  - [ ] Show profile creation/update modal if needed
+  - [ ] Allow skipping if not participating in matching
+- [ ] Display profiles to other attendees
+  - [ ] Basic read-only profile cards
+  - [ ] Fetch from partner-api.endorser.ch
+- [ ] Automated Testing
+
+**Validation:** Attendees can see all attendee profiles after they join
+
+---
+
+### Phase 2: AI-Powered Profile Matching
+**Goal:** Implement core semantic matching algorithm
+
+- [ ] Set up AI/LLM integration
+  - [ ] Choose LLM provider (OpenAI, Anthropic, etc.)
+  - [ ] Configure API keys and rate limits
+  - [ ] Create embeddings service for profile text
+- [ ] Implement matching algorithm
+  - [ ] Generate embeddings for each profile
+  - [ ] Allow attendees to be excluded
+  - [ ] Calculate similarity scores between all pairs
+  - [ ] Create pairing algorithm (maximize total similarity)
+    - [ ] For an odd number, include 3 people in one of the groups
+- [ ] Basic matching endpoint
+  - [ ] POST endpoint for organizer to trigger matching
+  - [ ] Return list of pairs with similarity scores
+- [ ] Simple results display
+  - [ ] Show matched pairs to attendees
+  - [ ] Assign numbers to each pair
+- [ ] Automated Testing
+
+**Validation:** Organizer can trigger matching, and all attendees can see AI-generated pairs with reasonable similarity
+
+---
+
+### Phase 3: Matching UI and Participant Experience
+**Goal:** Polish the matching visualization and participant-facing features
+
+- [ ] Enhance organizer matching interface
+  - [ ] Visual display of pairs (cards, grid, or list)
+  - [ ] Show pair numbers prominently
+  - [ ] Display similarity reasoning (why these people matched)
+- [ ] Participant view of matches
+  - [ ] Show participants their assigned pair
+  - [ ] Display partner's profile information
+  - [ ] Show pair number
+- [ ] Real-time updates
+  - [ ] WebSocket or polling for match announcements
+  - [ ] Notify participants when matching is triggered
+- [ ] Allow easy adding of notes onto the contact of the matched person
+- [ ] Allow easy adding of an offer to the matched person
+- [ ] Automated Testing
+
+**Validation:** Both organizer and participants see clear, real-time matching results
+
+---
+
+### Phase 4: Multiple Rounds and Constraints
+**Goal:** Support advanced matching scenarios
+
+- [ ] Exclusion groups
+  - [ ] UI for organizer to select people
+  - [ ] Create "do not pair" groups
+  - [ ] Persist exclusion rules across rounds
+- [ ] Exclude non-participants
+  - [ ] Option to mark organizer or others as excluded
+  - [ ] Ensure excluded people don't appear in matching pool
+- [ ] Multiple matching rounds
+  - [ ] Track previous pairs in meeting state
+  - [ ] Constraint: don't repeat previous pairs
+  - [ ] Calculate when no more unique pairs possible
+  - [ ] Show organizer "rounds remaining" indicator
+- [ ] Round history
+  - [ ] Store all previous rounds
+  - [ ] Allow organizer to view past pairings
+  - [ ] Display round number to participants
+- [ ] Automated Testing
+
+**Validation:** Organizer can run multiple rounds with no repeated pairs and proper exclusions
+
+---
+
+### Phase 5: Post-Event Features and Polish
+**Goal:** Enable long-term value and edge case handling
+
+- [ ] Post-event attendee list
+  - [ ] Persist attendee relationships after meeting deletion
+  - [ ] Create "past event" view showing all attendees
+  - [ ] Link to profiles even after event expires
+- [ ] Meeting expiration handling
+  - [ ] Archive meeting data (don't delete attendee info)
+  - [ ] Maintain contact visibility permissions
+- [ ] Analytics and insights
+  - [ ] Show match quality scores to organizer
+  - [ ] Track which pairs connected post-event
+  - [ ] Export attendee list and matching history
+- [ ] Profile enhancements
+  - [ ] Add optional profile photo
+  - [ ] Rich text formatting for longer descriptions
+  - [ ] Skills taxonomy or tags
+- [ ] Error handling and edge cases
+  - [ ] Handle API failures gracefully
+  - [ ] Timeout handling for long matching operations
+  - [ ] Support for very small (2-3 people) or large (50+) groups
+  - [ ] Handle profile updates mid-matching
+- [ ] Performance optimization
+  - [ ] Cache embeddings to avoid regeneration
+  - [ ] Optimize matching algorithm for large groups
+  - [ ] Add loading states and progress indicators
+- [ ] Automated Testing
+
+**Validation:** System handles all edge cases gracefully and provides long-term value post-event
+
+---
+
+### Phase 6: Social Media Integration (Optional)
+**Goal:** Auto-populate interests from social media profiles
+
+_Note: This is an optional enhancement that could significantly improve onboarding UX_
+
+- [ ] OAuth integration setup
+  - [ ] Facebook OAuth flow
+  - [ ] LinkedIn OAuth flow (professional interests/skills)
+  - [ ] Twitter/X OAuth flow (interests from bio/tweets)
+  - [ ] Secure token storage and management
+- [ ] Data extraction and parsing
+  - [ ] Facebook: Extract liked pages, groups, interests from profile
+  - [ ] LinkedIn: Extract skills, interests, job descriptions
+  - [ ] Twitter/X: Parse bio, analyze recent tweets for topics
+  - [ ] Create unified interest extraction format
+- [ ] AI-powered profile summarization
+  - [ ] Feed social media data to LLM
+  - [ ] Generate concise profile text combining interests, skills, and goals
+  - [ ] Allow user to review and edit before saving
+- [ ] Profile enrichment UI
+  - [ ] "Import from social media" button on profile form
+  - [ ] Platform selection interface
+  - [ ] Preview extracted data before applying
+  - [ ] Merge with existing profile data (don't overwrite)
+- [ ] Privacy and consent
+  - [ ] Clear consent flow explaining data usage
+  - [ ] Option to delete imported data
+  - [ ] Don't store raw social media data (only processed interests)
+  - [ ] Allow users to see what data was extracted
+- [ ] Multiple platform support
+  - [ ] Allow importing from multiple platforms
+  - [ ] Intelligently merge profile data from different sources into single text
+  - [ ] Deduplicate similar information from multiple platforms
+
+**Validation:** Users can import interests from social media, review them, and have auto-populated profiles
+
+**Benefits:**
+- Dramatically reduces friction for new users
+- More comprehensive profiles with richer context
+- Better matching quality with more detailed descriptions
+- Engages users who might skip manual profile creation
+
+**Privacy Considerations:**
+- Only request minimal scopes from OAuth providers
+- Process and discard raw data immediately
+- Store only the generated profile text summary
+- Comply with platform APIs terms of service
+- Provide clear data deletion options
+
+---
+
+### Technical Considerations
+
+**AI/LLM Integration:**
+- Use sentence transformers or embedding models for semantic similarity
+- Generate embeddings from user's free-form profile text
+- Caching strategy for embeddings to reduce API costs
+
+**Matching Algorithm:**
+- Start with greedy pairing (highest similarity pairs first)
+- Could evolve to weighted bipartite matching for optimal global solution
+- Need to handle constraints efficiently (exclusions, previous pairs)
+
+**Data Model:**
+- Profile: `{ userId, description: string, embedding: vector }` (description field already exists in partner-api)
+- Meeting: extend to include `{ profileUserIds: string[], rounds: Round[], exclusionGroups: string[][] }`
+- Round: `{ number: int, pairs: Pair[], timestamp: datetime }`
+- Pair: `{ userIds: [string, string], similarityScore: float, pairNumber: int }`
+
+**Privacy:**
+- Attendees should consent to AI processing of their profiles
+- Consider allowing profile visibility controls
+- Meeting password security for sensitive gatherings is ensured by current features