nexusAI/packages/memory-service/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

See the root [CLAUDE.md](../../CLAUDE.md) for overall architecture, service roles, and the dual-store memory model.

## Running This Service

```bash
npm run memory             # From repo root (node src/index.js)
npm -w packages/memory-service run dev   # With --watch
```

Default port: **3002**. Requires Qdrant and the embedding-service to be reachable on startup.

## SQLite Schema

`src/db/schema.js` is the source of truth for the data model. Key schema facts:

- `sessions` and `episodes` are linked by FK with cascade delete — deleting a session removes all its episodes automatically.
- `episodes_fts` is an FTS5 virtual table that mirrors `user_message` and `ai_response`. It is kept in sync via SQL triggers on INSERT/UPDATE/DELETE. On service startup, the FTS index is fully rebuilt from live episode data.
- Several columns (`sessions.name`, `sessions.project_id`, `entities.mention_count`, etc.) were added as migrations using `ALTER TABLE` wrapped in individual try-catch blocks. Failures are silently swallowed — if a column already exists, the alter fails and the service continues. The `idx_summaries_project` index is defined twice (benign duplicate).
- `summaries` rows with `session_id IS NULL` and a `project_id` represent project-level overviews, not session summaries. This distinction is how `GET /projects/:id/overview` works.
- `entity_episodes` is a join table linking entities to the episodes where they were first extracted. Used for provenance tracking and future orphan cleanup. Defined in `schema.js` (not a migration), so it exists on all installs.

**New columns on `entities` (added via migration):**
- `mention_count INTEGER DEFAULT 1` — incremented every time this entity is re-extracted
- `confidence REAL DEFAULT 1.0` — reserved for future confidence scoring
- `source TEXT DEFAULT 'extraction'` — `'extraction'` or `'manual'`
- `last_seen_at INTEGER` — Unix timestamp of most recent extraction hit

**New columns on `relationships` (added via migration):**
- `mention_count INTEGER DEFAULT 1` — incremented every time this edge is re-extracted
- `notes TEXT` — relationship context sentence from extraction

## Async Pipeline: Episode Creation

`POST /episodes` returns a 201 as soon as the SQLite insert succeeds. Two background tasks run after without blocking the response:

1. **Embedding** — Fetches a vector from embedding-service, stores to Qdrant with `{sessionId, createdAt}` as payload metadata.
2. **Entity + relationship extraction** — Sends the episode text to Ollama (`qwen2.5:3b`, temp 0.1, 1500 tokens) and upserts any recognized entities and relationships to both SQLite and Qdrant. Also links each entity to the episode via `entity_episodes`.

Both tasks catch and log errors silently. An episode can exist in SQLite with no corresponding Qdrant point if either step fails.

## Entity Extraction Details

`src/entities/extraction.js`:

- Fetches the last 20 known entities from SQLite before prompting the model, so the prompt can ask for name/type consistency with existing entries.
- Recognized entity types: `person`, `place`, `project`, `technology`, `concept`, `organization` — anything else is discarded.
- Ignores a hardcoded list of low-value names (`hello`, `thanks`, `good morning`, etc.).
- Extracts JSON using a regex (`{...}`) applied to raw model output, so surrounding prose doesn't break parsing.
- The model is asked to return both entities and relationships in a single JSON response: `{ "entities": [...], "relationships": [...] }`.
- Entity upsert uses `ON CONFLICT(name, type) DO UPDATE` — preserves existing `notes` if the new extraction returns null, increments `mention_count`, updates `last_seen_at`.
- Relationship upsert uses `ON CONFLICT(from_id, to_id, label) DO UPDATE` — increments `mention_count`, preserves existing `notes` if new is null.
- Relationships are resolved by looking up both endpoints in the `entityMap` built during entity processing — if either entity wasn't saved (filtered out or invalid type), the relationship is silently dropped.
- After upsert, embeds each entity as `"${name} (${type}): ${notes}"` and stores to Qdrant with `projectId` in the payload for project-scoped filtering.

> For full details see `docs/services/entity-extraction.md` and `docs/services/knowledge-graph.md`.

## Knowledge Graph

`src/graph/index.js` provides two SQLite traversal functions:

- **`getNeighborhood(entityId, depth)`** — Single-entity recursive CTE traversal. Bidirectional (follows edges in both directions). Returns `{ nodes: [...entities], edges: [...relationships] }`. Depth defaults to `ENTITIES.GRAPH_HOP_DEPTH` (1), max enforced to 3 at the HTTP layer.

- **`getEntityNeighbors(entityIds[])`** — Bulk 1-hop version for orchestration. Given a set of seed entity IDs, returns their immediate neighbors plus all edges within the combined node set.

The recursive CTE uses `UNION` (not `UNION ALL`) to eliminate cycles and duplicate visits automatically.

> For full design rationale and usage see `docs/services/knowledge-graph.md`.

## Summarization Strategy

`src/summarization/project.js`:

- Preferred path: generate a project overview from existing **session-level summaries** (higher-level abstraction, shorter input).
- Fallback path: if no session summaries exist, summarize raw episodes directly (up to `SUMMARIES.MAX_PROJECT_EPISODE_LIMIT`).
- Both paths truncate input at `SUMMARIES.MAX_SUMMARY_CHARS` (8,000 chars) by slicing from the end (most recent content wins).
- Strips ChatML tokens from the Ollama response (`<|im_start|>`, `<|im_end|>`).
- Uses temp 0.2 and `num_predict 1200`.

## Qdrant Client

`src/semantic/index.js` creates the Qdrant client lazily on first use and reuses it. All three collections (`episodes`, `entities`, `summaries`) are created at startup if missing. There is no connection health check — if Qdrant is unreachable, semantic operations throw at call time.

## API Endpoints Quick Reference

| Method | Path | Notes |
|---|---|---|
| GET | `/health` | Static response, no dependency checks |
| GET/POST | `/sessions` | POST requires `externalId`; duplicate → 409 |
| GET/PATCH | `/sessions/by-external/:externalId` | PATCH accepts `name`, `projectId` |
| DELETE | `/sessions/by-external/:externalId` | Cascades to episodes, summaries, relationships |
| GET/POST | `/episodes` | POST triggers async embedding + entity/relationship extraction |
| GET | `/episodes/search` | FTS5 search; route must precede `/:id` |
| GET | `/sessions/:id/episodes` | Paginated, ordered `created_at DESC` |
| DELETE | `/episodes/:id` | Removes from SQLite + async Qdrant delete |
| POST | `/entities` | Upsert by `(name, type)`; increments `mention_count` on conflict |
| GET | `/entities/by-type/:type` | All entities of given type |
| GET/DELETE | `/entities/:id` | |
| POST | `/relationships` | Upsert by `(fromId, toId, label)`; increments `mention_count` on conflict. Body: `fromId`, `toId`, `label`, `notes` (optional) |
| GET | `/entities/:id/relationships` | Outbound only |
| DELETE | `/relationships` | Body: `fromId`, `toId`, `label` |
| GET | `/graph/neighborhood/:entityId` | Single-entity neighborhood; `?depth=` (default 1, max 3) |
| POST | `/graph/neighbors` | Bulk 1-hop neighborhood; body: `{ entityIds: [...] }` |
| GET/POST | `/projects` | POST requires non-empty `name` |
| GET/PATCH/DELETE | `/projects/:id` | |
| POST | `/projects/:id/summarize` | On-demand overview generation; 422 if no data |
| GET | `/projects/:id/overview` | Returns null (not 404) if no overview exists |
| GET | `/projects/:id/summaries` | All summaries for project |
| POST | `/summaries` | Requires `content` + at least one of `sessionId`/`projectId` |
| GET | `/sessions/:id/summaries` | |
| PATCH/DELETE | `/summaries/:id` | |