# Retrieval Fusion

**Implementation:** `packages/orchestration-service/src/chat/index.js`  
**FTS scoping:** `packages/memory-service/src/episodic/index.js`, `src/index.js`  
**Settings:** `semanticWeight`, `keywordWeight` via `PATCH /settings`

## Purpose

Rather than relying solely on Qdrant vector similarity (which finds semantically
related content but misses exact keyword matches) or FTS5 keyword search alone
(which finds exact matches but not paraphrases), Reciprocal Rank Fusion (RRF)
merges the ranked results from both strategies into a single better-ranked list.

Episodes that rank highly in **both** lists score highest. An episode that is
the top semantic match but irrelevant by keyword, or vice versa, scores lower
than one that satisfies both.

## How RRF Works

For each episode `d`, its fused score is:

```
RRF(d) = w_semantic / (k + rank_semantic(d))
        + w_keyword  / (k + rank_keyword(d))
```

- `rank_i(d)` — 1-based position in that strategy's result list (episode absent from a list contributes 0 for that term)
- `k = 60` — smoothing constant (standard; not exposed in settings)
- `w_semantic`, `w_keyword` — user-tunable weights (both default-sourced from `RETRIEVAL` constants)

Setting a weight to `0` removes that strategy's contribution entirely. Setting
`keywordWeight` to `0` also short-circuits the FTS network call.

## Architecture

Fusion lives in orchestration — the service already coordinates multiple data
sources, and fusion is a retrieval strategy, not a storage concern.

```
getFusedEpisodes(…, queryVector, …)
├── getSemanticEpisodes(queryVector)  — Qdrant search → fetch full rows by ID
│   (query embedded ONCE upstream in assembleContext and shared with entity
│    search — no longer embedded separately here)
└── getFTSResults()           — memory-service /episodes/search → full rows directly
    (skipped entirely if keywordWeight == 0)
         ↓
fuseEpisodeResults()          — pure RRF, no I/O
         ↓
fusedEpisodes[]               — top semanticLimit episodes by RRF score
```

The query embedding is computed once per turn in `assembleContext` and passed
into both fused retrieval and entity search; if embedding fails, both receive
`null` and degrade to empty results rather than erroring.

### Data Shape Consistency

Both sides must enter fusion as `Episode[]` — full SQLite row objects with
the same shape — and both must be filtered against `recentIds` first:

- **Semantic path**: `recentIds` filter applied before `getEpisodeById` fetch (existing behaviour)
- **FTS path**: full rows returned directly; `recentIds` filter applied in `getFusedEpisodes` after receiving them

FTS requests `semanticLimit * 2` results to provide headroom for the
`recentIds` filter without under-serving the fusion.

## Query Tokenization

Before FTS5 sees the query, `buildFtsQuery(query)` (in
`memory-service/src/episodic/index.js`) turns the raw message into a MATCH
expression:

1. Lowercase and split on any non-letter/number (`/[^\p{L}\p{N}]+/u`, unicode-aware)
2. Drop stopwords (a small `FTS_STOPWORDS` set of common function words) and single-character tokens
3. Wrap each surviving token in double quotes and join with ` OR `

So `"How do I configure the Qdrant collection?"` becomes
`"configure" OR "qdrant" OR "collection"`. If nothing survives (an
all-stopword message like `"how do I do it?"`), it returns `null` and
`searchEpisodes` returns `[]` — keyword search sits out that turn and
semantic retrieval carries it.

**Why this matters:** the earlier implementation quoted the *entire* message
as one FTS5 phrase, which required the whole string to appear verbatim in an
episode — so keyword recall was effectively nil for conversational queries.
Tokenizing into OR-joined terms is what makes `keywordWeight > 0` actually
contribute anything.

**Injection safety:** quoting each token individually means any
FTS5-significant token inside the user's message (a literal `OR`, `*`, `"`,
etc.) is matched as a search term rather than parsed as an operator. This
replaces the safety the old whole-phrase quoting provided.

The stopword set is deliberately conservative and tuned iteratively — high
frequency filler (`the`, `is`, `one`, `there`, …) is dropped, but borderline
words that can carry signal (`time`, `good`, `way`) are kept. Add to the set
when a common word is observed producing noisy matches.

## FTS Session Scoping

Without scoping, FTS5 searches across all episodes in the database. For
context assembly, results must be constrained to the current session or
project session pool — the same scope used for Qdrant semantic search.

`searchEpisodes(query, limit, sessionIds)` in memory-service accepts an
optional `sessionIds` array. When provided, the SQL becomes:

```sql
SELECT e.* FROM episodes e
JOIN episodes_fts fts ON e.id = fts.rowid
WHERE episodes_fts MATCH ?
AND e.session_id IN (?, ?, ...)
ORDER BY rank
LIMIT ?
```

The HTTP endpoint `GET /episodes/search` accepts `sessionIds` as a
comma-separated query param: `?q=hello&sessionIds=1,2,3`.

In orchestration, `ftsSessionIds` is set to:
- `projectSessionIds` (all sessions in the project) — if the session belongs to a project
- `[session.id]` — otherwise (single session only)

This mirrors the Qdrant scoping logic exactly.

## `fuseEpisodeResults` — Implementation Detail

```js
function fuseEpisodeResults(semanticEps, keywordEps, { semanticWeight, keywordWeight, limit }) {
    const k = RETRIEVAL.RRF_K; // 60
    const scores = new Map();  // episode.id → { episode, score }

    // Score semantic results (already filtered against recentIds)
    semanticEps.forEach((ep, i) => {
        scores.set(ep.id, { episode: ep, score: semanticWeight / (k + i + 1) });
    });

    // Score + merge keyword results (already filtered against recentIds)
    keywordEps.forEach((ep, i) => {
        const contrib = keywordWeight / (k + i + 1);
        if (scores.has(ep.id)) {
            scores.get(ep.id).score += contrib;   // appears in both — sum scores
        } else if (contrib > 0) {
            scores.set(ep.id, { episode: ep, score: contrib });  // FTS-only episode
        }
        // contrib == 0 (keywordWeight: 0) → episode not added (guard prevents score-0 bleed-through)
    });

    return [...scores.values()]
        .sort((a, b) => b.score - a.score)
        .slice(0, limit)
        .map(({ episode }) => episode);
}
```

The `else if (contrib > 0)` guard prevents FTS-only episodes from entering
the result set with a score of 0 when `keywordWeight` is 0 — verified by
the test suite.

## Settings

| Setting | Default | Range | Description |
|---|---|---|---|
| `semanticWeight` | 1.0 | 0–5 | Weight applied to Qdrant semantic results |
| `keywordWeight` | 0 | 0–5 | Weight applied to FTS5 keyword results. `0` = disabled |

Both are readable via `GET /settings` and writable via `PATCH /settings`
without a service restart. Changes take effect on the next chat request.

**To enable keyword search:**
```bash
curl -X PATCH http://localhost:4000/settings \
  -H "Content-Type: application/json" \
  -d '{"keywordWeight": 1.0}'
```

**To favour keyword matches over semantic:**
```bash
curl -X PATCH http://localhost:4000/settings \
  -H "Content-Type: application/json" \
  -d '{"semanticWeight": 0.5, "keywordWeight": 2.0}'
```

## Constants (`packages/shared/src/config/constants.js`)

| Constant | Value | Description |
|---|---|---|
| `RETRIEVAL.RRF_K` | 60 | RRF smoothing constant — not exposed in settings |
| `RETRIEVAL.SEMANTIC_WEIGHT` | 1.0 | Default semantic weight |
| `RETRIEVAL.KEYWORD_WEIGHT` | 0 | Default keyword weight (off) |