191 lines
7.5 KiB
Markdown
191 lines
7.5 KiB
Markdown
# Retrieval Fusion
|
||
|
||
**Implementation:** `packages/orchestration-service/src/chat/index.js`
|
||
**FTS scoping:** `packages/memory-service/src/episodic/index.js`, `src/index.js`
|
||
**Settings:** `semanticWeight`, `keywordWeight` via `PATCH /settings`
|
||
|
||
## Purpose
|
||
|
||
Rather than relying solely on Qdrant vector similarity (which finds semantically
|
||
related content but misses exact keyword matches) or FTS5 keyword search alone
|
||
(which finds exact matches but not paraphrases), Reciprocal Rank Fusion (RRF)
|
||
merges the ranked results from both strategies into a single better-ranked list.
|
||
|
||
Episodes that rank highly in **both** lists score highest. An episode that is
|
||
the top semantic match but irrelevant by keyword, or vice versa, scores lower
|
||
than one that satisfies both.
|
||
|
||
## How RRF Works
|
||
|
||
For each episode `d`, its fused score is:
|
||
|
||
```
|
||
RRF(d) = w_semantic / (k + rank_semantic(d))
|
||
+ w_keyword / (k + rank_keyword(d))
|
||
```
|
||
|
||
- `rank_i(d)` — 1-based position in that strategy's result list (episode absent from a list contributes 0 for that term)
|
||
- `k = 60` — smoothing constant (standard; not exposed in settings)
|
||
- `w_semantic`, `w_keyword` — user-tunable weights (both default-sourced from `RETRIEVAL` constants)
|
||
|
||
Setting a weight to `0` removes that strategy's contribution entirely. Setting
|
||
`keywordWeight` to `0` also short-circuits the FTS network call.
|
||
|
||
## Architecture
|
||
|
||
Fusion lives in orchestration — the service already coordinates multiple data
|
||
sources, and fusion is a retrieval strategy, not a storage concern.
|
||
|
||
```
|
||
getFusedEpisodes(…, queryVector, …)
|
||
├── getSemanticEpisodes(queryVector) — Qdrant search → fetch full rows by ID
|
||
│ (query embedded ONCE upstream in assembleContext and shared with entity
|
||
│ search — no longer embedded separately here)
|
||
└── getFTSResults() — memory-service /episodes/search → full rows directly
|
||
(skipped entirely if keywordWeight == 0)
|
||
↓
|
||
fuseEpisodeResults() — pure RRF, no I/O
|
||
↓
|
||
fusedEpisodes[] — top semanticLimit episodes by RRF score
|
||
```
|
||
|
||
The query embedding is computed once per turn in `assembleContext` and passed
|
||
into both fused retrieval and entity search; if embedding fails, both receive
|
||
`null` and degrade to empty results rather than erroring.
|
||
|
||
### Data Shape Consistency
|
||
|
||
Both sides must enter fusion as `Episode[]` — full SQLite row objects with
|
||
the same shape — and both must be filtered against `recentIds` first:
|
||
|
||
- **Semantic path**: `recentIds` filter applied before `getEpisodeById` fetch (existing behaviour)
|
||
- **FTS path**: full rows returned directly; `recentIds` filter applied in `getFusedEpisodes` after receiving them
|
||
|
||
FTS requests `semanticLimit * 2` results to provide headroom for the
|
||
`recentIds` filter without under-serving the fusion.
|
||
|
||
## Query Tokenization
|
||
|
||
Before FTS5 sees the query, `buildFtsQuery(query)` (in
|
||
`memory-service/src/episodic/index.js`) turns the raw message into a MATCH
|
||
expression:
|
||
|
||
1. Lowercase and split on any non-letter/number (`/[^\p{L}\p{N}]+/u`, unicode-aware)
|
||
2. Drop stopwords (a small `FTS_STOPWORDS` set of common function words) and single-character tokens
|
||
3. Wrap each surviving token in double quotes and join with ` OR `
|
||
|
||
So `"How do I configure the Qdrant collection?"` becomes
|
||
`"configure" OR "qdrant" OR "collection"`. If nothing survives (an
|
||
all-stopword message like `"how do I do it?"`), it returns `null` and
|
||
`searchEpisodes` returns `[]` — keyword search sits out that turn and
|
||
semantic retrieval carries it.
|
||
|
||
**Why this matters:** the earlier implementation quoted the *entire* message
|
||
as one FTS5 phrase, which required the whole string to appear verbatim in an
|
||
episode — so keyword recall was effectively nil for conversational queries.
|
||
Tokenizing into OR-joined terms is what makes `keywordWeight > 0` actually
|
||
contribute anything.
|
||
|
||
**Injection safety:** quoting each token individually means any
|
||
FTS5-significant token inside the user's message (a literal `OR`, `*`, `"`,
|
||
etc.) is matched as a search term rather than parsed as an operator. This
|
||
replaces the safety the old whole-phrase quoting provided.
|
||
|
||
The stopword set is deliberately conservative and tuned iteratively — high
|
||
frequency filler (`the`, `is`, `one`, `there`, …) is dropped, but borderline
|
||
words that can carry signal (`time`, `good`, `way`) are kept. Add to the set
|
||
when a common word is observed producing noisy matches.
|
||
|
||
## FTS Session Scoping
|
||
|
||
Without scoping, FTS5 searches across all episodes in the database. For
|
||
context assembly, results must be constrained to the current session or
|
||
project session pool — the same scope used for Qdrant semantic search.
|
||
|
||
`searchEpisodes(query, limit, sessionIds)` in memory-service accepts an
|
||
optional `sessionIds` array. When provided, the SQL becomes:
|
||
|
||
```sql
|
||
SELECT e.* FROM episodes e
|
||
JOIN episodes_fts fts ON e.id = fts.rowid
|
||
WHERE episodes_fts MATCH ?
|
||
AND e.session_id IN (?, ?, ...)
|
||
ORDER BY rank
|
||
LIMIT ?
|
||
```
|
||
|
||
The HTTP endpoint `GET /episodes/search` accepts `sessionIds` as a
|
||
comma-separated query param: `?q=hello&sessionIds=1,2,3`.
|
||
|
||
In orchestration, `ftsSessionIds` is set to:
|
||
- `projectSessionIds` (all sessions in the project) — if the session belongs to a project
|
||
- `[session.id]` — otherwise (single session only)
|
||
|
||
This mirrors the Qdrant scoping logic exactly.
|
||
|
||
## `fuseEpisodeResults` — Implementation Detail
|
||
|
||
```js
|
||
function fuseEpisodeResults(semanticEps, keywordEps, { semanticWeight, keywordWeight, limit }) {
|
||
const k = RETRIEVAL.RRF_K; // 60
|
||
const scores = new Map(); // episode.id → { episode, score }
|
||
|
||
// Score semantic results (already filtered against recentIds)
|
||
semanticEps.forEach((ep, i) => {
|
||
scores.set(ep.id, { episode: ep, score: semanticWeight / (k + i + 1) });
|
||
});
|
||
|
||
// Score + merge keyword results (already filtered against recentIds)
|
||
keywordEps.forEach((ep, i) => {
|
||
const contrib = keywordWeight / (k + i + 1);
|
||
if (scores.has(ep.id)) {
|
||
scores.get(ep.id).score += contrib; // appears in both — sum scores
|
||
} else if (contrib > 0) {
|
||
scores.set(ep.id, { episode: ep, score: contrib }); // FTS-only episode
|
||
}
|
||
// contrib == 0 (keywordWeight: 0) → episode not added (guard prevents score-0 bleed-through)
|
||
});
|
||
|
||
return [...scores.values()]
|
||
.sort((a, b) => b.score - a.score)
|
||
.slice(0, limit)
|
||
.map(({ episode }) => episode);
|
||
}
|
||
```
|
||
|
||
The `else if (contrib > 0)` guard prevents FTS-only episodes from entering
|
||
the result set with a score of 0 when `keywordWeight` is 0 — verified by
|
||
the test suite.
|
||
|
||
## Settings
|
||
|
||
| Setting | Default | Range | Description |
|
||
|---|---|---|---|
|
||
| `semanticWeight` | 1.0 | 0–5 | Weight applied to Qdrant semantic results |
|
||
| `keywordWeight` | 0 | 0–5 | Weight applied to FTS5 keyword results. `0` = disabled |
|
||
|
||
Both are readable via `GET /settings` and writable via `PATCH /settings`
|
||
without a service restart. Changes take effect on the next chat request.
|
||
|
||
**To enable keyword search:**
|
||
```bash
|
||
curl -X PATCH http://localhost:4000/settings \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"keywordWeight": 1.0}'
|
||
```
|
||
|
||
**To favour keyword matches over semantic:**
|
||
```bash
|
||
curl -X PATCH http://localhost:4000/settings \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"semanticWeight": 0.5, "keywordWeight": 2.0}'
|
||
```
|
||
|
||
## Constants (`packages/shared/src/config/constants.js`)
|
||
|
||
| Constant | Value | Description |
|
||
|---|---|---|
|
||
| `RETRIEVAL.RRF_K` | 60 | RRF smoothing constant — not exposed in settings |
|
||
| `RETRIEVAL.SEMANTIC_WEIGHT` | 1.0 | Default semantic weight |
|
||
| `RETRIEVAL.KEYWORD_WEIGHT` | 0 | Default keyword weight (off) |
|