|
|
|
@@ -6,7 +6,7 @@ the full context window with raw episodes.
|
|
|
|
|
|
|
|
|
|
**Location:** `packages/orchestration-service/src/services/summarization.js`
|
|
|
|
|
**Triggered by:** `chat/index.js` after every episode write (fire-and-forget)
|
|
|
|
|
**Model:** `qwen2.5:3b` via Ollama on Mini PC 1 (192.168.0.81)
|
|
|
|
|
**Model:** the utility model served by the inference service (`/utility/complete`), set via `UTILITY_MODEL` (backed by Ollama on Mini PC 1, 192.168.0.81)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
@@ -56,47 +56,50 @@ not all episodes in the session.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Ollama Request
|
|
|
|
|
## Utility Inference Request
|
|
|
|
|
|
|
|
|
|
Summaries are generated through the shared `utilityInference()` helper, which
|
|
|
|
|
POSTs to the inference service's `/utility/complete` endpoint. `buildSummaryPrompt`
|
|
|
|
|
returns a plain instruction string (no template tags) passed as the `user` message:
|
|
|
|
|
|
|
|
|
|
```js
|
|
|
|
|
{
|
|
|
|
|
model: EXTRACTION_MODEL, // qwen2.5:3b (set via EXTRACTION_MODEL env var)
|
|
|
|
|
prompt: buildSummaryPrompt(episodesToSummarize, existingSummary),
|
|
|
|
|
stream: false,
|
|
|
|
|
// No format: 'json' — free-text output required for summaries
|
|
|
|
|
options: {
|
|
|
|
|
temperature: 0.2,
|
|
|
|
|
num_predict: 500,
|
|
|
|
|
},
|
|
|
|
|
}
|
|
|
|
|
const content = await utilityInference({
|
|
|
|
|
user: buildSummaryPrompt(episodesToSummarize, existingSummary),
|
|
|
|
|
temperature: SUMMARIES.TEMPERATURE, // 0.2
|
|
|
|
|
maxTokens: SUMMARIES.SESSION_GEN_MAX_TOKENS, // 500
|
|
|
|
|
});
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`temperature: 0.2` is slightly higher than extraction (0.1) — summaries
|
|
|
|
|
benefit from some fluency. `num_predict: 500` gives room for 5 thorough
|
|
|
|
|
sentences without risk of runoff.
|
|
|
|
|
`TEMPERATURE` (0.2) is slightly higher than extraction (0.1) — summaries benefit
|
|
|
|
|
from some fluency. `SESSION_GEN_MAX_TOKENS` (500) gives room for ~5 thorough
|
|
|
|
|
sentences without runoff. Both live in `@nexusai/shared` `SUMMARIES` constants.
|
|
|
|
|
|
|
|
|
|
There is no `json: true` here — summaries are free-text, unlike entity extraction.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Prompt Format
|
|
|
|
|
|
|
|
|
|
ChatML format — native to qwen2.5:
|
|
|
|
|
The prompt is plain text describing the task; the model's own prompt template
|
|
|
|
|
(ChatML for qwen, etc.) is applied **server-side** by the inference service via
|
|
|
|
|
Ollama's `/api/chat`. No `<|im_start|>` tags belong in this codebase, and the
|
|
|
|
|
utility model can be swapped (via `UTILITY_MODEL` on the inference service) with
|
|
|
|
|
no prompt changes here.
|
|
|
|
|
|
|
|
|
|
Fresh summary instruction:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
<|im_start|>user
|
|
|
|
|
Summarize the conversation below in 3-5 sentences.
|
|
|
|
|
Write in third person. Do not quote directly — paraphrase only.
|
|
|
|
|
Do not include greetings, sign-offs, or filler. Output only the summary text.
|
|
|
|
|
|
|
|
|
|
Conversation:
|
|
|
|
|
{context}
|
|
|
|
|
<|im_end|>
|
|
|
|
|
<|im_start|>assistant
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
For cumulative updates, the instruction and context change:
|
|
|
|
|
Cumulative update instruction:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
<|im_start|>user
|
|
|
|
|
Update the summary below to incorporate the new exchanges.
|
|
|
|
|
Write 3-5 sentences in third person. Do not quote directly — paraphrase only.
|
|
|
|
|
Do not include greetings, sign-offs, or filler. Output only the updated summary text.
|
|
|
|
@@ -106,35 +109,22 @@ Previous summary:
|
|
|
|
|
|
|
|
|
|
New exchanges:
|
|
|
|
|
{context}
|
|
|
|
|
<|im_end|>
|
|
|
|
|
<|im_start|>assistant
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Input truncation
|
|
|
|
|
|
|
|
|
|
Episode context is truncated to `MAX_CHARS = 3000` characters, keeping the
|
|
|
|
|
most recent exchanges (sliced from the end). This keeps Qwen focused and
|
|
|
|
|
most recent exchanges (sliced from the end). This keeps the model focused and
|
|
|
|
|
prevents the prompt from exceeding its effective context window.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## ChatML Token Stripping
|
|
|
|
|
## Output Handling
|
|
|
|
|
|
|
|
|
|
Qwen occasionally echoes ChatML tokens back into its response. The raw output
|
|
|
|
|
is cleaned before saving:
|
|
|
|
|
|
|
|
|
|
```js
|
|
|
|
|
const raw = data.response?.trim() ?? '';
|
|
|
|
|
const content = raw
|
|
|
|
|
.replace(/<\|im_start\|>.*?<\|im_end\|>/gs, '')
|
|
|
|
|
.replace(/<\|im_start\|>|<\|im_end\|>|<\|im_sep\|>/g, '')
|
|
|
|
|
.trim();
|
|
|
|
|
return content;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Without this, leaked tokens get stored in the summary and then injected
|
|
|
|
|
back into the next summarisation prompt — causing the model to append a new
|
|
|
|
|
summary after the old one rather than replacing it.
|
|
|
|
|
Because `/api/chat` applies and removes the prompt template server-side, the
|
|
|
|
|
returned text is already clean — the previous ChatML token-stripping step (and
|
|
|
|
|
the class of bug where leaked tokens got stored and re-injected into the next
|
|
|
|
|
summarisation prompt) no longer applies.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
@@ -207,8 +197,7 @@ Set in `packages/orchestration-service/src/.env`:
|
|
|
|
|
|
|
|
|
|
| Variable | Default | Description |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| `EXTRACTION_URL` | `http://localhost:11434` | Ollama instance URL |
|
|
|
|
|
| `EXTRACTION_MODEL` | `qwen2.5:3b` | Model for summarisation |
|
|
|
|
|
| `INFERENCE_SERVICE_URL` | `http://localhost:3001` | Inference service — summaries route through its `/utility/complete` endpoint (model set via `UTILITY_MODEL` there) |
|
|
|
|
|
| `MEMORY_SERVICE_URL` | `http://localhost:3002` | Memory service URL |
|
|
|
|
|
| `SUMMARY_THRESHOLD_TOKENS` | `200` | Token threshold before summarisation triggers |
|
|
|
|
|
| `SUMMARY_MAX_TOKENS` | `800` | Max summary length before a new row is created |
|
|
|
|
|
Storme-bit