utility inference cleanup
This commit is contained in:
Storme-bit
@@ -6,7 +6,7 @@ the full context window with raw episodes.
|
||||
|
||||
**Location:** `packages/orchestration-service/src/services/summarization.js`
|
||||
**Triggered by:** `chat/index.js` after every episode write (fire-and-forget)
|
||||
**Model:** `qwen2.5:3b` via Ollama on Mini PC 1 (192.168.0.81)
|
||||
**Model:** the utility model served by the inference service (`/utility/complete`), set via `UTILITY_MODEL` (backed by Ollama on Mini PC 1, 192.168.0.81)
|
||||
|
||||
---
|
||||
|
||||
@@ -56,47 +56,50 @@ not all episodes in the session.
|
||||
|
||||
---
|
||||
|
||||
## Ollama Request
|
||||
## Utility Inference Request
|
||||
|
||||
Summaries are generated through the shared `utilityInference()` helper, which
|
||||
POSTs to the inference service's `/utility/complete` endpoint. `buildSummaryPrompt`
|
||||
returns a plain instruction string (no template tags) passed as the `user` message:
|
||||
|
||||
```js
|
||||
{
|
||||
model: EXTRACTION_MODEL, // qwen2.5:3b (set via EXTRACTION_MODEL env var)
|
||||
prompt: buildSummaryPrompt(episodesToSummarize, existingSummary),
|
||||
stream: false,
|
||||
// No format: 'json' — free-text output required for summaries
|
||||
options: {
|
||||
temperature: 0.2,
|
||||
num_predict: 500,
|
||||
},
|
||||
}
|
||||
const content = await utilityInference({
|
||||
user: buildSummaryPrompt(episodesToSummarize, existingSummary),
|
||||
temperature: SUMMARIES.TEMPERATURE, // 0.2
|
||||
maxTokens: SUMMARIES.SESSION_GEN_MAX_TOKENS, // 500
|
||||
});
|
||||
```
|
||||
|
||||
`temperature: 0.2` is slightly higher than extraction (0.1) — summaries
|
||||
benefit from some fluency. `num_predict: 500` gives room for 5 thorough
|
||||
sentences without risk of runoff.
|
||||
`TEMPERATURE` (0.2) is slightly higher than extraction (0.1) — summaries benefit
|
||||
from some fluency. `SESSION_GEN_MAX_TOKENS` (500) gives room for ~5 thorough
|
||||
sentences without runoff. Both live in `@nexusai/shared` `SUMMARIES` constants.
|
||||
|
||||
There is no `json: true` here — summaries are free-text, unlike entity extraction.
|
||||
|
||||
---
|
||||
|
||||
## Prompt Format
|
||||
|
||||
ChatML format — native to qwen2.5:
|
||||
The prompt is plain text describing the task; the model's own prompt template
|
||||
(ChatML for qwen, etc.) is applied **server-side** by the inference service via
|
||||
Ollama's `/api/chat`. No `<|im_start|>` tags belong in this codebase, and the
|
||||
utility model can be swapped (via `UTILITY_MODEL` on the inference service) with
|
||||
no prompt changes here.
|
||||
|
||||
Fresh summary instruction:
|
||||
|
||||
```
|
||||
<|im_start|>user
|
||||
Summarize the conversation below in 3-5 sentences.
|
||||
Write in third person. Do not quote directly — paraphrase only.
|
||||
Do not include greetings, sign-offs, or filler. Output only the summary text.
|
||||
|
||||
Conversation:
|
||||
{context}
|
||||
<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
For cumulative updates, the instruction and context change:
|
||||
Cumulative update instruction:
|
||||
|
||||
```
|
||||
<|im_start|>user
|
||||
Update the summary below to incorporate the new exchanges.
|
||||
Write 3-5 sentences in third person. Do not quote directly — paraphrase only.
|
||||
Do not include greetings, sign-offs, or filler. Output only the updated summary text.
|
||||
@@ -106,35 +109,22 @@ Previous summary:
|
||||
|
||||
New exchanges:
|
||||
{context}
|
||||
<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
### Input truncation
|
||||
|
||||
Episode context is truncated to `MAX_CHARS = 3000` characters, keeping the
|
||||
most recent exchanges (sliced from the end). This keeps Qwen focused and
|
||||
most recent exchanges (sliced from the end). This keeps the model focused and
|
||||
prevents the prompt from exceeding its effective context window.
|
||||
|
||||
---
|
||||
|
||||
## ChatML Token Stripping
|
||||
## Output Handling
|
||||
|
||||
Qwen occasionally echoes ChatML tokens back into its response. The raw output
|
||||
is cleaned before saving:
|
||||
|
||||
```js
|
||||
const raw = data.response?.trim() ?? '';
|
||||
const content = raw
|
||||
.replace(/<\|im_start\|>.*?<\|im_end\|>/gs, '')
|
||||
.replace(/<\|im_start\|>|<\|im_end\|>|<\|im_sep\|>/g, '')
|
||||
.trim();
|
||||
return content;
|
||||
```
|
||||
|
||||
Without this, leaked tokens get stored in the summary and then injected
|
||||
back into the next summarisation prompt — causing the model to append a new
|
||||
summary after the old one rather than replacing it.
|
||||
Because `/api/chat` applies and removes the prompt template server-side, the
|
||||
returned text is already clean — the previous ChatML token-stripping step (and
|
||||
the class of bug where leaked tokens got stored and re-injected into the next
|
||||
summarisation prompt) no longer applies.
|
||||
|
||||
---
|
||||
|
||||
@@ -207,8 +197,7 @@ Set in `packages/orchestration-service/src/.env`:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `EXTRACTION_URL` | `http://localhost:11434` | Ollama instance URL |
|
||||
| `EXTRACTION_MODEL` | `qwen2.5:3b` | Model for summarisation |
|
||||
| `INFERENCE_SERVICE_URL` | `http://localhost:3001` | Inference service — summaries route through its `/utility/complete` endpoint (model set via `UTILITY_MODEL` there) |
|
||||
| `MEMORY_SERVICE_URL` | `http://localhost:3002` | Memory service URL |
|
||||
| `SUMMARY_THRESHOLD_TOKENS` | `200` | Token threshold before summarisation triggers |
|
||||
| `SUMMARY_MAX_TOKENS` | `800` | Max summary length before a new row is created |
|
||||
|
||||
Reference in New Issue
Block a user