nexusAI/docs/services/inference-service.md

Inference Service

Package: @nexusai/inference-service
Location: packages/inference-service
Deployed on: Main PC (192.168.0.79)
Port: 3001

Purpose

Thin adapter layer around the local LLM runtime. Receives assembled context packages from the orchestration service and returns model responses. Uses a provider pattern to abstract the underlying runtime, making it straightforward to switch inference backends without changes to the rest of the system.

Dependencies

• express — HTTP API
• ollama — Ollama client (used by the Ollama provider, kept as fallback)
• dotenv — environment variable loading
• @nexusai/shared — shared utilities

Environment Variables

Variable	Required	Default	Description
PORT	No	3001	Port to listen on
INFERENCE_PROVIDER	No	llamacpp	Active provider (ollama or llamacpp)
INFERENCE_URL	No	http://localhost:8080	URL of the inference runtime
DEFAULT_MODEL	No	local-model	Default model name passed to the provider

INFERENCE_URL points to llama-server directly (port 8080), not to this service. The orchestration service uses INFERENCE_SERVICE_URL to reach this service on port 3001.

Provider Architecture

The active provider is selected at startup via INFERENCE_PROVIDER and loaded from src/providers/. Both providers expose identical function signatures.

Supported Providers

Provider	Value	Runtime
llama.cpp	llamacpp	llama.cpp server (OpenAI-compatible API) — current default
Ollama	ollama	Ollama via the ollama npm package — available as fallback

Switching providers requires only a .env change — no code modifications:

INFERENCE_PROVIDER=llamacpp
INFERENCE_URL=http://localhost:8080

The provider loader throws immediately on an unknown value, preventing silent misconfiguration.

LM Studio compatibility note: LM Studio exposes an OpenAI-compatible /v1/chat/completions endpoint with the same request shape as llama.cpp. A future lmstudio.js provider would be nearly identical to llamacpp.js — only the BASE_URL would differ. No architectural changes required.

Internal Structure

src/
├── providers/
│   ├── ollama.js      # Ollama provider
│   └── llamacpp.js    # llama.cpp provider (OpenAI-compatible REST)
├── routes/
│   └── inference.js   # /complete and /complete/stream route handlers
├── infer.js           # Provider loader — selects and re-exports active provider
└── index.js           # Express app + route definitions

llama.cpp Provider

Uses the OpenAI-compatible REST API exposed by llama-server.

Starting llama-server

Must be started manually on the main PC before the inference service can handle requests:

.\llama-gpu\llama-server.exe `
  -m .\models\gemma-4-26B-A4B-Claude-Distill-APEX-I-Mini.gguf `
  -ngl 99 `
  --reasoning off `
  --host 0.0.0.0 `
  --port 8080 `
  -c 64000

Flag	Description
-ngl 99	Offload as many layers as possible to GPU
--reasoning off	Disables thinking delay on Gemma 4 models
--host 0.0.0.0	Allows LAN connections
-c 64000	Context window size in tokens

-c 64000 is intentionally large. NexusAI's memory architecture handles context injection so 6–8K is often sufficient if VRAM pressure builds.

Model Naming

The model name in requests must match the name reported by llama-server including the .gguf extension:

Invoke-RestMethod -Uri "http://192.168.0.79:8080/v1/models"

Set DEFAULT_MODEL in .env to the exact reported name.

Inference Parameters

All parameters are resolved in resolveOptions() — falling back to INFERENCE_DEFAULTS from @nexusai/shared if not provided in the request. In normal usage, orchestration reads these from settings.json and forwards them on every request.

NexusAI option	API field	Default	Description
temperature	temperature	0.7	Response randomness (0 = deterministic)
maxTokens	max_tokens	1024	Max tokens to generate
topP	top_p	0.9	Nucleus sampling probability mass
topK	top_k	40	Top-K token candidates per step
repeatPenalty	repeat_penalty	1.1	Penalty for recently used tokens
seed	seed	null	null = random; integer for reproducible output

Streaming Response Format

The llama.cpp provider yields chunks in this shape:

{ response: "token text", done: false }
// final chunk:
{ response: '', done: true, model: "model-name.gguf", tokenCount: 42 }

The inference route re-emits as SSE:

data: {"response":"token text"}
data: {"done":true,"model":"model-name.gguf","tokenCount":42}
data: [DONE]

model and tokenCount are captured from the llama.cpp finish_reason: stop chunk and emitted on the done event.

SSE parsing (buffered)

llama-server emits SSE events as data: lines, but a single network chunk from res.body is not guaranteed to contain whole lines — an event can be split across two chunks (mid-data: prefix or even mid-JSON). The provider therefore accumulates a string buffer, splits on \n, and retains the final (possibly incomplete) element for the next chunk rather than parsing each raw chunk in isolation:

let buffer = '';
for await (const chunk of res.body) {
  buffer += Buffer.from(chunk).toString('utf-8');
  const lines = buffer.split('\n');
  buffer = lines.pop() ?? '';   // keep incomplete tail for next chunk
  for (const line of lines) yield* processLine(line.trim());
}
if (buffer.trim()) yield* processLine(buffer.trim());  // flush remainder

processLine() handles exactly one line: it skips non-data: lines and [DONE], guards JSON.parse in a try/catch (a malformed line logs a warning and is skipped rather than throwing and killing the stream), extracts delta.content, and captures model / token usage from the terminal chunks. Content is emitted only when a delta is present (if (delta) yield { response, done: false }).

This mirrors the same buffering the orchestration service uses when reading the inference stream. Without it, long responses intermittently die mid-stream on a chunk boundary, and a run where every content delta lands on a split line surfaces as an empty response with a done event.

For all HTTP endpoints, see api-routes.md.