memory isolation fix and session grouping in client

2026-04-19 02:09:12 -07:00
parent 56355d232b
commit 9c903a56ae
7 changed files with 238 additions and 96 deletions
--- a/docs/reference/Memory-isolation.md
+++ b/docs/reference/Memory-isolation.md
@@ -13,19 +13,21 @@ the toggle has been removed from the UI and `isolated: 1` is hardcoded on
 project creation.

 **Semantic search** — at inference time, the user's message is embedded and
-compared against past episodes in Qdrant to surface relevant context. The
-scope of this search is controlled by the project context.
+compared against past episodes and entities in Qdrant to surface relevant
+context. The scope of this search is controlled by the project context.

 ## Semantic Search Scope

-| Session state | Semantic search scope |
-|---|---|
-| No project | Own session's episodes only |
-| Assigned to a project | All episodes across all sessions in that project |
-| Removed from a project | Own session's episodes only (from that point) |
+| Session state | Episode search scope | Entity search scope |
+|---|---|---|
+| No project | All non-project episodes (shared pool) | No entity context |
+| Assigned to a project | All episodes across all sessions in that project | Entities tagged with that project |
+| Removed from a project | Back to shared non-project pool | Back to no entity context |

-Sessions with no project assigned behave as they always have —
-only their own past episodes are searched.
+Non-project sessions share a common memory pool — they can draw on each
+other's episodes via semantic search, but cannot access episodes from any
+project session. Project sessions are fully isolated from all non-project
+sessions and from other projects.

 ## How It Works

@@ -48,7 +50,7 @@ If the session belongs to any project, `projectSessionIds` is populated with
 the internal integer IDs of all sessions in that project — creating a shared
 memory pool across all conversations in the project.

-### Step 2 — Qdrant filter construction
+### Step 2 — Qdrant episode filter construction

 In `services/qdrant.js`, `searchEpisodes` builds the filter:

@@ -68,14 +70,45 @@ if (projectSessionIds) {
 `WHERE sessionId IN (...)`. When `projectSessionIds` is set, the single-session
 filter is not used.

-### Step 3 — Episode payloads
+### Step 3 — Entity search scoping
+
+Entity search is also project-scoped. `searchEntities` in `services/qdrant.js`
+accepts a `projectId` parameter and filters accordingly:
+
+```js
+if (projectId) {
+  body.filter = {
+    must: [{ key: 'projectId', match: { value: projectId } }]
+  };
+}
+// No filter for non-project sessions — entity context not provided
+```
+
+Non-project sessions receive no entity context. Project sessions only see
+entities extracted from conversations within that project.
+
+### Step 4 — Episode payloads

 Every episode upserted into Qdrant carries `{ sessionId, createdAt }` in its
-payload. `sessionId` here is the **internal integer ID** from SQLite. This
-is what the Qdrant filter matches against.
+payload. `sessionId` here is the **internal integer ID** from SQLite.

-This means the filter works correctly regardless of when episodes were created
-or when a session was added to a project — the payload is immutable.
+### Step 5 — Entity payloads
+
+Every entity upserted into Qdrant carries `{ name, type, notes, projectId }`
+in its payload. `projectId` is the integer project ID.
+
+Entities are extracted and stored with `projectId` by `extraction.js`, which
+receives it from `createEpisode` in `episodic/index.js`, which receives it
+from the memory service episode route, which receives it from orchestration's
+`createEpisode` call in `chat/index.js`. The full chain:
+
+```
+chat/index.js → memory.createEpisode(session.id, ..., session.project_id)
+  → POST /episodes { projectId }
+  → episodic.createEpisode(..., projectId)
+  → extractAndStoreEntities(userMessage, aiResponse, projectId)
+  → semantic.upsertEntity(id, vector, { name, type, notes, projectId })
+```

 ## Important Behaviours

@@ -88,6 +121,12 @@ project's session list.
 message, `getProjectSessions` will not include that session's ID, so its
 episodes disappear from the semantic search scope.

+**Entity tags are immutable.** Entities extracted from a session's episodes
+are tagged with the `projectId` at extraction time. If a session is later
+moved to a different project, its previously extracted entities retain the
+original `projectId`. New entities extracted after the move will use the new
+`projectId`. Re-tagging existing entities requires a Qdrant payload update.
+
 **New sessions created from ProjectView are assigned after the first message.**
 `handleNewProjectChat` in `App.jsx` calls `sendMessage` with the project ID,
 which is passed to `useChat`. After `onDone` fires, `useChat` calls
@@ -95,12 +134,27 @@ which is passed to `useChat`. After `onDone` fires, `useChat` calls
 brief window during the first message where the session has no project assigned.
 The project is correctly applied from the second message onward.

-## Qdrant Payload Structure
+## Verified Behaviours (tested April 2026)

-Episodes are stored with this payload:
+- Project sessions cannot read episodes from non-project sessions ✓
+- Non-project sessions cannot read episodes from project sessions ✓
+- Non-project sessions can read each other's episodes ✓
+- Adding a session to a project — its history joins the project pool immediately ✓
+- Removing a session from a project — exits the project pool immediately ✓
+- Entity contamination across projects eliminated by `projectId` filter ✓
+
+## Qdrant Payload Structures
+
+**Episodes:**
 ```json
 { "sessionId": 42, "createdAt": 1776080188 }
 ```

+**Entities:**
+```json
+{ "name": "NexusAI", "type": "project", "notes": "...", "projectId": 3 }
+```
+
 `sessionId` is the SQLite `sessions.id` integer, not the `external_id` UUID.
-This is important when building filters — always use internal IDs.
+`projectId` is the SQLite `projects.id` integer.
+Always use internal IDs when building Qdrant filters.
--- a/docs/services/chat-client.md
+++ b/docs/services/chat-client.md
@@ -92,14 +92,14 @@ src/
 │   └── useContextMenu.js    # Right-click context menu position and visibility
 ├── components/
 │   ├── App.jsx              # Root component — layout, shared state, view routing
-│   ├── Sidebar.jsx          # Left sidebar — projects, recent chats, navigation
+│   ├── Sidebar.jsx          # Left sidebar — projects, grouped recent chats, navigation
 │   ├── HomeView.jsx         # Landing screen — greeting, centred input, quick actions
-│   ├── ChatWindow.jsx       # Centre panel — message thread and input bar
+│   ├── ChatWindow.jsx       # Centre panel — message thread, back button, model pill
 │   ├── MessageBubble.jsx    # Individual message bubble — renders markdown via react-markdown
 │   ├── InfoPanel.jsx        # Right panel — model selector and session metadata (slide-in)
 │   ├── SessionModal.jsx     # Modal for session rename, project assignment, delete
 │   ├── ProjectModal.jsx     # Modal for project create, edit, delete
-│   ├── AllChatsView.jsx     # Full paginated session list with multi-select bulk delete
+│   ├── AllChatsView.jsx     # Paginated session list with project indicator column
 │   ├── AllProjectsView.jsx  # Project tile grid with create/edit/delete; tile click navigates to ProjectView
 │   ├── ProjectView.jsx      # Individual project — conversations, new chat input, memory
 │   │                        #   placeholder, user notes, ⋮ edit/delete menu
@@ -129,8 +129,13 @@ panel are persistent across all views.
 │  All Projects →  │  memory       → MemoryView    │
 │                  │                               │
 │ RECENT CHATS ▾   │                               │
-│  Session 1       │                               │
-│  Session 2       │                               │
+│  ● Project A     │                               │
+│    Session 1     │                               │
+│    Session 2     │                               │
+│  ● Project B     │                               │
+│    Session 3     │                               │
+│  Other           │                               │
+│    Session 4     │                               │
 │  All Chats →     │                               │
 │                  │                               │
 │ ⚙ Settings       │                               │
@@ -167,13 +172,17 @@ leaving `'home'` expands it.

 ## Home View

-`HomeView` is the landing screen shown on initial load and when there are no
-active sessions. It displays:
+`HomeView` is the landing screen shown on initial load. It displays:
 - Time-based greeting ("Morning / Afternoon / Evening, Tim")
 - Currently loaded model name (from `modelProps.modelAlias`, stripped of `.gguf`)
 - Centred textarea input — sending creates a new session and navigates to chat
 - Quick action pills that populate the input without auto-sending

+Sending from HomeView uses `handleHomeSend` in `App.jsx`, which calls
+`createSession()` (returns the new session object), then immediately calls
+`sendMessage` with the session passed directly as a parameter — avoiding the
+React state settling race condition that would cause the message to be dropped.
+
 ## CSS Architecture

 Styles follow a hybrid approach — CSS utility classes for static reusable
@@ -225,6 +234,11 @@ are written into the active assistant bubble token by token via
 `updateLastMessage`. The blinking cursor in `MessageBubble` is shown while
 `message.streaming === true`.

+`useChat.sendMessage` accepts an optional `session` parameter (4th arg) that
+overrides the closed-over `activeSession`. This is used by `handleHomeSend`
+and `handleNewProjectChat` in `App.jsx` to pass the newly created session
+object directly, avoiding React state settling races.
+
 `useChat` accepts an optional `projectId` parameter in `sendMessage`. After
 the first message completes in a new session, if `projectId` is set,
 `updateSession` is called to write the project assignment to the backend.
@@ -236,6 +250,9 @@ the `uuid` package. New sessions are created locally and auto-registered in
 the memory service on the first message. The session list refreshes after
 each completed response to surface newly created sessions.

+`useSession.createSession` returns the new session object — callers can pass
+it directly to `sendMessage` rather than waiting for React state to update.
+
 `useSession.selectSession` skips the history fetch for new (`isNew: true`)
 sessions — fetching history for an unsaved session would 404 since it doesn't
 exist in the backend yet.
@@ -267,6 +284,16 @@ mode, and delete confirmation in `confirm-delete` mode.
 - Dynamic `updateSession` SQL builds `SET` clause from only the fields passed — prevents accidental overwrites
 - `AllChatsView` pagination uses `CLIENT_DEFAULTS.PAGE_SIZE` (not `API_DEFAULTS.PAGE_SIZE` which doesn't exist)

+## Sidebar — Session Grouping
+
+Recent sessions in the sidebar are grouped by project under a colour dot +
+project name label. Unassigned sessions appear under "Other" if any project
+groups are present. The grouping is computed client-side from the `sessions`
+array and `projects` list already available in `App.jsx` — no extra API call.
+
+`AllChatsView` receives `projects` as a prop from `App.jsx` and displays a
+project indicator column (colour dot + truncated name) in each session row.
+
 ## Project Management

 All projects are isolated by default (`isolated: 1` hardcoded on create).
--- a/docs/services/memory-service.md
+++ b/docs/services/memory-service.md
@@ -180,7 +180,8 @@ After extraction, each entity is:
   the entity is new (`COALESCE(entities.notes, excluded.notes)` prevents
   overwriting existing notes with speculative updates)
 2. Embedded via the embedding service and upserted into the `entities`
-   Qdrant collection with `{ name, type, notes }` as payload
+   Qdrant collection with `{ name, type, notes, projectId }` as payload —
+   `projectId` scopes entities to their project for isolated retrieval

 The Qdrant payload stores enough information to reconstruct entity context
 at retrieval time without a SQLite roundtrip.
--- a/packages/chat-client/src/App.jsx
+++ b/packages/chat-client/src/App.jsx
@@ -166,6 +166,7 @@ export default function App() {
        <AllChatsView
          onBack={goBack}
          onSelectSession={session => { selectSession(session); navigate('chat'); }}
+          projects={projects}
        />
      )}

--- a/packages/chat-client/src/components/AllChatsView.jsx
+++ b/packages/chat-client/src/components/AllChatsView.jsx
@@ -1,10 +1,10 @@
 import React, { useState, useEffect } from 'react';
 import { fetchSessions, deleteSession } from '../api/orchestration';
-import { API_DEFAULTS, CLIENT_DEFAULTS } from '../config/constants';
+import { CLIENT_DEFAULTS } from '../config/constants';

 const PAGE_SIZE = CLIENT_DEFAULTS.PAGE_SIZE;

-export default function AllChatsView({ onSelectSession, onBack }) {
+export default function AllChatsView({ onSelectSession, onBack, projects }) {
  const [sessions, setSessions] = useState([]);
  const [loading, setLoading] = useState(true);
  const [page, setPage] = useState(0);
@@ -23,8 +23,6 @@ export default function AllChatsView({ onSelectSession, onBack }) {
    try {
      const data = await fetchSessions(PAGE_SIZE, p * PAGE_SIZE);
      setSessions(data);
-      // We don't have a total count from the API yet — infer pagination
-      // from whether we got a full page back
      setTotal(data.length === PAGE_SIZE ? (p + 2) * PAGE_SIZE : p * PAGE_SIZE + data.length);
    } catch (err) {
      console.error('[AllChatsView] Failed to load sessions:', err.message);
@@ -75,10 +73,14 @@ export default function AllChatsView({ onSelectSession, onBack }) {
    if (diffMins < 60)  return `${diffMins}m ago`;
    if (diffHours < 24) return `${diffHours}h ago`;
    if (diffDays === 1) return 'Yesterday';
-    // Absolute date past 1 day
    return date.toLocaleDateString([], { month: 'short', day: 'numeric', year: 'numeric' });
  }

+  function getProject(projectId) {
+    if (!projectId || !projects) return null;
+    return projects.find(p => p.id === projectId) ?? null;
+  }
+
  const totalPages = Math.ceil(total / PAGE_SIZE);
  const allSelected = sessions.length > 0 && selected.size === sessions.length;

@@ -119,7 +121,6 @@ export default function AllChatsView({ onSelectSession, onBack }) {
            <thead>
              <tr style={{ borderBottom: '1px solid var(--border)' }}>
                <th style={{ width: '36px', padding: '8px 0' }}>
-                  {/* Select all checkbox */}
                  <input
                    type="checkbox"
                    checked={allSelected}
@@ -128,12 +129,14 @@ export default function AllChatsView({ onSelectSession, onBack }) {
                  />
                </th>
                <th className="label-upper" style={{ textAlign: 'left', padding: '8px 12px' }}>Name</th>
-                <th className="label-upper" style={{ textAlign: 'right', padding: '8px 0', width: '120px' }}>Last Active</th>
+                <th className="label-upper" style={{ textAlign: 'left', padding: '8px 12px', width: '130px' }}>Project</th>
+                <th className="label-upper" style={{ textAlign: 'right', padding: '8px 0', width: '110px' }}>Last Active</th>
              </tr>
            </thead>
            <tbody>
              {sessions.map(session => {
                const isSelected = selected.has(session.external_id);
+                const project = getProject(session.project_id);
                return (
                  <tr
                    key={session.external_id}
@@ -162,6 +165,21 @@ export default function AllChatsView({ onSelectSession, onBack }) {
                        {session.name || session.external_id}
                      </button>
                    </td>
+                    <td style={{ padding: '10px 12px' }}>
+                      {project ? (
+                        <div style={{ display: 'flex', alignItems: 'center', gap: '6px' }}>
+                          <div style={{
+                            width: '6px', height: '6px', borderRadius: '50%', flexShrink: 0,
+                            background: project.colour ?? 'var(--accent)',
+                          }} />
+                          <span className="text-xs text-muted truncate" style={{ maxWidth: '90px' }}>
+                            {project.name}
+                          </span>
+                        </div>
+                      ) : (
+                        <span className="text-xs text-muted">—</span>
+                      )}
+                    </td>
                    <td className="text-xs text-muted" style={{ textAlign: 'right', padding: '10px 0' }}>
                      {formatTimestamp(session.updated_at)}
                    </td>
@@ -171,7 +189,7 @@ export default function AllChatsView({ onSelectSession, onBack }) {

              {sessions.length === 0 && (
                <tr>
-                  <td colSpan={3} className="text-base text-muted"
+                  <td colSpan={4} className="text-base text-muted"
                    style={{ textAlign: 'center', padding: '40px' }}>
                    No conversations yet
                  </td>
--- a/packages/chat-client/src/components/Sidebar.jsx
+++ b/packages/chat-client/src/components/Sidebar.jsx
@@ -45,11 +45,6 @@ export default function Sidebar({
    }
  }

-  function getPreview(session) {
-    if (session.isNew) return 'New conversation';
-    return session.name || session.external_id;
-  }
-
  // ── Collapsed rail ───────────────────────────────────────

  if (!isOpen) {
@@ -96,6 +91,30 @@ export default function Sidebar({

  const recentSessions = sessions.slice(0, 10);

+  // Group recent sessions by project
+  const grouped = {};
+  const unassigned = [];
+  for (const session of recentSessions) {
+    if (session.project_id) {
+      if (!grouped[session.project_id]) grouped[session.project_id] = [];
+      grouped[session.project_id].push(session);
+    } else {
+      unassigned.push(session);
+    }
+  }
+
+  const sessionRowProps = (session) => ({
+    key: session.external_id,
+    session,
+    isActive: activeSession?.external_id === session.external_id,
+    isHovered: hoveredId === session.external_id,
+    onHover: setHoveredId,
+    onSelect: () => { onSelectSession(session); onNavigate('chat'); },
+    onRename: () => { setModalMode('settings'); setModalSession(session); },
+    onDelete: () => { setModalMode('confirm-delete'); setModalSession(session); },
+    onContextMenu: e => !session.isNew && openMenu(e, session),
+  });
+
  return (
    <>
      <div className="flex-col" style={{
@@ -141,45 +160,45 @@ export default function Sidebar({
            isOpen={projectsOpen}
            onToggle={() => setProjectsOpen(o => !o)}
          />
-{projectsOpen && (
-  <div style={{ padding: '4px 10px 8px' }}>
-    {!projects?.length  ? (
-      <div style={{
-        padding: '10px',
-        borderRadius: 'var(--radius-md)',
-        border: '1px dashed var(--border)',
-        color: 'var(--text-muted)',
-        fontSize: '12px',
-        textAlign: 'center',
-      }}>
-        No projects yet
-      </div>
-    ) : (
-      <div style={{ display: 'flex', flexWrap: 'wrap', gap: '6px' }}>
-        {projects.slice(0, 6).map(project => (
-          <button
-            key={project.id}
-            onClick={() => {onSelectProject(project); onNavigate('project')}}
-            className="btn-reset text-xs"
-            style={{
-              padding: '4px 8px',
-              borderRadius: 'var(--radius-sm)',
-              background: 'var(--bg-elevated)',
-              border: `1px solid ${project.colour ?? 'var(--border)'}`,
-              color: 'var(--text-secondary)',
-              maxWidth: '100%',
-            }}
-            title={project.description ?? project.name}
-          >
-            <span className="truncate" style={{ display: 'block', maxWidth: '140px' }}>
-              {project.name}
-            </span>
-          </button>
-        ))}
-      </div>
-    )}
-  </div>
-)}
+          {projectsOpen && (
+            <div style={{ padding: '4px 10px 8px' }}>
+              {!projects?.length ? (
+                <div style={{
+                  padding: '10px',
+                  borderRadius: 'var(--radius-md)',
+                  border: '1px dashed var(--border)',
+                  color: 'var(--text-muted)',
+                  fontSize: '12px',
+                  textAlign: 'center',
+                }}>
+                  No projects yet
+                </div>
+              ) : (
+                <div style={{ display: 'flex', flexWrap: 'wrap', gap: '6px' }}>
+                  {projects.slice(0, 6).map(project => (
+                    <button
+                      key={project.id}
+                      onClick={() => { onSelectProject(project); onNavigate('project'); }}
+                      className="btn-reset text-xs"
+                      style={{
+                        padding: '4px 8px',
+                        borderRadius: 'var(--radius-sm)',
+                        background: 'var(--bg-elevated)',
+                        border: `1px solid ${project.colour ?? 'var(--border)'}`,
+                        color: 'var(--text-secondary)',
+                        maxWidth: '100%',
+                      }}
+                      title={project.description ?? project.name}
+                    >
+                      <span className="truncate" style={{ display: 'block', maxWidth: '140px' }}>
+                        {project.name}
+                      </span>
+                    </button>
+                  ))}
+                </div>
+              )}
+            </div>
+          )}

          <div style={{ height: '1px', background: 'var(--border)', margin: '2px 0' }} />

@@ -189,28 +208,54 @@ export default function Sidebar({
            isOpen={chatsOpen}
            onToggle={() => setChatsOpen(o => !o)}
          />
+
          {chatsOpen && (
            <>
-              {recentSessions.map(session => (
-                <SessionRow
-                  key={session.external_id}
-                  session={session}
-                  isActive={activeSession?.external_id === session.external_id}
-                  isHovered={hoveredId === session.external_id}
-                  onHover={setHoveredId}
-                  onSelect={() => { onSelectSession(session); onNavigate('chat'); }}
-                  onRename={() => { setModalMode('settings'); setModalSession(session); }}
-                  onDelete={() => { setModalMode('confirm-delete'); setModalSession(session); }}
-                  onContextMenu={e => !session.isNew && openMenu(e, session)}
-                />
-              ))}
-
              {recentSessions.length === 0 && (
                <div className="text-xs text-muted" style={{ padding: '12px 16px', textAlign: 'center' }}>
                  No conversations yet
                </div>
              )}

+              {/* Project groups */}
+              {Object.entries(grouped).map(([projectId, projectSessions]) => {
+                const project = projects?.find(p => p.id === Number(projectId));
+                return (
+                  <div key={projectId}>
+                    {/* Project group label */}
+                    <div style={{
+                      display: 'flex', alignItems: 'center', gap: '6px',
+                      padding: '6px 16px 2px',
+                    }}>
+                      <div style={{
+                        width: '6px', height: '6px', borderRadius: '50%', flexShrink: 0,
+                        background: project?.colour ?? 'var(--accent)',
+                      }} />
+                      <span className="text-xs text-muted truncate">
+                        {project?.name ?? 'Project'}
+                      </span>
+                    </div>
+                    {projectSessions.map(session => (
+                      <SessionRow {...sessionRowProps(session)} />
+                    ))}
+                  </div>
+                );
+              })}
+
+              {/* Unassigned sessions */}
+              {unassigned.length > 0 && (
+                <>
+                  {Object.keys(grouped).length > 0 && (
+                    <div style={{ padding: '6px 16px 2px' }}>
+                      <span className="text-xs text-muted">Other</span>
+                    </div>
+                  )}
+                  {unassigned.map(session => (
+                    <SessionRow {...sessionRowProps(session)} />
+                  ))}
+                </>
+              )}
+
              {sessions.length > 0 && (
                <button
                  onClick={() => onNavigate('all-chats')}
--- a/packages/orchestration-service/src/services/qdrant.js
+++ b/packages/orchestration-service/src/services/qdrant.js
@@ -14,10 +14,6 @@ async function searchEpisodes( vector, {limit = ORCHESTRATION.RECENT_EPISODE_LIM
    } else if (sessionId) {
        body.filter = { must: [{key: 'sessionId', match: {value: sessionId} }] };
    }
-
-    console.log('[qdrant] searchEpisodes filter:', JSON.stringify(body.filter));
-    console.log('[qdrant] projectSessionIds:', projectSessionIds);
-
    const res = await fetch (
        `${BASE_URL}/collections/${COLLECTIONS.EPISODES}/points/search`,
        {