@@ -0,0 +1,447 @@
1+# LangChain RAG Integration Plan
2+
3+## Overview
4+
5+This plan outlines the steps to integrate LangChain for implementing a Retrieval-Augmented Generation (RAG) system in the personal assistant application. The RAG system will use `data.txt` as the knowledge base.
6+
7+## Architecture Decision
8+
9+**Approach**: Integrate RAG directly into the chat generation flow. For each user message, automatically retrieve relevant context from the knowledge base and inject it into the conversation context before generation. This ensures the LLM always has access to relevant information from `data.txt` without requiring explicit tool calls.
10+
11+## Implementation Steps
12+
13+### 1. Install Dependencies
14+
15+Add the following LangChain packages:
16+
17+- `langchain` - Core LangChain library
18+- `@langchain/ollama` - Ollama integration for embeddings and LLM
19+- `@langchain/community` - Community integrations (vector stores, document loaders)
20+- `langchain-text-splitters` - Text splitting utilities
21+- `pgvector` extension for PostgreSQL (requires DB migration)
22+- `tsx` (dev dependency) - For running TypeScript scripts directly
23+
24+### 2. Create RAG Module
25+
26+**File**: `src/lib/rag.ts`
27+
28+- Define constants: `EMBEDDING_MODEL`, `CHUNK_COUNT`
29+- Create embeddings function
30+- Create functions to store documents and perform similarity search
31+- Create `retrieveRelevantContext` function with caching
32+- All RAG functionality in one file
33+
34+**Code snippet:**
35+
36+```typescript
37+import { db } from "@/src/lib/db";
38+import { documentChunks } from "@/src/lib/db/schema";
39+import { OllamaEmbeddings } from "@langchain/ollama";
40+import { Document } from "@langchain/core/documents";
41+import { cosineDistance, desc, sql } from "drizzle-orm";
42+import { getCache, setCache } from "@/src/lib/cache";
43+
44+const EMBEDDING_MODEL = "nomic-embed-text";
45+const CHUNK_COUNT = 4;
46+const CACHE_TTL_MS = 60 * 60 * 1000; // 1 hour
47+
48+function createEmbeddings(): OllamaEmbeddings {
49+ return new OllamaEmbeddings({
50+ model: EMBEDDING_MODEL,
51+ });
52+}
53+
54+export async function storeDocuments(
55+ documents: Document[],
56+ embeddings: OllamaEmbeddings
57+): Promise<void> {
58+ // Generate embeddings for all chunks
59+ const texts = documents.map((doc) => doc.pageContent);
60+ const embeddingsList = await embeddings.embedDocuments(texts);
61+
62+ // Store in database (Drizzle handles vector type conversion automatically)
63+ for (let i = 0; i < documents.length; i++) {
64+ await db.insert(documentChunks).values({
65+ content: documents[i].pageContent,
66+ embedding: embeddingsList[i],
67+ metadata: documents[i].metadata,
68+ });
69+ }
70+}
71+
72+async function similaritySearch(
73+ queryEmbedding: number[],
74+ limit: number
75+): Promise<Array<{ content: string; metadata: unknown; similarity: number }>> {
76+ const similarity = sql<number>`1 - (${cosineDistance(documentChunks.embedding, queryEmbedding)})`;
77+
78+ const results = await db
79+ .select({
80+ content: documentChunks.content,
81+ metadata: documentChunks.metadata,
82+ similarity,
83+ })
84+ .from(documentChunks)
85+ .orderBy(desc(similarity))
86+ .limit(limit);
87+
88+ return results;
89+}
90+
91+export async function retrieveRelevantContext(query: string): Promise<string | null> {
92+ // Check cache first
93+ const cacheKey = `rag:${query.toLowerCase().trim()}`;
94+ const cached = await getCache<string>(cacheKey);
95+ if (cached !== null) {
96+ return cached;
97+ }
98+
99+ try {
100+ const embeddings = createEmbeddings();
101+ const queryEmbedding = await embeddings.embedQuery(query);
102+
103+ const results = await similaritySearch(queryEmbedding, CHUNK_COUNT);
104+
105+ if (results.length === 0) {
106+ return null;
107+ }
108+
109+ // Format chunks into context string
110+ const context = results
111+ .map((result, index) => `[Chunk ${index + 1}]\n${result.content}`)
112+ .join("\n\n");
113+
114+ // Cache the result
115+ await setCache(cacheKey, context, CACHE_TTL_MS);
116+
117+ return context;
118+ } catch (error) {
119+ console.error("RAG retrieval error:", error);
120+ return null;
121+ }
122+}
123+```
124+
125+### 3. Integration into Chat Flow
126+
127+**File**: `src/lib/chat.ts`
128+
129+- Modify `generateResponse` function to:
130+ - Extract the latest user message from message history (last message with role "user")
131+ - Call `retrieveRelevantContext` with the user message content
132+ - If context is retrieved, inject it into the message history:
133+ - Find the position after all system messages but before user messages
134+ - Insert a system message with formatted RAG context at that position
135+ - Format: `**Relevant Context from Knowledge Base:**\n\n[formatted chunks]\n\nUse this information to help answer the user's question.`
136+ - Pass enhanced message history to `agentLoop`
137+- Ensure RAG context is only injected when relevant chunks are found
138+- Handle errors gracefully - if RAG retrieval fails, continue without context
139+- Note: RAG context is retrieved fresh for each generation, based on the latest user message
140+
141+**Code snippet for `src/lib/chat.ts` modification:**
142+
143+```typescript
144+import { retrieveRelevantContext } from "@/src/lib/rag";
145+
146+export async function generateResponse(
147+ threadId: number,
148+ enqueue: (message: MessageHistoryEntry) => void
149+) {
150+ if (!threadId) {
151+ throw new Error("Thread ID is required");
152+ }
153+
154+ // Get the thread:
155+ const thread = await db.select().from(threads).where(eq(threads.id, threadId)).limit(1);
156+ if (thread.length === 0) {
157+ throw new Error("Thread not found");
158+ }
159+
160+ // Get all of the messages for this thread:
161+ const messagesList = (await db
162+ .select({
163+ role: messages.role,
164+ content: messages.content,
165+ thinking: messages.thinking,
166+ })
167+ .from(messages)
168+ .where(eq(messages.threadId, threadId))
169+ .orderBy(asc(messages.createdAt))) as MessageHistory;
170+
171+ if (messagesList.length === 0) {
172+ throw new Error("No messages found for this thread");
173+ }
174+
175+ // Extract latest user message
176+ const latestUserMessage = messagesList.filter((msg) => msg.role === "user").pop();
177+
178+ // Retrieve RAG context if user message exists
179+ let ragContext: string | null = null;
180+ if (latestUserMessage) {
181+ try {
182+ ragContext = await retrieveRelevantContext(latestUserMessage.content);
183+ } catch (error) {
184+ console.error("Failed to retrieve RAG context:", error);
185+ // Continue without RAG context
186+ }
187+ }
188+
189+ // Inject RAG context into message history
190+ let enhancedMessageHistory = [...messagesList];
191+ if (ragContext) {
192+ // Find position after all system messages
193+ const lastSystemIndex =
194+ enhancedMessageHistory
195+ .map((msg, idx) => (msg.role === "system" ? idx : -1))
196+ .filter((idx) => idx !== -1)
197+ .pop() ?? -1;
198+
199+ const ragSystemMessage: MessageHistoryEntry = {
200+ role: "system",
201+ content: `**Relevant Context from Knowledge Base:**\n\n${ragContext}\n\nUse this information to help answer the user's question.`,
202+ };
203+
204+ enhancedMessageHistory.splice(lastSystemIndex + 1, 0, ragSystemMessage);
205+ }
206+
207+ const messageHistory = await agentLoop(thread[0].model, enhancedMessageHistory, enqueue);
208+
209+ // Update the thread with the new message history:
210+ for (const message of messageHistory) {
211+ await createMessage(message.content, message.role, threadId, message.thinking);
212+ }
213+}
214+```
215+
216+### 4. Vector Store Initialization
217+
218+#### 4.1 Initialization Strategy
219+
220+- Initialization is done via an npm script, not automatically
221+- Chunking, embedding generation, and vector DB updates happen when the script is run
222+- Run the script manually after updating `data.txt` or as part of deployment
223+
224+#### 4.2 Initialization Script
225+
226+**File**: `scripts/update-vector-data.ts`
227+
228+- Load `data.txt` file directly
229+- Split document into chunks using RecursiveCharacterTextSplitter (define `CHUNK_SIZE` and `CHUNK_OVERLAP` constants)
230+- Generate embeddings for all chunks
231+- Store all chunks with embeddings in vector database
232+- Handle errors gracefully
233+- Optionally clear existing data before re-indexing (add `--force` flag)
234+
235+**Code snippet:**
236+
237+```typescript
238+import { readFile } from "fs/promises";
239+import { join } from "path";
240+import { Document } from "@langchain/core/documents";
241+import { RecursiveCharacterTextSplitter } from "langchain-text-splitters";
242+import { OllamaEmbeddings } from "@langchain/ollama";
243+import { storeDocuments } from "@/src/lib/rag";
244+import { db } from "@/src/lib/db";
245+import { documentChunks } from "@/src/lib/db/schema";
246+import { count, eq } from "drizzle-orm";
247+
248+const DATA_FILE_PATH = "./data.txt";
249+const CHUNK_SIZE = 1000;
250+const CHUNK_OVERLAP = 200;
251+
252+async function updateVectorData(force: boolean = false): Promise<void> {
253+ try {
254+ // Check if documents are already indexed
255+ const existingCount = await db.select({ count: count() }).from(documentChunks);
256+ if (existingCount[0].count > 0 && !force) {
257+ console.log("RAG vector store already initialized. Use --force to re-index.");
258+ return;
259+ }
260+
261+ if (force && existingCount[0].count > 0) {
262+ console.log("Clearing existing vector data...");
263+ await db.delete(documentChunks);
264+ }
265+
266+ console.log("Loading and chunking data.txt...");
267+
268+ // Load data.txt file
269+ const filePath = join(process.cwd(), DATA_FILE_PATH);
270+ const content = await readFile(filePath, "utf-8");
271+ const document = new Document({
272+ pageContent: content,
273+ metadata: { source: DATA_FILE_PATH },
274+ });
275+
276+ // Split document into chunks
277+ const textSplitter = new RecursiveCharacterTextSplitter({
278+ chunkSize: CHUNK_SIZE,
279+ chunkOverlap: CHUNK_OVERLAP,
280+ });
281+ const chunks = await textSplitter.splitDocuments([document]);
282+
283+ console.log(`Generated ${chunks.length} chunks, generating embeddings...`);
284+
285+ // Generate embeddings for all chunks
286+ const embeddings = new OllamaEmbeddings({ model: "nomic-embed-text" });
287+ await storeDocuments(chunks, embeddings);
288+
289+ console.log("Vector data update complete!");
290+ } catch (error) {
291+ console.error("Failed to update vector data:", error);
292+ process.exit(1);
293+ }
294+}
295+
296+// Parse command line arguments
297+const force = process.argv.includes("--force");
298+updateVectorData(force);
299+```
300+
301+**NPM Script Entry:**
302+
303+Add to `package.json`:
304+
305+```json
306+{
307+ "scripts": {
308+ "rag:update": "tsx scripts/update-vector-data.ts",
309+ "rag:update:force": "tsx scripts/update-vector-data.ts --force"
310+ }
311+}
312+```
313+
314+**Note**: Requires `tsx` package for running TypeScript files directly. Install with `npm install --save-dev tsx`.
315+
316+### 5. Database Schema
317+
318+- Add migration to enable pgvector extension
319+- Create `document_chunks` table with:
320+ - `id` (serial)
321+ - `content` (text)
322+ - `embedding` (vector type) - using pgvector
323+ - `metadata` (jsonb) - store chunk index, source file, etc.
324+ - `created_at` (timestamp)
325+- Add Drizzle schema definition in `src/lib/db/schema.ts`
326+- Create index on embedding column for similarity search performance
327+
328+**Migration Steps:**
329+
330+1. Create the pgvector extension manually (Drizzle doesn't create extensions automatically):
331+ - Use `npx drizzle-kit generate --custom` to create an empty migration file
332+ - Add: `CREATE EXTENSION IF NOT EXISTS vector;`
333+
334+2. Generate the table migration:
335+ - Run `npx drizzle-kit generate` to create migration for the `document_chunks` table
336+ - Drizzle will automatically generate the table and HNSW index based on the schema definition
337+
338+**Example migration SQL (auto-generated by drizzle-kit):**
339+
340+```sql
341+CREATE TABLE IF NOT EXISTS "document_chunks" (
342+ "id" serial PRIMARY KEY NOT NULL,
343+ "content" text NOT NULL,
344+ "embedding" vector(768),
345+ "metadata" jsonb,
346+ "created_at" timestamp with time zone DEFAULT now() NOT NULL
347+);
348+--> statement-breakpoint
349+CREATE INDEX IF NOT EXISTS "embeddingIndex" ON "document_chunks" USING hnsw (embedding vector_cosine_ops);
350+```
351+
352+**Code snippet for `src/lib/db/schema.ts`:**
353+
354+```typescript
355+import { pgTable, serial, text, timestamp, jsonb, vector, index } from "drizzle-orm/pg-core";
356+
357+export const documentChunks = pgTable(
358+ "document_chunks",
359+ {
360+ id: serial("id").primaryKey(),
361+ content: text("content").notNull(),
362+ embedding: vector("embedding", { dimensions: 768 }), // Adjust dimensions based on embedding model
363+ metadata: jsonb("metadata"),
364+ createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
365+ },
366+ (table) => [index("embeddingIndex").using("hnsw", table.embedding.op("vector_cosine_ops"))]
367+);
368+```
369+
370+**Note**: The pgvector extension must be created manually in a migration. Use `npx drizzle-kit generate --custom` to create an empty migration file and add `CREATE EXTENSION IF NOT EXISTS vector;`
371+
372+### 6. Error Handling
373+
374+- Handle Ollama connection errors
375+- Handle embedding generation failures
376+- Provide fallback behavior when RAG is unavailable
377+
378+## File Structure
379+
380+```
381+src/lib/
382+├── rag/
383+│ └── index.ts # All RAG functionality (embeddings, vector store, retrieval)
384+└── chat.ts # (update to integrate RAG)
385+scripts/
386+└── update-vector-data.ts # Script to update vector database (chunking, embedding, storage)
387+```
388+
389+## Implementation Order
390+
391+1. **Phase 1: Setup**
392+ - Install dependencies
393+ - Create RAG directory structure
394+ - Set up database schema and migrations
395+
396+2. **Phase 2: Core RAG Components**
397+ - Create unified RAG module (embeddings, vector store, retrieval)
398+ - Create initialization script (chunking, embedding, storage)
399+ - Add npm script to run initialization
400+
401+3. **Phase 3: Chat Integration**
402+ - Modify `generateResponse` to retrieve context
403+ - Inject context into message history
404+ - Test integration
405+
406+4. **Phase 4: Optimization**
407+ - Error handling
408+ - Performance tuning
409+
410+5. **Phase 5: Enhancement (Optional)**
411+ - Add file watching for `data.txt` updates
412+ - Add re-indexing endpoint/functionality
413+ - Add monitoring/logging
414+
415+## Considerations
416+
417+### Performance
418+
419+- Embedding generation happens when running the update script (can be slow)
420+- Vector store is populated manually via npm script, not automatically
421+- Cache retrieval results by query to avoid redundant searches
422+- Script checks if vector store is already populated to skip re-indexing (unless `--force` flag is used)
423+
424+### Context Injection Strategy
425+
426+- Inject as additional system message with clear formatting
427+ - Format: "**Relevant Context from Knowledge Base:**\n[chunks]"
428+ - Pros: Clear separation, easy to identify in logs
429+ - Cons: May be treated differently by model
430+
431+### Scalability
432+
433+- pgvector with PostgreSQL handles larger datasets well
434+- Consider chunking strategy for very large files
435+- Monitor query performance and adjust `CHUNK_COUNT` constant as needed
436+
437+### Model Selection
438+
439+- Use an embedding model compatible with Ollama
440+- `nomic-embed-text` is a good default choice
441+- Ensure embedding model is available locally
442+
443+### Data Updates
444+
445+- `data.txt` is loaded and indexed via the `npm run rag:update` script
446+- To update the knowledge base: modify `data.txt`, then run `npm run rag:update:force` to re-index
447+- The script can be run manually or as part of deployment/CI pipeline