Link to GrowDB
0. Introduction
1. Algorithm Behind Question Generation (rough draft)
User data is chunked and stored as embeddings. Each chunks are nodes.
Automatic edges are ‘next’ and ‘prev’. There are also contextual edges. For each chunk, get its top 5 closest chunks. (could overlap with ‘next’ and ‘prev’) and they are passed to an llm to define an unstructured relationship (edge).
For question generation, random sets of chunks are drawn. Then question generator (llm agent) generates a new question with several factors. A:current time B:chunks and their edges C:examples of good questions (exploration, clarification, scenarios) D:What if we add edge A (relationship) to chunk B In total 5 unique question is generated per run as list of strings.
Duplicate detector (llm agent) runs OK25BM search to check if another chunk already answer the question.
Questions are embedded and collect top 5 chunks per questions. Question and RAG contexts are passed to answer agent (llm agent + web search) that generate best guess to help user label quickly.
Upcoming updates - Automatically label and upload if confident is above a threshold - User can view and edit any chunks in the database - Image uploads