The author explores the concept of shaping stories and its application in making RAG better. Chunking, a major issue in RAG, can be solved by understanding the shape of stories. The right size of a chunk is crucial to maintain specificity and context. By identifying jumps in the latent space, chunking can be done effectively. An API is being released for experimenting with this chunking strategy. A comparison between semantic chunking in LlamaIndex and the author’s chunking on Paul Graham’s essay shows that the author’s chunking results in cleaner topics. Manual chunking and comparison are encouraged.
https://gpt3experiments.substack.com/p/a-new-chunking-approach-to-rag