Why even bother chunking?
Well, long documents might contain the info you need - but they probably also contain a lot of cruft.
Chunking lets you make individual parts of documents searchable. Pretty important for any RAG app.

@SrinivasSi78619 Let me ask some dumb questions:
Are you processing the embeddings in parallel or sequentially?
Can you work with a subset of the data in dev?
Are you processing the embeddings in parallel or sequentially?
Can you work with a subset of the data in dev?
Generated by Thread Navigator
Press ⌘ + S to quick-export
