Q: Context limit?
I tried many AI assistants and all ended up being a disappointment due to the same issue: the limitations of the context window of the LLM. I can add tens of thousands of pages of user content, but most of it will be ignored as it doesn't fit in the context window, and answers will only be based on a tiny fraction of selected content from the documents. So IMO today's LLMs, while a feasible solution to get data from a small amount of user documents, are not the right tool to analyze very large collections of user data. Sadly, a simple full text search (30+ year old technology) of the documents gets more complete results than AI assistants.
I am waiting for the AI assistant that somehow manages to overcome this limitation. Did you manage to solve this, or this context limitation of LLMs also affects IKI.AI the same as other AI Assistants?
eb9b0ca94fe442e1b6f97d103b66f3beThanks - I have a basic understanding of RAG and that's also the context of my question.
The chunks sent to the LLM have to fit into the context window, e.g. if there are 10,000 results, you still can't embed 10,000 chunks with the prompt, only the top hits. So the response will be incomplete - based on those top hits only - and the results depend on what the retriever chooses to send to the LLM.
Ivan_IKIAIYou never serve that amount of context to an LLM, it's just not working well that way if you expect a fast and precise answer.
Yes, that's what I was afraid of... my use case is based on a TON of data, so until the technology gets there, sadly, a full text search in a text file is an ancient, but better option. AI will get there, but it's not yet the right solution for large collections of documents. Expectation management is important.