Q: Why doesn't Docxter get the same answers as native Claude?
I uploaded an 8-page pdf tender document and asked DocXter multiple times with multiple LLMs
how many items there are. DocXter never got the answer right but native free Claude got it right the first time. I tried the same with other longer docs and the performance was worse. What am I missing here?
Adeeb_DocXter
Oct 18, 2024A: Hi @konradd,
DocXter is built to interact and get actionable intelligence from your knowledge. It solely takes into spotlight, the knowledge (docs) that you share.
Now, with Claude, it's trained on huge data sets, so no matter what the question, it'll never say 'No' to an answer.
All I'd say is, that DocXter's built for specific purposes, while Claude is generic for many use cases.
112341464655977068817Hello, I did a positive review to Docxter but have not noticed this issue. I want to check if I can reproduce it, as reliability is important for me. What do you mean for "items"?
konrad98Hi. I tested 2 construction tender documents. One had 8 pages with 13 main items to be priced on. Docxter could only retrieve 4 or 5 items after several attempts. The other was over a hundred pages and I asked it to give me all the companies involved with the contact details of each company. It could only produce between 3 and 5 out of the 10 companies listed. Socartes, and Claude got all.
Hi Adeeb, I did not ask Docxter anything that was not in the document, the information retrieval test I did was actually either as heading in the doc or from a bullet list. No outside information was needed at all. I did around 30 tests with all LLMs, OCR and even coverted the pdf to text and docx files, but i could not retreive the data that native Claude, Socrates, Straico and ChatGPT got right
I understand what you meant now. The difference in behavior is likely due to how they handle context. The Anthropic interface seems to pass the full conversation or document as context, while DocXter probably uses a summary context. This allows for longer conversations. The founders should clarify how the context is managed.