AI is not that accurate, getting many errors
I recently tried using GraphDev, a toolkit designed to enhance GraphQL development, primarily drawn by its promise of AI-powered request generation. Unfortunately, my experience with the AI feature was less than satisfactory. Despite the potential benefits and capabilities advertised, the AI consistently produced errors in generating GraphQL requests.
While GraphDev offers a range of other useful tools, such as debugging and team collaboration features, the AI capability was the main reason for my purchase. I found that these AI-generated requests frequently led to errors, which significantly hindered my workflow. It seems that the AI struggles with understanding complex schemas, leading to failed queries.
However, I can see potential in GraphDev, and with further development, particularly in enhancing the AI's accuracy and reliability, it could become a very powerful tool. For now, I would recommend it with caution to those who are considering it primarily for its AI features.
In conclusion, if you're intrigued by the other functions GraphDev offers, it might still be worthwhile. But if you are focused on leveraging its AI for GraphQL requests, be prepared for a tool that requires additional refinement.
Warren_GraphDev
Sep 4, 2024Thank you for the detailed review. @digitalstemcell
We're working hard on improving the AI features and still have the feature in Beta for the reasons you're describing.
Hallucinations are still something we're working towards improving and plan to do so with the following updates:
1. When the AI generates a query, we will execute it against your schema to validate if the result is correct. If not, we will inform the AI to produce a new results, repeating until the result is accurate.
2. We are still experimenting with different models. Currently testing OpenAI 4o, Anthropic's Claud 3.5 and Gemini Flash. It's about finding a balance between speed, context length and cost.
We will move the AI feature out of beta only once these issues are addressed.
If you have any other queries or concerns please comment back, or reach out directly through the support panel on GraphDev.
All the best!
Warren