teycirQ: Is it possible for you to store Dolphin Mixtral or other free open source LLM on a VPS and offer its access for free as you do for GPT 3.
5 turbo? This would allow to reduce the high burn rate of coins. We basically can use a more intelligent LLM than GPT 3.5 for some tasks and keep the coins for expensive paying models like GPT 4 TURBO.
Arturo_Straico
May 14, 2024A: Hello teycir! đ
Thanks for your insightful suggestion! Integrating more free, open-source LLMs like Dolphin Mixtral onto a VPS is certainly an interesting idea. Although itâs not part of our current setup, it's something we'll consider as our infrastructure grows and adapts. đ ď¸
Our current approach relies on providers like OpenAI and OpenRouter to access and deploy LLMs, including open-source models. This strategy allows us to ensure reliability and seamless integration. However, as our infrastructure grows and matures, we'll definitely explore the feasibility of hosting our own LLMs to provide more options for our users.
Moreover, we are actively considering including models like Claude 3 Haiku as free of coin charge in our offerings. This aligns with our ongoing efforts to deliver the best possible value to our community.
Best regards
The VPS strategy should be considered. As the open router is a third party and you do not have full control. In order to preserve this service, a backup on VPS at least for the fully free open source models is an assurance of the continuity of the service in case OpenRouter has a problem.
Hosting open-source LLMs on our own VPS is an idea we're considering as we scale our infrastructure. It could provide a backup option and ensure service continuity.
For now, join our Discord to stay updated and share feedback directly with the team. We prioritize user needs there.
Your suggestions are appreciated as we continue improving Straico's capabilities and cost-effectiveness.