People can quick deploy a LLM service with minimal configurations.

llmaz supports a wide range of advanced inference backends for different scenarios, like vLLM, Text-Generation-Inference, SGLang, llama.cpp. Find the full list of supported backends here.

llmaz supports serving the same LLM with various accelerators to optimize cost and performance.

llmaz supports a wide range of model providers, such as HuggingFace, ModelScope, ObjectStores. llmaz will automatically handle the model loading, requiring no effort from users.

llmaz supports both single-host and multi-host scenarios with LWS from day 0.

Offering capabilities like token-based rate limiting, model routing with the integration of Envoy AI Gateway.

Out-of-the-box chatbot support with the integration of Open WebUI, offering capacities like function call, RAG, web search and more, see configurations here.

llmaz supports horizontal scaling with HPA by default and will integrate with autoscaling components like Cluster-Autoscaler or Karpenter for smart scaling across different clouds.

Out-of-the-box model cache system support with Manta, still under development right now with architecture reframing.