Name represents the flavor name, which will be used in model claim.

Limits defines the required accelerators to serve the model for each replica, like <nvidia.com/gpu: 8>. For multi-hosts cases, the limits here indicates the resource requirements for each replica, usually equals to the TP size. Not recommended to set the cpu and memory usage here:

• if using playground, you can define the cpu/mem usage at backendConfig.
• if using inference service, you can define the cpu/mem at the container resources. However, if you define the same accelerator resources at playground/service as well, the resources will be overwritten by the flavor limit here.

NodeSelector represents the node candidates for Pod placements, if a node doesn't meet the nodeSelector, it will be filtered out in the resourceFungibility scheduler plugin. If nodeSelector is empty, it means every node is a candidate.

Params stores other useful parameters and will be consumed by cluster-autoscaler / Karpenter for autoscaling or be defined as model parallelism parameters like TP or PP size. E.g. with autoscaling, when scaling up nodes with 8x Nvidia A00, the parameter can be injected with <INSTANCE-TYPE: p4d.24xlarge> for AWS. Preset parameters: TP, PP, INSTANCE-TYPE.

Flavors represents the accelerator requirements to serve the model. Flavors are fungible following the priority represented by the slice order.

Name refers to the model registry, such as huggingface.

ModelID refers to the model identifier on model hub, such as meta-llama/Meta-Llama-3-8B.

Filename refers to a specified model file rather than the whole repo. This is helpful to download a specified GGUF model rather than downloading the whole repo which includes all kinds of quantized models. TODO: this is only supported with Huggingface, add support for ModelScope in the near future. Note: once filename is set, allowPatterns and ignorePatterns should be left unset.

Revision refers to a Git revision id which can be a branch name, a tag, or a commit hash.

AllowPatterns refers to files matched with at least one pattern will be downloaded.

IgnorePatterns refers to files matched with any of the patterns will not be downloaded.

Name represents the model name.

Role represents the model role once more than one model is required. Such as a draft role, which means running with SpeculativeDecoding, and default arguments for backend will be searched in backendRuntime with the name of speculative-decoding.

ModelHub represents the model registry for model downloads.

URI represents a various kinds of model sources following the uri protocol, protocol://, e.g.

• oss://./
• ollama://llama3.3
• host://

FamilyName represents the model type, like llama2, which will be auto injected to the labels with the key of llmaz.io/model-family-name.

Source represents the source of the model, there're several ways to load the model such as loading from huggingface, OCI registry, s3, host path and so on.

InferenceConfig represents the inference configurations for the model.

Conditions represents the Inference condition.

BackendName represents the inference backend under the hood, e.g. vLLM.

Version represents the backend version if you want a different one from the default version.

Envs represents the environments set to the container.

ConfigName represents the recommended configuration name for the backend, It will be inferred from the models in the runtime if not specified, e.g. default, speculative-decoding.

Args defined here will "append" the args defined in the recommendedConfig, either explicitly configured in configName or inferred in the runtime.

Resources represents the resource requirements for backend, like cpu/mem, accelerators like GPU should not be defined here, but at the model flavors, or the values here will be overwritten. Resources defined here will "overwrite" the resources in the recommendedConfig.

SharedMemorySize represents the size of /dev/shm required in the runtime of inference workload. SharedMemorySize defined here will "overwrite" the sharedMemorySize in the recommendedConfig.

Command represents the default command for the backendRuntime.

Image represents the default image registry of the backendRuntime. It will work together with version to make up a real image.

Version represents the default version of the backendRuntime. It will be appended to the image as a tag.

Envs represents the environments set to the container.

Lifecycle represents hooks executed during the lifecycle of the container.

Periodic probe of backend liveness. Backend will be restarted if the probe fails. Cannot be updated.

Periodic probe of backend readiness. Backend will be removed from service endpoints if the probe fails.

StartupProbe indicates that the Backend has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the backend will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a backend's lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation.

RecommendedConfigs represents the recommended configurations for the backendRuntime.

Conditions represents the Inference condition.

MinReplicas indicates the minimum number of inference workloads based on the traffic. Default to 1. MinReplicas couldn't be 0 now, will support serverless in the future.

MaxReplicas indicates the maximum number of inference workloads based on the traffic. Default to nil means there's no limit for the instance number.

ScaleTrigger defines the rules to scale the workloads. Only one trigger cloud work at a time, mostly used in Playground. ScaleTrigger defined here will "overwrite" the scaleTrigger in the recommendedConfig.

metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used). The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods. Ergo, metrics used must decrease as the pod count is increased, and vice-versa. See the individual metric source types for more information about how each type of metric must respond.

behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used.

Replicas represents the replica number of inference workloads.

ModelClaim represents claiming for one model, it's a simplified use case of modelClaims. Most of the time, modelClaim is enough. ModelClaim and modelClaims are exclusive configured.

ModelClaims represents claiming for multiple models for more complicated use cases like speculative-decoding. ModelClaims and modelClaim are exclusive configured.

BackendRuntimeConfig represents the inference backendRuntime configuration under the hood, e.g. vLLM, which is the default backendRuntime.

ElasticConfig defines the configuration for elastic usage, e.g. the max/min replicas.

Conditions represents the Inference condition.

Replicas track the replicas that have been created, whether ready or not.

Selector points to the string form of a label selector which will be used by HPA.

Name represents the identifier of the config.

Args represents all the arguments for the command. Argument around with {{ .CONFIG }} is a configuration waiting for render.

Resources represents the resource requirements for backend, like cpu/mem, accelerators like GPU should not be defined here, but at the model flavors, or the values here will be overwritten.

SharedMemorySize represents the size of /dev/shm required in the runtime of inference workload.

ScaleTrigger defines the rules to scale the workloads. Only one trigger cloud work at a time.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

HPA represents the trigger configuration of the HorizontalPodAutoscaler.

ModelClaims represents multiple claims for different models.

Replicas represents the replica number of inference workloads.

WorkloadTemplate defines the template for leader/worker pods

RolloutStrategy defines the strategy that will be applied to update replicas when a revision is made to the leaderWorkerTemplate.

Conditions represents the Inference condition.

Replicas track the replicas that have been created, whether ready or not.

Selector points to the string form of a label selector, the HPA will be able to autoscale your resource.