Deployment & High Availability
Kubernetes packaging — image, storage, resources, health, scheduling, Service exposure — plus replicas and standalone.
These settings cover how KubeMQ is packaged and run on Kubernetes — the container
image, persistent storage, resource requests/limits, health probes, node scheduling, and
Service exposure — plus the high-availability controls (replicas and standalone).
Unlike the rest of this reference, these fields do not exist in the server's
config.yaml: they are typed on the KubemqCluster CRD and consumed by the operator,
which renders the StatefulSet, Services, and PersistentVolumeClaim. The pattern is therefore
inverted — the Helm/CRD path is the real one, and the Docker column is —
(Kubernetes-only) except where the operator translates a CRD field into a pod env var (shown
as · env VAR). On Docker single-node, the equivalent concerns are docker run flags
(-p, -v, --cpus / --memory) covered in the Kubernetes guide.
Kubernetes packaging
Container image
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| Image | string | europe-docker.pkg.dev/kubemq/images/kubemq:next | image reference | — (K8s-only) | spec.image.image | Resolution order: spec.image.image → operator env RELATED_IMAGE_KUBEMQ_CLUSTER → built-in fallback (config/image.go:8,18-30). No server env binding. |
| Pull policy | string (enum) | Always | IfNotPresent / Always / Never | — | spec.image.pullPolicy | CRD pattern (IfNotPresent|Always|Never); empty is coerced to Always (config/image.go:14,34-37). |
| Image pull secrets | string[] | [] | Secret names | — | — (chart value imagePullSecrets[]) | Not a typed CRD field — supplied via Helm values / a spec.statefulsetConfigData override, not the KubemqCluster spec. |
Storage (volume)
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| Volume size | string | unset → ephemeral (no PVC) | k8s quantity (e.g. 50Gi) | — | spec.volume.size | When set, the operator renders a volumeClaimTemplates PVC (ReadWriteOnce) mounted at ./kubemq/store. When unset/empty, the pod has no persistent volume and the store is ephemeral (deployment.go:78-81, deployment/statefulset.go:108-124). There is no built-in 10Gi default in code. Required for durable next-engine data — a next cluster with no spec.volume.size set raises the EphemeralNextStore warning (its durable data would otherwise live on ephemeral container storage). |
| Storage class | string | "" → cluster default | StorageClass name | — | spec.volume.storageClass | Only consulted when size is set; empty renders a blank storageClassName: → the cluster's default StorageClass (config/volume.go:8). |
spec.store.path on the next engine: the operator rejects a leading /. The
operator itself supplies the mount-rooted absolute path for the PVC, so a
spec.store.path value starting with / is rejected outright — the operator, not
the server, owns that absolute path. This is the inverse of the server-layer
behavior documented on Storage & Queues:
the next engine's server process honors an absolute StorePath verbatim (it's
the legacy engine that rewrites a leading / to ./). Keep the two facts distinct —
"the operator rejects a leading / in the CR field" and "the next-engine server
process honors an absolute StorePath" are both true, at different layers.
Resources
Each field is rendered into the pod's resources: block only when non-empty; there are no
defaults (config/resources.go:32-47).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| CPU limit | string | unset | k8s CPU quantity (e.g. 2, 500m) | — | spec.resources.limitsCpu | Pod resources.limits.cpu. |
| Memory limit | string | unset | k8s memory quantity (e.g. 2Gi) | — | spec.resources.limitsMemory | Pod resources.limits.memory. |
| Ephemeral-storage limit | string | unset | k8s quantity | — | spec.resources.limitsEphemeralStorage | Pod resources.limits.ephemeral-storage. |
| CPU request | string | unset | k8s CPU quantity | — | spec.resources.requestsCpu | Pod resources.requests.cpu. |
| Memory request | string | unset | k8s memory quantity | — | spec.resources.requestsMemory | Pod resources.requests.memory. |
| Ephemeral-storage request | string | unset | k8s quantity | — | spec.resources.requestsEphemeralStorage | Pod resources.requests.ephemeral-storage. |
Health probe
The liveness probe is off by default and only injected when enabled: true
(config/health.go:46-60).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| Enabled | bool | false | true / false | · env API_BIND_ADDRESS=0.0.0.0 (side-effect when true) | spec.health.enabled | When true, the operator adds a livenessProbe httpGet /health on the API port and sets API_BIND_ADDRESS=0.0.0.0 so the kubelet (pod IP) can reach the probe — the API binds 127.0.0.1 otherwise (config/health.go:48-60). |
| Initial delay (s) | int32 | 5 | any; ≤0 → 5 | — | spec.health.initialDelaySeconds | config/health.go:38-40. |
| Period (s) | int32 | 10 | any; ≤0 → 10 | — | spec.health.periodSeconds | config/health.go:35-37. |
| Timeout (s) | int32 | 5 | any; ≤0 → 5 | — | spec.health.timeoutSeconds | config/health.go:32-34. |
| Success threshold | int32 | 1 | any; ≤0 → 1 | — | spec.health.successThreshold | config/health.go:29-31. |
| Failure threshold | int32 | 12 | any; ≤0 → 12 | — | spec.health.failureThreshold | config/health.go:41-43. |
Node scheduling
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| Node selectors | map[string]string | {} | node label key/value pairs | — | spec.nodeSelectors.keys | Rendered as the pod nodeSelector; an empty map applies no constraint (config/node_selectors.go:9-24). |
Service exposure & interface toggles
The three built-in interface Services — gRPC (50000), REST/WebSocket (9090), and
API/dashboard (8080) — each expose a Service type, optional NodePort, custom port, and a
disable toggle. These HTTP/interface Services are opt-out via disabled: true.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| gRPC disabled | bool | false | true / false | · env CONNECTORS_GRPC_ENABLE=false | spec.grpc.disabled | Deletes the -grpc Service and disables the listener (config/grpc.go:64-66, deployment.go:203-206). |
| gRPC Service type | string (enum) | ClusterIP | ClusterIP / NodePort / LoadBalancer | — | spec.grpc.expose | Empty → ClusterIP (config/grpc.go:56); CRD pattern-validated. |
| gRPC NodePort | int32 | 0 (auto-assigned) | 30000–32767 | — | spec.grpc.nodePort | Applied only when expose: NodePort and value > 0 (config/grpc.go:77-81). |
| gRPC port | int32 | 50000 | port | · CONNECTORS_GRPC_PORT | spec.grpc.port | Moves the container/Service/target port and emits the env (config/grpc.go:71-75). |
| REST disabled | bool | false | true / false | · env CONNECTORS_REST_ENABLE=false | spec.rest.disabled | The -rest Service (9090) is shared by REST/MCP/Agents/CE — the operator removes it only when all four are disabled (deployment.go:99-120). |
| REST Service type | string (enum) | ClusterIP | ClusterIP / NodePort / LoadBalancer | — | spec.rest.expose | Empty → ClusterIP (config/rest.go:71). |
| REST NodePort | int32 | 0 (auto) | 30000–32767 | — | spec.rest.nodePort | Applied only when expose: NodePort and > 0 (config/rest.go:91-95). |
| REST port | int32 | 9090 | port | · CONNECTORS_REST_PORT | spec.rest.port | config/rest.go:85-88. |
| API disabled | bool | false | true / false | · env API_ENABLE=false | spec.api.disabled | Deletes the -api Service (config/api.go:65-66, deployment.go:89-92). |
| API Service type | string (enum) | ClusterIP | ClusterIP / NodePort / LoadBalancer | — | spec.api.expose | Empty → ClusterIP (config/api.go:58). |
| API NodePort | int32 | 0 (auto) | 30000–32767 | — | spec.api.nodePort | Applied only when expose: NodePort and > 0 (config/api.go:79-83). |
| API port | int32 | 8080 | port | · API_PORT | spec.api.port | config/api.go:73-76. |
Only the exposure subset of spec.grpc / spec.rest / spec.api lives here. Their
tuning fields (buffer/body limits, gRPC reflection, REST read/write timeouts, CORS, API
allow-origins, and the API-auth block) are documented on the
Interfaces and
Security pages.
Advanced / other top-level spec fields
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| StatefulSet override | string (YAML) | unset | StatefulSet YAML fragment | — | spec.statefulsetConfigData | Escape hatch — when set, replaces the operator-generated StatefulSet body with your fragment (deployment/statefulset.go:126-135,268-271). All other packaging fields above are ignored for the STS. |
| Config data (OIDC) | string | unset | OIDC block | — | spec.configData | OIDC-only passthrough — not a generic config.yaml. See Security. |
| License | string | unset | license key | — (Secret LICENSE_KEY_DATA) | spec.license | Injected as Secret data, not env (deployment.go:66-68). Config/secret-only. |
| Key | string | unset | activation key | — | spec.key | Top-level activation key; secret-only. |
| Env overlay | map[string]string | {} | env key/value pairs | — | spec.env | Last-wins overlay onto the pod ConfigMap — the general escape hatch for any server env key the CRD doesn't type. Operator-computed identity keys are rejected: STORE_ENGINE, CLUSTER_ENABLE, CLUSTER_NAME, CLUSTER_ROUTES, API_BIND_ADDRESS, CHECKSUM, POD_NAME, and any CLUSTER_REPLICATION_* key — not a CLUSTER_* wildcard (e.g. CLUSTER_PORT is still allowed). Not for secrets. |
| Env from Secrets | string[] | [] | existing Secret names | — | spec.envFromSecrets | Projects existing Secret(s) as pod env (envFrom) — the general Secret-envFrom escape hatch; Secret values never transit the operator. This is the mechanism behind the Kafka SASL-credentials note (there is no dedicated #sasl anchor). |
High availability
High availability on Kubernetes is multiple replicas managed by the operator — not the
Docker cluster.* block.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
|---|---|---|---|---|---|---|
| Replica count | *int32 | 3 | CRD Minimum=0; 0/nil coerced to 3 | — | spec.replicas | nil → 3 (deployment.go:59-62), and SetReplicas(0) → 3 (deployment/statefulset.go:183-191). Backs the scale subresource (specpath=.spec.replicas). More than one replica = clustered HA (operator wires CLUSTER_* + the 5228 cluster port). Once the engine is established as next, spec.replicas is immutable — a CEL rule plus an operator fail-closed guard reject any change (see the engine-establishment guard below). |
| Standalone | bool | false | true / false | — | spec.standalone | true → a single non-clustered node: the operator omits CLUSTER_NAME/CLUSTER_ROUTES/CLUSTER_ENABLE and the 5228 cluster-port from the pod (deployment/statefulset.go:40-47,103-107). On the next engine, standalone runs a loopback raft and skips replication wiring entirely. |
On Kubernetes, high availability is spec.replicas (operator-managed clustering) — not
the config.yaml-only cluster.* block, which has no Helm/CRD path. See the
Advanced reference for the Docker cluster.* path.
Connector Services & ports (operator-exposed). The pod always publishes container ports
for every wire connector — MQTT 1883/8883/8083, AMQP 5672/5671, STOMP 61613/61614,
AWS 4566, GCP 8085, Kafka 9092/9093 (deployment/statefulset.go:70-102,
deployment/service.go:192-256). The matching ClusterIP Service is created only when
that connector is enabled (spec.<connector>.enabled: true) — otherwise the operator
prunes it (deployment.go:122-199). Wire connectors are opt-in (enabled: true,
default off); the HTTP interfaces above (gRPC/REST/API) remain opt-out (disabled: true).
status.engine. The KubemqCluster status subresource exposes status.engine
(legacy / next) — the engine established for this cluster by the guard below. Read it
alongside status.replicas when auditing the replicas-freeze caveat above: once
status.engine reports next, spec.replicas can no longer change.
Engine-establishment guard
The operator records the live persistence engine in the
core.k8s.kubemq.io/established-engine annotation (legacy / next) — once present,
this annotation is authoritative, overriding spec.store.engine for every guard
decision. Derivation runs in order: (1) the annotation, if present; (2) else the explicit
spec.store.engine; (3) else the pod ConfigMap's STORE_ENGINE key (absent ⇒ legacy);
(4) else, if a StatefulSet or retained PVC already exists, the engine is unknown and
the operator refuses to guess — set spec.store.engine explicitly or delete the retained
PVCs; (5) else the cluster is genuinely fresh and the engine is inferred (Kafka enabled →
next, otherwise legacy).
The operator never changes a live cluster's engine. Two advisory CEL rules on the CRD
back this: spec.store.engine is immutable once set (born-one-mode), and spec.replicas
is immutable once the engine is established as next.
This operator-side inference is the pre-boot mirror of the server-side auto-select probe documented on Storage Engines and the Kafka callout on Connectors — two layers of one decision: the operator decides what the pod's env will say before the pod ever boots, and the server-side probe is what actually runs once it does.
Example
Publish the gRPC port on each target. On Kubernetes you set the Service exposure and node
port; on Docker you publish the port with -p. This is a single-setting snippet — see the
Kubernetes guide for complete, runnable configurations.
docker run -d \ --name kubemq \ -p 50000:50000 \ -e KUBEMQ_TOKEN=YOUR_LICENSE_KEY \ europe-docker.pkg.dev/kubemq/images/kubemq:nextgrpc:
expose: NodePort
nodePort: 32000For the full install flow (CRDs → operator → cluster), values.yaml mapped to the
KubemqCluster spec, and single-node vs HA, see the
Kubernetes guide.
Was this page helpful?