# Deployment & High Availability (/docs/configure/reference/deployment)
These settings cover how KubeMQ is **packaged and run on Kubernetes** — the container
image, persistent storage, resource requests/limits, health probes, node scheduling, and
`Service` exposure — plus the **high-availability** controls (`replicas` and `standalone`).
Unlike the rest of this reference, these fields do **not** exist in the server's
`config.yaml`: they are typed on the `KubemqCluster` **CRD** and consumed by the operator,
which renders the StatefulSet, Services, and PersistentVolumeClaim. The pattern is therefore
**inverted** — the **Helm/CRD path is the real one**, and the Docker column is `—`
(Kubernetes-only) except where the operator translates a CRD field into a pod env var (shown
as `· env VAR`). On Docker single-node, the equivalent concerns are `docker run` flags
(`-p`, `-v`, `--cpus` / `--memory`) covered in the [Kubernetes guide](/docs/configure/kubernetes).
## Kubernetes packaging [#kubernetes-packaging]
### Container image [#container-image]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------------ | ------------- | ------------------------------------------------- | ----------------------------------- | ---------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | string | `europe-docker.pkg.dev/kubemq/images/kubemq:next` | image reference | — (K8s-only) | `spec.image.image` | Resolution order: `spec.image.image` → operator env `RELATED_IMAGE_KUBEMQ_CLUSTER` → built-in fallback (`config/image.go:8,18-30`). No server env binding. |
| Pull policy | string (enum) | `Always` | `IfNotPresent` / `Always` / `Never` | — | `spec.image.pullPolicy` | CRD pattern `(IfNotPresent\|Always\|Never)`; empty is coerced to `Always` (`config/image.go:14,34-37`). |
| Image pull secrets | string\[] | `[]` | Secret names | — | — (chart value `imagePullSecrets[]`) | **Not** a typed CRD field — supplied via Helm values / a `spec.statefulsetConfigData` override, not the KubemqCluster spec. |
### Storage (volume) [#storage-volume]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------- | ------ | ---------------------------- | -------------------------- | ---------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Volume size | string | *unset → ephemeral (no PVC)* | k8s quantity (e.g. `50Gi`) | — | `spec.volume.size` | When set, the operator renders a `volumeClaimTemplates` PVC (`ReadWriteOnce`) mounted at `./kubemq/store`. When **unset/empty, the pod has no persistent volume** and the store is ephemeral (`deployment.go:78-81`, `deployment/statefulset.go:108-124`). There is **no built-in `10Gi` default** in code. **Required for durable `next`-engine data** — a `next` cluster with no `spec.volume.size` set raises the `EphemeralNextStore` warning (its durable data would otherwise live on ephemeral container storage). |
| Storage class | string | `""` → cluster default | StorageClass name | — | `spec.volume.storageClass` | Only consulted when `size` is set; empty renders a blank `storageClassName:` → the cluster's default StorageClass (`config/volume.go:8`). |
**`spec.store.path` on the `next` engine: the operator rejects a leading `/`.** The
operator itself supplies the mount-rooted absolute path for the PVC, so a
`spec.store.path` value starting with `/` is rejected outright — the operator, not
the server, owns that absolute path. This is the **inverse** of the server-layer
behavior documented on [Storage & Queues](/docs/configure/reference/storage-queues):
the `next` engine's **server process** honors an absolute `StorePath` verbatim (it's
the `legacy` engine that rewrites a leading `/` to `./`). Keep the two facts distinct —
"the operator rejects a leading `/` in the CR field" and "the next-engine server
process honors an absolute `StorePath`" are both true, at different layers.
### Resources [#resources]
Each field is rendered into the pod's `resources:` block only when non-empty; there are no
defaults (`config/resources.go:32-47`).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------------------- | ------ | ------- | ----------------------------------- | ---------------------------------- | ----------------------------------------- | ------------------------------------------- |
| CPU limit | string | unset | k8s CPU quantity (e.g. `2`, `500m`) | — | `spec.resources.limitsCpu` | Pod `resources.limits.cpu`. |
| Memory limit | string | unset | k8s memory quantity (e.g. `2Gi`) | — | `spec.resources.limitsMemory` | Pod `resources.limits.memory`. |
| Ephemeral-storage limit | string | unset | k8s quantity | — | `spec.resources.limitsEphemeralStorage` | Pod `resources.limits.ephemeral-storage`. |
| CPU request | string | unset | k8s CPU quantity | — | `spec.resources.requestsCpu` | Pod `resources.requests.cpu`. |
| Memory request | string | unset | k8s memory quantity | — | `spec.resources.requestsMemory` | Pod `resources.requests.memory`. |
| Ephemeral-storage request | string | unset | k8s quantity | — | `spec.resources.requestsEphemeralStorage` | Pod `resources.requests.ephemeral-storage`. |
### Health probe [#health-probe]
The liveness probe is **off by default** and only injected when `enabled: true`
(`config/health.go:46-60`).
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ----------------- | ----- | ------- | ---------------- | -------------------------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Enabled | bool | `false` | true / false | · env `API_BIND_ADDRESS=0.0.0.0` (side-effect when true) | `spec.health.enabled` | When true, the operator adds a `livenessProbe` httpGet `/health` on the API port **and** sets `API_BIND_ADDRESS=0.0.0.0` so the kubelet (pod IP) can reach the probe — the API binds `127.0.0.1` otherwise (`config/health.go:48-60`). |
| Initial delay (s) | int32 | `5` | any; `≤0` → `5` | — | `spec.health.initialDelaySeconds` | `config/health.go:38-40`. |
| Period (s) | int32 | `10` | any; `≤0` → `10` | — | `spec.health.periodSeconds` | `config/health.go:35-37`. |
| Timeout (s) | int32 | `5` | any; `≤0` → `5` | — | `spec.health.timeoutSeconds` | `config/health.go:32-34`. |
| Success threshold | int32 | `1` | any; `≤0` → `1` | — | `spec.health.successThreshold` | `config/health.go:29-31`. |
| Failure threshold | int32 | `12` | any; `≤0` → `12` | — | `spec.health.failureThreshold` | `config/health.go:41-43`. |
### Node scheduling [#node-scheduling]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| -------------- | ------------------ | ------- | -------------------------- | ---------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------- |
| Node selectors | map\[string]string | `{}` | node label key/value pairs | — | `spec.nodeSelectors.keys` | Rendered as the pod `nodeSelector`; an empty map applies no constraint (`config/node_selectors.go:9-24`). |
### Service exposure & interface toggles [#service-exposure--interface-toggles]
The three built-in interface Services — **gRPC** (`50000`), **REST/WebSocket** (`9090`), and
**API/dashboard** (`8080`) — each expose a Service type, optional NodePort, custom port, and a
disable toggle. These HTTP/interface Services are **opt-out** via `disabled: true`.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ----------------- | ------------- | ------------------- | ----------------------------------------- | ------------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| gRPC disabled | bool | `false` | true / false | · env `CONNECTORS_GRPC_ENABLE=false` | `spec.grpc.disabled` | Deletes the `-grpc` Service and disables the listener (`config/grpc.go:64-66`, `deployment.go:203-206`). |
| gRPC Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.grpc.expose` | Empty → `ClusterIP` (`config/grpc.go:56`); CRD pattern-validated. |
| gRPC NodePort | int32 | `0` (auto-assigned) | 30000–32767 | — | `spec.grpc.nodePort` | Applied only when `expose: NodePort` **and** value > 0 (`config/grpc.go:77-81`). |
| gRPC port | int32 | `50000` | port | · `CONNECTORS_GRPC_PORT` | `spec.grpc.port` | Moves the container/Service/target port and emits the env (`config/grpc.go:71-75`). |
| REST disabled | bool | `false` | true / false | · env `CONNECTORS_REST_ENABLE=false` | `spec.rest.disabled` | The `-rest` Service (9090) is **shared** by REST/MCP/Agents/CE — the operator removes it only when **all four** are disabled (`deployment.go:99-120`). |
| REST Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.rest.expose` | Empty → `ClusterIP` (`config/rest.go:71`). |
| REST NodePort | int32 | `0` (auto) | 30000–32767 | — | `spec.rest.nodePort` | Applied only when `expose: NodePort` and > 0 (`config/rest.go:91-95`). |
| REST port | int32 | `9090` | port | · `CONNECTORS_REST_PORT` | `spec.rest.port` | `config/rest.go:85-88`. |
| API disabled | bool | `false` | true / false | · env `API_ENABLE=false` | `spec.api.disabled` | Deletes the `-api` Service (`config/api.go:65-66`, `deployment.go:89-92`). |
| API Service type | string (enum) | `ClusterIP` | `ClusterIP` / `NodePort` / `LoadBalancer` | — | `spec.api.expose` | Empty → `ClusterIP` (`config/api.go:58`). |
| API NodePort | int32 | `0` (auto) | 30000–32767 | — | `spec.api.nodePort` | Applied only when `expose: NodePort` and > 0 (`config/api.go:79-83`). |
| API port | int32 | `8080` | port | · `API_PORT` | `spec.api.port` | `config/api.go:73-76`. |
Only the **exposure** subset of `spec.grpc` / `spec.rest` / `spec.api` lives here. Their
tuning fields (buffer/body limits, gRPC reflection, REST read/write timeouts, CORS, API
allow-origins, and the API-auth block) are documented on the
[Interfaces](/docs/configure/reference/interfaces) and
[Security](/docs/configure/reference/security) pages.
### Advanced / other top-level spec fields [#advanced--other-top-level-spec-fields]
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| -------------------- | ------------------ | ------- | ------------------------- | ---------------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| StatefulSet override | string (YAML) | unset | StatefulSet YAML fragment | — | `spec.statefulsetConfigData` | Escape hatch — when set, **replaces the operator-generated StatefulSet body** with your fragment (`deployment/statefulset.go:126-135,268-271`). All other packaging fields above are ignored for the STS. |
| Config data (OIDC) | string | unset | OIDC block | — | `spec.configData` | OIDC-only passthrough — **not** a generic `config.yaml`. See [Security](/docs/configure/reference/security). |
| License | string | unset | license key | — (Secret `LICENSE_KEY_DATA`) | `spec.license` | Injected as Secret data, not env (`deployment.go:66-68`). Config/secret-only. |
| Key | string | unset | activation key | — | `spec.key` | Top-level activation key; secret-only. |
| Env overlay | map\[string]string | `{}` | env key/value pairs | — | `spec.env` | Last-wins overlay onto the pod ConfigMap — the general escape hatch for any server env key the CRD doesn't type. Operator-computed identity keys are **rejected**: `STORE_ENGINE`, `CLUSTER_ENABLE`, `CLUSTER_NAME`, `CLUSTER_ROUTES`, `API_BIND_ADDRESS`, `CHECKSUM`, `POD_NAME`, and any `CLUSTER_REPLICATION_*` key — **not** a `CLUSTER_*` wildcard (e.g. `CLUSTER_PORT` is still allowed). Not for secrets. |
| Env from Secrets | string\[] | `[]` | existing Secret names | — | `spec.envFromSecrets` | Projects existing Secret(s) as pod env (`envFrom`) — the general Secret-envFrom escape hatch; Secret values never transit the operator. This is the mechanism behind [the Kafka SASL-credentials note](/docs/configure/reference/connectors#kafka) (there is no dedicated `#sasl` anchor). |
## High availability [#high-availability]
High availability on Kubernetes is **multiple replicas** managed by the operator — not the
Docker `cluster.*` block.
| Setting | Type | Default | Valid values | Docker (config.yaml key · env var) | Helm/CRD path | Notes |
| ------------- | -------- | ------- | --------------------------------------- | ---------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Replica count | `*int32` | `3` | CRD `Minimum=0`; `0`/nil coerced to `3` | — | `spec.replicas` | nil → 3 (`deployment.go:59-62`), and `SetReplicas(0)` → 3 (`deployment/statefulset.go:183-191`). Backs the `scale` subresource (`specpath=.spec.replicas`). More than one replica = clustered HA (operator wires `CLUSTER_*` + the 5228 cluster port). **Once the engine is established as `next`, `spec.replicas` is immutable** — a CEL rule plus an operator fail-closed guard reject any change (see the engine-establishment guard below). |
| Standalone | bool | `false` | true / false | — | `spec.standalone` | true → a single **non-clustered** node: the operator omits `CLUSTER_NAME`/`CLUSTER_ROUTES`/`CLUSTER_ENABLE` and the `5228` cluster-port from the pod (`deployment/statefulset.go:40-47,103-107`). On the `next` engine, standalone runs a **loopback raft** and skips replication wiring entirely. |
On Kubernetes, high availability is `spec.replicas` (operator-managed clustering) — **not**
the `config.yaml`-only `cluster.*` block, which has no Helm/CRD path. See the
[Advanced reference](/docs/configure/reference/advanced) for the Docker `cluster.*` path.
**Connector Services & ports (operator-exposed).** The pod always publishes container ports
for every wire connector — MQTT `1883/8883/8083`, AMQP `5672/5671`, STOMP `61613/61614`,
AWS `4566`, GCP `8085`, **Kafka `9092/9093`** (`deployment/statefulset.go:70-102`,
`deployment/service.go:192-256`). The matching ClusterIP `Service` is created **only when
that connector is enabled** (`spec..enabled: true`) — otherwise the operator
prunes it (`deployment.go:122-199`). Wire connectors are **opt-in** (`enabled: true`,
default off); the HTTP interfaces above (gRPC/REST/API) remain **opt-out** (`disabled: true`).
**`status.engine`.** The `KubemqCluster` status subresource exposes `status.engine`
(`legacy` / `next`) — the engine established for this cluster by the guard below. Read it
alongside `status.replicas` when auditing the replicas-freeze caveat above: once
`status.engine` reports `next`, `spec.replicas` can no longer change.
### Engine-establishment guard [#engine-establishment-guard]
The operator records the **live** persistence engine in the
`core.k8s.kubemq.io/established-engine` annotation (`legacy` / `next`) — once present,
this annotation is **authoritative**, overriding `spec.store.engine` for every guard
decision. Derivation runs in order: (1) the annotation, if present; (2) else the explicit
`spec.store.engine`; (3) else the pod ConfigMap's `STORE_ENGINE` key (absent ⇒ `legacy`);
(4) else, if a StatefulSet or retained PVC already exists, the engine is **unknown** and
the operator refuses to guess — set `spec.store.engine` explicitly or delete the retained
PVCs; (5) else the cluster is genuinely fresh and the engine is inferred (Kafka enabled →
`next`, otherwise `legacy`).
The operator **never changes a live cluster's engine.** Two advisory CEL rules on the CRD
back this: `spec.store.engine` is immutable once set (born-one-mode), and `spec.replicas`
is immutable once the engine is established as `next`.
This operator-side inference is the **pre-boot mirror** of the server-side auto-select
probe documented on [Storage Engines](/docs/configure/reference/storage-engines#zero-config-engine-selection)
and the Kafka callout on [Connectors](/docs/configure/reference/connectors#kafka) — two
layers of one decision: the operator decides what the pod's env will say before the pod
ever boots, and the server-side probe is what actually runs once it does.
## Example [#example]
Publish the gRPC port on each target. On Kubernetes you set the `Service` exposure and node
port; on Docker you publish the port with `-p`. This is a single-setting snippet — see the
[Kubernetes guide](/docs/configure/kubernetes) for complete, runnable configurations.
```yaml title="values.yaml"
grpc:
expose: NodePort
nodePort: 32000
```
For the full install flow (CRDs → operator → cluster), `values.yaml` mapped to the
`KubemqCluster` spec, and single-node vs HA, see the
[Kubernetes guide](/docs/configure/kubernetes).