# Database operations > **For AI agents:** the complete documentation index is at [llms.txt](https://questdb.com/docs/llms.txt). Every page is available as markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` request header. Operate, scale, resize, maintain, and delete Operator-managed QuestDB clusters. # Database operations Examples use `` and ``. Status is trustworthy only after the controller has observed the current spec. ## Check database health Check generation freshness first, then conditions. `status.phase` is only a human-readable summary. ```sh kubectl get questdbcluster -n \ -o jsonpath='generation={.metadata.generation}{" observedGeneration="}{.status.observedGeneration}{" phase="}{.status.phase}{"\n"}' kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{" message="}{.message}{"\n"}{end}' ``` Do not act on old conditions while `.metadata.generation` differs from `.status.observedGeneration`. For an ordinary writable cluster, writer readiness requires all four signals: current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone can mean only that a replica or a read-serving primary is available. `WriteHealthy=False/PrimarySuspended` names the impaired WAL table or tables in its message; reads and writes to other tables can remain available. `WriteHealthy=True` reports the latest `wal_tables()` observation. It does not perform a synthetic write and does not guarantee free disk capacity, so retain application write probes and storage monitoring where those guarantees are required. An intentional replica-only follower is the exception: no primary is correct, so `WriteHealthy` is omitted rather than reported healthy. Require current generation, `phase=Following`, `status.replication.following=true`, the expected `readyInstances`, and an appropriate `ReplicationHealthy` follower result. `True/FollowingExternalSource` reports observed progress; a quiet source may report `Unknown/StreamNotDetermined`, which is acceptable only after confirming the source identity and roots. Treat `ReplicationHealthy=False` as unhealthy. ```sh kubectl get questdbcluster -n -o wide kubectl get pods -n -l questdb.io/cluster= \ -L questdb.io/instance,questdb.io/role -o wide kubectl get pvc -n -l questdb.io/cluster= \ -L questdb.io/role,questdb.io/bootstrap -o wide ``` ## Services and ports | Service | Purpose | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-rw` | Current primary. Use for writes and administration. | | `-ro` | Created for store-backed replication. Routes to qualified replicas, but falls back to the primary when no replica qualifies. It is **not** strict replica isolation. | | `` | Headless, internal identity/DNS service. It publishes unready addresses and must not be used as a client availability endpoint. | | Port | Scope | | -------- | ----------------------------------------------------------------------- | | 9000/TCP | Published on database Services | | 8812/TCP | Published on database Services; optional TLS chosen at cluster creation | | 9009/TCP | Published on database Services | | 9003/TCP | Always on Pods for kubelet/operator health and metrics; not on Services | | 9007/UDP | Published on headless and `-rw` only when enabled | QWP over WebSocket has no port of its own. Ingestion (`/write/v4`) and streaming query results (`/read/v1`) are served by the HTTP server on 9000 and share its network settings, so they are available wherever 9000 is. Existing Service-based database metrics scrapers must move to Pod discovery or a `PodMonitor` that targets the named `metrics` port on QuestDB Pods. A Service port-forward to 9003 no longer works. The QWP UDP receiver is off by default. Enable it with [`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#qwp-udp); until then, port 9007 is neither opened on the pod nor published on any Service. It is not published on `-ro`. The operator owns these Services and reconciles them as `ClusterIP` (the identity Service is headless). For temporary access, use port-forwarding. For durable external access, create a separate customer-managed Ingress, Gateway, or LoadBalancer that targets the operator Service. Do not mutate the operator-owned Service type. ## Connect Find the admin Secret without printing its value: ```sh ADMIN_SECRET="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.adminSecretName}')" kubectl get secret "$ADMIN_SECRET" -n \ -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp' ``` When an administrator explicitly needs the credential, load it into the environment without echoing it and clear it after use: ```sh ADMIN_PASSWORD="$(kubectl get secret "$ADMIN_SECRET" -n \ -o jsonpath='{.data.password}' | base64 -d)" kubectl port-forward -n service/-rw 8812:8812 \ >/tmp/questdb-pgwire-port-forward.log 2>&1 & PF_PID=$! trap 'kill "$PF_PID" 2>/dev/null; unset ADMIN_PASSWORD' EXIT sleep 2 PGPASSWORD="$ADMIN_PASSWORD" psql -h 127.0.0.1 -p 8812 -U admin -d qdb kill "$PF_PID" unset ADMIN_PASSWORD trap - EXIT ``` For a TLS-enabled cluster whose certificate uses the default Service identity, keep the same port-forward but give `psql` both the certificate identity and the local address: ```bash PGPASSWORD="$ADMIN_PASSWORD" psql \ "host=-rw..svc hostaddr=127.0.0.1 port=8812 user=admin dbname=qdb sslmode=verify-full sslrootcert=/secure/path/ca.crt" ``` `host` supplies the certificate identity and SNI; `hostaddr` directs the connection to the local tunnel. Use `service/-ro` only for read-only traffic that can tolerate primary fallback. To access the Web Console temporarily, forward `service/-rw` from local port 9000 and stop the process after the session. ## Make mutable changes safely Changes to `spec.image`, `spec.imagePullSecrets`, `spec.resources`, `spec.config`, replication tuning, protocol fields within their allowed mode, or scheduling recreate affected pods. Ordinary rollout is serialized: replicas roll in serial order before the primary, and at most one ordinary delete is attempted per instanceset reconcile. The operator may hold a rollout for exact topology, usable PVCs, healthy peers, node readiness, or read-route safety. A rolling change remains single-writer-safe, but recreating the primary can briefly interrupt writes; a singleton has read/write downtime while its only Pod restarts. Before deleting the only `ro-ready` replica, the operator first adds the primary as an overlapping Service-selector candidate. That overlap is not an EndpointSlice acknowledgement, connection drain, or zero-gap guarantee. ### Before you start 1. For an ordinary writable cluster, confirm current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone is not writer readiness. 2. For an intentional follower, instead confirm current generation, `phase=Following`, `status.replication.following=true`, the expected ready count, and an appropriate non-failing follower `ReplicationHealthy` result as described above. 3. Save the current spec. 4. Make one logical change at a time. ```sh kubectl get questdbcluster -n -o yaml \ > /secure/path/-before.yaml ``` ### Change and watch Apply a reviewed manifest or a narrow patch. This example changes an engine setting: ```sh kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"config":{"cairo.max.uncommitted.rows":"500000"}}}' GEN="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" for _ in $(seq 1 60); do OBS="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" [ "$OBS" = "$GEN" ] && break sleep 5 done [ "$OBS" = "$GEN" ] kubectl wait questdbcluster/ -n \ --for=condition=Available --timeout=5m ``` The `kubectl wait` above is only an availability gate. Before declaring the writer ready, re-read the current-generation conditions and require `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. Then verify pod UIDs/restarts, connectivity, and the intended setting. `Progressing=True/RollingUpdate` can mean the controller is safely waiting on topology, PVC, peer, Node, or read-routing conditions. If the change fails, inspect `ConfigRejected`, events, and pod logs. Recover by reverting the mutable spec to the saved value and repeat the bounded generation and full writer-health checks. Do not try to revert immutable fields; create a new cluster when required by the [API Reference](/docs/enterprise-kubernetes-operator/reference/api/). ### Rotate static object-store credentials safely 1. Make old and new provider credentials valid concurrently. 2. Update the same-namespace Secret referenced by `QuestDBObjectStore`. 3. Allow the manager's roughly two-minute resync plus serialized Pod restart to converge. 4. Verify expected Pod UID changes and `BackupHealthy`/`ReplicationHealthy`. 5. Revoke the old credentials only after convergence. Immediate provider-side revocation can interrupt in-flight QuestDB object-store I/O. ## Scale out A replica is born from a completed backup and then consumes object-store WAL. Before increasing `spec.instances`, the cluster needs: - `spec.objectStoreRef`; - enabled scheduled backup; and - a completed seed in `.status.replication.seed`. An explicit `spec.replication` block is optional; add it only for tuning. ```sh BACKUP_STATUS="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.backup.lastBackup.status}')" SEED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.replication.seed.backupInstanceName}')" printf 'backup=%s seed=%s\n' "$BACKUP_STATUS" "$SEED" [ "$BACKUP_STATUS" = "completed" ] && [ -n "$SEED" ] kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"instances":3}}' ``` Watch with bounded checks: ```sh for _ in $(seq 1 90); do READY="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.readyInstances}')" [ "$READY" = "3" ] && break sleep 10 done [ "$READY" = "3" ] kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}' ``` Verify `readyInstances` equals the requested count, `ReplicationHealthy=True`, and each current replica is healthy. `caughtUp` means it has caught up at least once; it is a latch. `caughtUpNow` is the live freshness reading. An absent `caughtUpNow` means freshness was not determined and must fail closed. ## Scale in Lowering `spec.instances` removes replicas. Instance serials are monotonic and are never reused, so names may not remain contiguous after scale or promotion. | `spec.pvcRetentionPolicy` | Removed replica PVC | | ------------------------- | ----------------------------------------------------------------------------------- | | `Retain` (default) | Kept for deliberate inspection or cleanup. | | `Delete` | Deleted during replica scale-in. A later replica gets a new serial and seeds again. | The policy applies only to replicas removed by scale-in. It never deletes the current primary PVC and does not govern whole-cluster deletion. On every managed PVC, `questdb.io/role` is the instance's **current** role and changes across promotion; `questdb.io/bootstrap` records how the volume was born and remains ancestry/state. Do not infer the current primary from `bootstrap`. Before scaling in, identify the primary and replicas, choose the retention policy, and inventory PVCs: ```sh kubectl get questdbcluster -n -o wide kubectl get pvc -n -l questdb.io/cluster= kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"instances":1,"pvcRetentionPolicy":"Retain"}}' GENERATION="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" OBSERVED="" AVAILABLE_OBSERVED="" for _ in $(seq 1 120); do OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" AVAILABLE_OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')" [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break sleep 5 done if [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then kubectl wait questdbcluster/ -n \ --for=condition=Available=True --timeout=10m && kubectl get questdbcluster -n -o wide && kubectl get pods,pvc -n -l questdb.io/cluster= -o wide else printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \ "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2 false fi ``` Verify the managed and ready counts, current primary, the full ordinary writer-health or separate follower contract, and the expected retained or deleted replica PVCs. Confirm each current primary/replica PVC's `questdb.io/role` label matches that role. ## Move the cold-storage manager With [`spec.coldStorage`](/docs/enterprise-kubernetes-operator/configuration/#cold-storage), the cold-storage [manager role](/docs/concepts/cold-storage/#roles) is moved by changing `spec.coldStorage.manager`, never by running [`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/) against instances. Run the manager on a replica: that moves upload, manifest, and garbage-collection work off the primary, and a `Planned` promotion requires the manager settled away from the departing primary. Before the change, require current generation, `ColdStorageHealthy=True/ManagerReady`, and an empty `handoffSource`: ```sh kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[?(@.type=="ColdStorageHealthy")]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{"\n"}{end}manager={.spec.coldStorage.manager}{" current="}{.status.coldStorage.currentManager}{" term="}{.status.coldStorage.managerTerm}{" handoffSource="}{.status.coldStorage.handoffSource}{"\n"}' ``` Choose the serial of a ready replica (`-3` has serial `3`) and patch: ```sh kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"coldStorage":{"manager":3}}}' ``` The operator demotes the old manager, verifies it settled as a refresher, then promotes the replacement — never two managers at once. While the handoff is in flight, `status.coldStorage.handoffSource` names the demoted instance. Uploads and remote garbage collection pause during the short managerless interval; reads continue everywhere. Wait for `True/ManagerReady` with `currentManager` naming the target and a positive `managerTerm`. `False/ManagerHandoffBlocked` fails closed rather than risking two managers — for example while the current manager is unreachable. Recover the named instance; do not work around the block with direct SQL. Two interactions to plan around: - An active promotion pauses new cold-manager handoffs. Finish or resolve the cutover first. - `manager` must not exceed `instances`, so scale in only after moving the manager to a remaining serial. ## Grow storage Storage is expand-only, and `spec.storage.storageClassName` is immutable. ### Before you start ```sh STORAGE_CLASS="$(kubectl get questdbcluster -n \ -o jsonpath='{.spec.storage.storageClassName}')" kubectl get storageclass "$STORAGE_CLASS" \ -o jsonpath='allowVolumeExpansion={.allowVolumeExpansion}{"\n"}' kubectl get pvc -n -l questdb.io/cluster= ``` Proceed only when `allowVolumeExpansion=true` and the CSI driver supports the requested expansion. ### Change and verify ```sh kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"storage":{"size":"200Gi"}}}' GENERATION="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" OBSERVED="" AVAILABLE_OBSERVED="" for _ in $(seq 1 120); do OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" AVAILABLE_OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')" [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break sleep 5 done if [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then kubectl wait questdbcluster/ -n \ --for=condition=Available=True --timeout=10m && kubectl get questdbcluster -n -o wide && kubectl get pvc -n -l questdb.io/cluster= \ -o custom-columns='NAME:.metadata.name,REQUESTED:.spec.resources.requests.storage,CAPACITY:.status.capacity.storage,PHASE:.status.phase' else printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \ "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2 false fi ``` The availability wait is not proof of writer health. Require the current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy` conditions before closing the change. If `StorageResizeBlocked=True`, read its reason/message and fix the StorageClass or CSI limitation. A size reduction is rejected. To change StorageClass, restore into a new cluster. ## Node maintenance and disruption budgets A replicated cluster gets a default PodDisruptionBudget (PDB) with `minAvailable: instances-1`, which permits one voluntary disruption at a time. A singleton gets no PDB by default. `kubectl drain` and managed node upgrades evict its only pod, and the operator recreates it on another node in the same zone as its volume. The database is down until the new pod is ready. To avoid that outage, scale out and wait for a healthy, current replica before planned maintenance. To make a singleton block voluntary eviction instead, enable the PDB. A drain of its node then waits until you lower `minAvailable` or disable the PDB again: ```yaml spec: scheduling: podDisruptionBudget: enabled: true ``` After maintenance, verify the current-generation writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. A PDB protects only voluntary disruption. On node loss, the operator reports `InstanceUnreachable` and does not automatically promote a replica. It avoids recreating a pod while the old pod may still run on an unreachable node. Restore the node/volume or follow the explicit [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) procedure. A node that is `NotReady` while its machine keeps running leaves the pod stuck until the machine is repaired or replaced. Turn on node auto-repair so this happens without a person: AKS repairs unhealthy nodes automatically, and on EKS you enable node auto repair on the managed node group. Once the node is repaired or replaced, the pod runs again. ## Delete a database cluster Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, and related resources. Its data PVCs are deliberately unowned and remain, and object-store backup/WAL data also remains. `pvcRetentionPolicy` does not change this whole-cluster behavior. ### Before you start Inventory and export the CR, PVCs, PV reclaim policies, store reference, and effective object prefixes: ```sh kubectl get questdbcluster -n -o yaml \ > /secure/path/-cluster.yaml kubectl get pvc -n -l questdb.io/cluster= -o wide kubectl get questdbobjectstore -n ``` :::danger Deleting the custom resource stops the database and removes its managed runtime objects. Confirm applications are stopped and that the CR export and data-retention inventory are complete. ::: ```sh kubectl delete questdbcluster -n --timeout=5m kubectl get pvc -n -l questdb.io/cluster= ``` Decide separately whether to retain or delete each PVC/PV according to its StorageClass reclaim policy. Decide separately whether to retain or delete cloud objects using customer-owned cloud tooling and policy. The operator never cleans the object store.