Perform a Rolling Update on a DaemonSet
This page shows how to perform a rolling update on a DaemonSet.
- Before you begin
- DaemonSet Update Strategy
- Limitations
- Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before
- Setting DaemonSet update strategy for rolling update
- Troubleshooting
- What’s next
Before you begin
- The DaemonSet rolling update feature is only supported in Kubernetes version 1.6 or later.
DaemonSet Update Strategy
DaemonSet has two update strategy types :
- OnDelete: This is the default update strategy for backward-compatibility. With
OnDeleteupdate strategy, after you update a DaemonSet template, new DaemonSet pods will only be created when you manually delete old DaemonSet pods. This is the same behavior of DaemonSet in Kubernetes version 1.5 or before. - RollingUpdate: With
RollingUpdateupdate strategy, after you update a DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods will be created automatically, in a controlled fashion.
Limitations
- DaemonSet rollout history is not supported yet.
- DaemonSet rollback is not directly supported in
kubectlyet. You can rollback by updating DaemonSet template to match the previous version.
Caveat: Updating DaemonSet created from Kubernetes version 1.5 or before
If you try to rolling update a DaemonSet that was created from Kubernetes
version 1.5 or before, a rollout will be triggered when you first change the
DaemonSet update strategy to RollingUpdate, even when DaemonSet template isn’t
modified. All existing DaemonSet pods will be restarted.
To avoid this restart, first find the DaemonSet’s current
.spec.templateGeneration:
kubectl get ds/<daemonset-name> -o go-template='{{.spec.templateGeneration}}{{"\n"}}'
The output should be a number N. If the output shows <no value>, N = 0.
Then, simply label the existing DaemonSet pods with pod-template-generation=<N>
before changing DaemonSet .spec.updateStrategy to RollingUpdate:
# Replace N with DaemonSet `.spec.templateGeneration`
# Only run this on pods created from current DaemonSet template
kubectl label pods -l <daemonset-selector-key>=<daemonset-selector-value> pod-template-generation=<N>
This tells the DaemonSet that the labeled DaemonSet pods are created from current DaemonSet template. Therefore, you should only run this command to existing DaemonSet pods that are generated from current DaemonSet template.
Note that you only need to do this when you first change the update strategy of
a DaemonSet created from Kubernetes version 1.5 or before to RollingUpdate,
but don’t want to update its template and start a new rollout yet.
Setting DaemonSet update strategy for rolling update
To enable the rolling update feature of a DaemonSet, you must set its
.spec.updateStrategy.type to RollingUpdate.
You may want to set .spec.updateStrategy.rollingUpdate.maxUnavailable (default
to 1) and .spec.minReadySeconds (default to 0) as well.
Step 1: Checking DaemonSet RollingUpdate update strategy
First, check the update strategy of your DaemonSet, and make sure it’s set to RollingUpdate:
kubectl get ds/<daemonset-name> -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
If you haven’t created the DaemonSet in the system, check your DaemonSet manifest with the following command instead:
kubectl create -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
The output from both commands should be:
RollingUpdate
If the output isn’t RollingUpdate, go back and modify the DaemonSet object or
manifest accordingly.
Step 2: Creating a DaemonSet with RollingUpdate update strategy
If you have already created the DaemonSet, you may skip this step and jump to step 3.
After verifying the update strategy of the DaemonSet manifest, create the DaemonSet:
kubectl create -f ds.yaml
Alternatively, use kubectl apply to create the same DaemonSet if you plan to
update the DaemonSet with kubectl apply.
kubectl apply -f ds.yaml
Step 3: Updating a DaemonSet template
Any updates to a RollingUpdate DaemonSet .spec.template will trigger a rolling
update. This can be done with several different kubectl commands.
Declarative commands
If you update DaemonSets using
configuration files,
use kubectl apply:
kubectl apply -f ds-v2.yaml
Imperative commands
If you update DaemonSets using
imperative commands,
use kubectl edit or kubectl patch:
kubectl edit ds/<daemonset-name>
kubectl patch ds/<daemonset-name> -p=<strategic-merge-patch>
Updating only the container image
If you just need to update the container image in the DaemonSet template, i.e.
.spec.template.spec.containers[*].image, use kubectl set image:
kubectl set image ds/<daemonset-name> <container-name>=<container-new-image>
Step 4: Watching the rolling update status
Finally, watch the rollout status of the latest DaemonSet rolling update:
kubectl rollout status ds/<daemonset-name>
When the rollout is complete, the output is similar to this:
daemon set "<daemonset-name>" successfully rolled out
Troubleshooting
DaemonSet rolling update is stuck
Sometimes, a DaemonSet rolling update may be stuck. Here are some possible causes:
Some nodes run out of resources
The rollout is stuck because new DaemonSet pods can’t be scheduled on at least one node. This is possible when the node is running out of resources.
When this happens, find the nodes that don’t have the DaemonSet pods scheduled on
by comparing the output of kubectl get nodes and the output of:
kubectl get pods -l <daemonset-selector-key>=<daemonset-selector-value> -o wide
Once you’ve found those nodes, delete some non-DaemonSet pods from the node to make room for new DaemonSet pods. Note that this will cause service disruption if the deleted pods are not controlled by any controllers, or if the pods aren’t replicated. This doesn’t respect PodDisruptionBudget either.
Broken rollout
If the recent DaemonSet template update is broken, for example, the container is crash looping, or the container image doesn’t exist (often due to a typo), DaemonSet rollout won’t progress.
To fix this, just update the DaemonSet template again. New rollout won’t be blocked by previous unhealthy rollouts.
Clock skew
If .spec.minReadySeconds is specified in the DaemonSet, clock skew between
master and nodes will make DaemonSet unable to detect the right rollout
progress.
What’s next
TODO: Link to “Task: Creating a DaemonSet to adopt existing DaemonSet pods”