Everything you Need to Know About Kubernetes DaemonSets

Kubernetes is a leading orchestration system for containerized applications, offering DevOps professionals varying degrees of control over their deployments. While many higher-level configurations are possible within Kubernetes, deeper control is often necessary to extract ideal backend performance. DaemonSets allow admins to transcend some fundamental limitations of the K8s scheduler.

This article provides an introduction to Kubernetes DaemonSets—including their benefits, use cases, and pertinent best practices. Though functioning as a starter guide, it goes beyond the superficial to tackle some deeper nuances. 

What are DaemonSets?

Essentially, DaemonSets are Kubernetes objects that allow you to exert greater control over application processes throughout your deployment. Why is this important? Certain applications and clusters have their own unique processes; occasionally, these background behaviors are isolated to certain nodes. The scheduler by itself makes it more challenging to apply blanket processes across all cluster nodes simultaneously. By leveraging DaemonSets, you can specify crucial tasks and services across every active node. This has a few core applications:

• Maintenance

• Monitoring and logging

• Node and cluster oversight

Every cluster has a “desired state” which indicates how containerized applications should run; these states are tied to YAML configuration files and the Kubernetes API. However, desired states and observed states don’t always match. The same goes for deployed pods. When nodes have mismatching pods or aren’t performing to spec, then a DaemonSet will automatically create an appropriate pod within that node. 

How does a DaemonSet know when performance isn’t up to snuff? Operational awareness ties back to the DaemonSet use cases—specifically automated logging and resource monitoring. These processes are typically extended, meaning they occur over longer periods of time. 

Although system health can vary at a moment’s notice, these enduring processes make DaemonSets extremely valuable for observing systemic performance trends. DaemonSet controllers govern the application of these DaemonSets.

What makes DaemonSets so useful? There are some cons to controlling daemons via their associated nodes alone. Daemon processes must run continuously as long as the node is active and not evicted. Additionally, deferring to the node can mean accidentally pairing daemons with mismatched runtime environments. Each daemon process should run properly within a container and remain aware of its place within the greater Kubernetes system. DaemonSets facilitate this awareness by comparison, where processes might otherwise be lost in the shuffle.  

Creating a DaemonSet

As with many objects in Kubernetes, creating a DaemonSet is as simple as assembling a YAML configuration file. This will determine how Daemon-related pods are applied across varying compatible nodes. The term “compatible” is appropriate here, since DaemonSet YAML files often specify some sort of tolerance—or condition allowing pods to schedule themselves on certain nodes. Those pod-deployment rules directly impact the use of customized DaemonSets. 

The Kubernetes documentation, however, only outlines a need for four specific YAML components when creating a functional DaemonSet: fields for apiVersion, kind, and metadata, plus a section for .spec. This section will include any tolerations, operator conditions, effects, container configurations, and terminationGracePeriodSeconds.

DaemonSet pods have pod templates, outlined via .spec.template. These templates must specify labels and have a RestartPolicy set to Always or unspecified. 

Here’s what a partial YAML configuration file might look like if you elect to use an Alpine Linux image:

Once you create the configuration file for your DaemonSet, you must apply it via `kubectl`. This requires a simple command, according to the file’s URL: 

kubectl apply -f https://respository.io/example/controllers/daemonlogging.yaml

This will manually apply your DaemonSet to all matching nodes, but all pod deployments will occur according to your specifications. We’ve previously mentioned the importance of tolerations; however, these conditions have counterparts called taints. These properties let nodes reject sets of scheduled pods. Taints work together with tolerations. If a taint satisfies a certain toleration, and conditions match appropriately, then pods are scheduled successfully.

Both of these properties thus work in tandem to ensure nodes aren’t matched with mismatched pods. It’s also easy to evict pods that shouldn’t be running according to these configurations. This can naturally affect how DaemonSets are applied—if they can be at all in a given scenario. 

Node Affinity and Scheduling Variations

Evictions aside, taints are useful for designating pods for dedicated nodes or nodes with specialized hardware. You also have to account for something called node affinity—or properties which attract pods to certain nodes. Within your YAML file, you can specify NodeAffinity using nodeSelectorTerms, including matchFields with keys, operators, and qualitative values. Note that the NodeAffinity specification replaces the default .spec.nodeName term. While affinity can be based on hard specifications, it may also be based on your selected preferences. 

If you prefer to leverage the Kubernetes default scheduler to deploy DaemonSets, you’ll want to use NodeAffinity in conjunction with ScheduleDaemonSetPods. Otherwise, the system will deploy DaemonSets using the DaemonSet controller. Similar to node affinity, you can also match DaemonSets to nodes using specific node selectors. The .spec.template.spec.nodeSelector field instructs the DaemonSet controller accordingly.

Updating and Removing DaemonSets

A given DaemonSet will update itself automatically any time node labels are changed; in response, the DaemonSet creates new pods and adds them to compatible nodes. On the other hand, it will also delete any pods that become mismatched as a result. While you CAN modify pod configurations, some fields cannot be changed.

This places de facto limits on the scope of your updates. Furthermore, the DaemonSet controller will defer to your original templates once new nodes are created. This does hamper adaptability to an extent. It may also be a reason to leverage the default Kubernetes scheduler—depending on your configurations and affinity specs. 

Update Strategies

That said, Kubernetes applies DaemonSet updates on a rolling basis. The rolling method is more controlled and doesn’t wipe everything in one fell swoop; conversely, the update process occurs in stages. Only one DaemonSet pod will run on a given node while the update progresses. Meanwhile, the OnDelete update process dictates that new DaemonSet pods are only created once old pods are deleted. This method is comparatively more tedious, yet does offer a degree of control to admins who prefer stricter processes. 

Designating a strategy is pretty simple within your DaemonSet YAML file. All you need to do is create an updateStrategy field, with your specific update type denoted underneath. The updateStrategy fields live within your overarching .spec section, notably before your template fields and your aforementioned tolerations sub-specification. 

For the rolling strategy—which is the default—some other helpful properties are helpful. For instance, you may specify a maximum number of unavailable pods during the process, using .spec.strategy.rollingUpdate.maxUnavailable, or a percentage. This limits the active impact of your updates, ensuring that your system won’t be hobbled throughout its duration. You can also set a deployment timeout limit, after which Kubernetes will attempt to apply your updates (and create your new pods) once more. 

Notes on DaemonSet Deletion

Deleting a DaemonSet is also relatively straightforward. The --cascade=false conditional within kubectl will ensure that pods remain within their nodes; however, DaemonSets will replace these pods according to your update strategy, and based on their unique selectors. 

You can delete a DaemonSet within kubectl by using the following command:

kubectl delete -f daemonset.yaml

This command will also clean up any pods created via the DaemonSet, which is something to keep in mind. 

DaemonSet Usage Examples

Logging is one clear use case for a DaemonSet. Visibility and monitoring are everything when it comes to measuring Kubernetes performance or health. Collecting logs on each node is essential to retroactive performance analysis. Because DaemonSets run on all applicable nodes, they’re natural conduits for capturing those critical events as they occur.

You can run a node-level logging agent as a DaemonSet—which exposes logs and sends them to a designated backend. This directory of log files remains accessible and catalogues all events from node application containers. This setup is also favorable since no application changes are needed to facilitate it. 

Similar to logging agents, the DaemonSet also helps third-party monitoring agents run within each target node. You might’ve heard of Prometheus, DataDog, or AppDynamics; each offers its own connected monitoring solution that you can deploy universally. Specifically, the Prometheus Node Explorer set allows Kubernetes to export crucial metrics to a remote backend—connecting teams with aggregated data in dashboards. 

DaemonSet Alternatives

DaemonSets are extremely capable, yet there are other ways to achieve similar workflows within Kubernetes. Firstly, you could use init scripts to automatically run daemon processes on node startup. You can also leverage both bare pods (directly and tied to specified nodes) or static pods (writing files to directories). These methods have their drawbacks, however, and the latter might even be deprecated according to official documentation. 

DaemonSets are Powerful

There’s little doubt that DaemonSets allow the automation and streamlining of many tedious Kubernetes processes. Harnessing these objects is fairly user friendly, and the level of configurability available makes them full-fledged solutions for your nodes. Their deployment hierarchy is easy to grasp, and they’re just as easy to deploy as they are to remove. Like Deployments and StatefulSets, DaemonSets remain one of the most-popular controllers within the Kubernetes system. 

Ensure that your tags, labels, and YAML metadata is configured appropriately to get the most out of your DaemonSets. You’ll be using one of K8s’ most-powerful objects to directly influence your deployments. Be sure to follow best practices and ensure your use cases align with DaemonSet capabilities before proceeding. 

Header image courtesy of Karolis, via Dev.to.