In November 2019 I wrote an article on using kustomize with Cluster API (CAPI) manifests. The idea was to use kustomize to simplify the management of CAPI manifests for clusters that are generally similar but have minor differences (like the AWS region in which they are running, or the number of Machines in a MachineDeployment). In this post, I’d like to show a slightly different way of using kustomize with Cluster API that involves configuring the kustomize transformers.
If you aren’t familiar with kustomize, I’d recommend having a look at the kustomize web site and/or reading my introductory post. A transformer in kustomize is the part that is responsible for modifying a resource, or gathering information about a resource over the course of a kustomize build process. This page has some useful terminology definitions.
Looking back at the earlier article on using kustomize with CAPI, you can see that—due to the links/references between objects—modifying the name of the AWSCluster object also means modifying the reference to the AWSCluster object from the Cluster object. The same goes for the KubeadmConfigTemplate and AWSMachineTemplate objects referenced from a MachineDeployment. Out of the box, the namePrefix transformer will change the names of these objects (because they all have a metadata.name field), but it will not change the references to these objects. Thus, modifying the names of all the objects becomes an exercise in writing multiple patch files (somewhere around 15-20 patch files).
As it turns out, though, there’s a way to fix this by configuring the transformers within kustomize (more information here).
The nameReference transformer within kustomize is used to track references between objects like this, so that when the name of an object is changed—say, using the namePrefix transformer—then kustomize knows all the other places it needs to change as well. By default, this transformer isn’t aware of CAPI objects, and thus doesn’t know how (or that it should) update references to other objects. By configuring this transformer, users can change that behavior.
Here’s how you can give the nameReference transformer a bit more awareness of CAPI objects so it does know how to update references to objects.
First, get the default transformer configurations by saving them with the kustomize config save -d <directory> command. The specific file you need is the namereference.yaml file created by that command. You can discard the others, unless you plan to make modifications to other transformers. There is one other configuration change I’ll recommend, so don’t ditch all the files just yet.
Second, edit the namereference.yaml file to include this:
- kind: AWSCluster group: infrastructure.cluster.x-k8s.io version: v1alpha2 fieldSpecs: - path: spec/infrastructureRef/name kind: Cluster - kind: AWSMachine group: infrastructure.cluster.x-k8s.io version: v1alpha2 fieldSpecs: - path: spec/infrastructureRef/name kind: Machine - kind: KubeadmConfig group: bootstrap.cluster.x-k8s.io version: v1alpha2 fieldSpecs: - path: spec/bootstrap/configRef/name kind: Machine - kind: AWSMachineTemplate group: infrastructure.cluster.x-k8s.io version: v1alpha2 fieldSpecs: - path: spec/template/spec/infrastructureRef/name kind: MachineDeployment - kind: KubeadmConfigTemplate group: bootstrap.cluster.x-k8s.io version: v1alpha2 fieldSpecs: - path: spec/template/spec/bootstrap/configRef/name kind: MachineDeployment
Third, add this line to your kustomization.yaml file (see here for more information on using configurations):
configurations:
- /path/to/customized/namereference.yaml
Now, when you use something like namePrefix in your kustomization.yaml file for an overlay, not only will the metadata.name fields of all objects get transformed, but the references to those objects will also be updated. So, the reference from the Cluster object to the AWSCluster object will be updated. The reference from the MachineDeployment object to the KubeadmConfigTemplate and AWSMachineTemplate objects will be updated. The reference from a Machine object to the KubeadmConfig object will be updated.
With this one change to the nameReference transformer, you can go from individually patching objects to using a single namePrefix or nameSuffix transformer and having all objects appropriately updated. This greatly simplifies the use of kustomize with CAPI.
There’s a second change I’d also recommend you consider. The commonLabels transformer is used to add labels to objects. By default, this transformer doesn’t know how to add labels to a MachineDeployment, and this means that you can’t use the commonLabels transformer to modify the labels used by a MachineDeployment in your CAPI manifest. Instead, you’d have to manually patch these objects. (This is necessary because CAPI uses labels to provide cluster membership information. Change the name of the cluster, and you have to change the labels as well.)
To fix this, edit the commonlabels.yaml file generated by kustomize config save to add this content:
- path: spec/selector/matchLabels create: true group: cluster.x-k8s.io kind: MachineDeployment - path: spec/template/metadata/labels create: true group: cluster.x-k8s.io kind: MachineDeployment
Now the commonLabels transformer knows that labels have to be added to the spec.selector.matchLabels and spec.template.metadata.labels fields for MachineDeployment objects. This keeps you from having to patch individual objects; instead, you can use a single commonLabels entry in your kustomization.yaml file (you also have to reference the customized configuration using a configurations line as shown above). Sweet!
I’m still exploring what other changes may be beneficial, but these are the two I’ve found (so far) that have had the most impact on the usability of kustomize with CAPI. If I find more, I’ll either update this post or publish a follow-up post.
If you have questions, comments, suggestions for improvement, or corrections, please let me know. Feel free to contact me on Twitter, reach out to me on the Kubernetes Slack instance, or hit me up via e-mail (my address isn’t too hard to find/figure out).
