Using External Etcd with Cluster API on AWS
CloudIf you’ve used Cluster API (CAPI), you may have noticed that workload clusters created by CAPI use, by default, a “stacked master” configuration—that is, the etcd cluster is running co-located on the control plane node(s) alongside the Kubernetes control plane components. This is a very common configuration and is well-suited for most deployments, so it makes perfect sense that this is the default. There may be cases, however, where you’ll want to use a dedicated, external etcd cluster for your Kubernetes clusters. In this post, I’ll show you how to use an external etcd cluster with CAPI on AWS.
The information in this blog post is based on this upstream document. I’ll be adding a little bit of AWS-specific information, since I primarily use the AWS provider for CAPI. This post is written with CAPI v1alpha3 in mind.
The key to this solution is building upon the fact that CAPI leverages kubeadm for bootstrapping cluster nodes. This puts the full power of the kubeadm API at your fingertips—which in turn means you have a great deal of flexibility. This is the mechanism whereby you can tell CAPI to use an external etcd cluster instead of creating a co-located etcd cluster.
I should note that I am referring to using CAPI to create a workload cluster with an external etcd environment. If you’re not familiar with some of the CAPI terminology, check out my introductory post.
At a high level, the steps involved are:
- Create the required Secrets in the management cluster.
- Modify the CAPI manifests to reference the external etcd cluster.
- Profit!
Let’s take a look at these two steps in more detail. (I’ll omit step 3.) Note that I’m not going to cover the process of establishing the etcd cluster, as that’s something I’ve covered sufficiently elsewhere (like here, here, or here).
Creating the Required Secrets
When bootstrapping a typical workload cluster with a stacked master configuration, kubeadm typically generates all the necessary public key infrastructure (certificate authority and associated certificates). In the case of an external etcd cluster, though, some of these certificates already exist, and you need a mechanism to provide those to CAPI.
This page gave me the first hint on how it should be handled, and this upstream document finishes it out. Using Kubernetes Secrets, you can provide the certificates necessary to CAPI.
To create the required Secrets, you’ll need four files from the etcd cluster:
- The etcd certificate authority (CA) certificate
- The etcd CA’s private key
- The API server etcd client certificate
- The API server etcd client private key
Files #1 and #2 are, in the event of a typical kubeadm-bootstrapped etcd cluster, found at /etc/kubernetes/pki/etcd/ca.{crt,key}. Files #3 and #4 are typically found at /etc/kubernetes/pki/apiserver-etcd-client.{crt,key}. (Keep in mind these paths may change depending on how the etcd cluster was created.)
Once you have this files, create the first Secret using this command:
kubectl create secret tls <cluster-name>-apiserver-etcd-client \
--cert /path/to/apiserver-etcd-client.crt \
--key /path/to/apiserver-etcd-client.key
Next, create the second Secret with this command:
kubectl create secret tls <cluster-name>-etcd \
--cert /path/to/etcd/ca.crt --key /path/to/etcd/ca.key
In both of these commands, you’ll need to replace <cluster-name> with the name of the workload cluster you’re going to create with CAPI.
If you are going to create your workload cluster in a specific namespace on the management cluster, you’ll want to be sure you create these Secrets in the same namespace.
Once the Secrets are in place, the next step is to modify the CAPI manifests.
Updating the CAPI Manifests
The first change is configuring the CAPI manifests to use an external etcd cluster. CAPI v1alpha3 introduces the KubeadmControlPlane object, and the KubeadmControlPlane object has a KubeadmConfigSpec that contains the equivalent of a valid kubeadm configuration file. This is the section you’ll need to modify to instruct CAPI to create a workload cluster with an external etcd cluster.
Here’s a snippet (unrelated entries have been removed for the sake of brevity) of YAML to add to the KubeadmControlPlane object in order to use an external etcd cluster:
spec: kubeadmConfigSpec: clusterConfiguration: etcd: external: endpoints: - https://ip-10-11-12-13.us-west-2.compute.internal:2379 - https://ip-10-11-14-15.us-west-2.compute.internal:2379 - https://ip-10-11-16-17.us-west-2.compute.internal:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
Obviously, you’d want to use the correct hostnames for the nodes in the external etcd cluster (the names above have been randomized). The certificate files referenced here will be created by CAPI using the Secrets created in the previous section.
The second change is to instruct CAPI to use the existing VPC where the etcd cluster resides. (Technically, this isn’t required, but this sidesteps the need for VPC peering and similar configurations). For that, you can refer to the upstream documentation for using existing AWS infrastructure, which indicates you need to add the following to your CAPI workload manifest (this is in the spec field for the AWSCluster object):
spec: networkSpec: vpc: id: vpc-0123456789abcdef0
You will want to ensure that your existing AWS infrastructure is appropriately configured for use with CAPI; again, refer to the upstream documentation for full details.
The third and final step is to ensure the new CAPI workload cluster is able to communicate with the existing external etcd cluster. On AWS, that means configuring security groups appropriately to ensure access to the etcd cluster. As I explained in this post on using existing AWS security groups with CAPI, you’ll want to configure your CAPI manifest to reference the existing AWS security group that permits access to your etcd cluster. That would look something like this in the AWSMachineTemplate that’s referenced by the KubeadmControlPlane object:
spec: template: spec: additionalSecurityGroups: - id: <etcd_security_group_id> - id: <any_other_needed_security_group_id>
Once you’ve modified the manifests for the CAPI workload cluster—adding the etcd section to the KubeadmConfigSpec, specifying the existing VPC, and adding any necessary security group IDs in the AWSMachineTemplate for the KubeadmControlPlane object—then you can create the CAPI cluster by applying the modified manifest with kubectl apply -f manifest.yaml when your kubectl context is set to your CAPI management cluster.
Everything should work fine, but in the event you run into issues be sure to check the CAPI and CAPI provider logs (using kubectl logs) in the management cluster for entries that will point you in the right direction to help resolve the problem.
Caveats/Disclaimer
Although this is a supported configuration with CAPI, you should be aware that you are giving up etcd management/upgrades via CAPI with this arrangement. It will fall on you, the cluster operator, to manage etcd upgrades and the lifecycle of the etcd cluster. Because you’re also leveraging existing AWS infrastructure, you’re also giving up CAPI management of that infrastructure. These may be acceptable trade-offs for your specific use case, but be sure to take that into consideration.
How I Tested
To test this procedure, I used Pulumi to create some existing AWS infrastructure, in according with the upstream project’s guidelines for using existing AWS infrastructure. In that existing infrastructure, the Pulumi code created three instances that I then set up as an etcd cluster using kubeadm (using a variation of these instructions). Once the external etcd cluster had been established, then I proceeded with testing the configuration and procedure outlined above.
Please don’t hesitate to contact me if you have any questions, or if you spot an error in this post. All constructive feedback is welcome! You can contact me on Twitter, or feel free to contact me via the Kubernetes Slack instance.
…
91% of businesses already using or planning to adopt AR or VR technology
CloudEnterprises surveyed by Grid Raster noted a boost in productivity and cost savings after implementation. …
Read More91% of businesses already using or planning to adopt AR or VR technology
VMware Cloud Director App Launchpad 1.0 Step by Step Installation
Cloud, VMware Cloud, vCloud DirectorAs the VMware Cloud Director has been evolving and adding features to help cloud providers better serve their customers, and help them target new markets, we are introducing Cloud Director App Launchpad bundled as a free extension with Cloud Director 10.1. App Launchpad will allow the Cloud provider to offer a Curated catalog of applications for vCD tenants, where they can deploy applications at a single click. The end user will be able to deploy these applications without having to understand the infrastructure it hosts it, networking configuration, firewall, Public IP or so on, as all of it will be automated in the back-end.
As VMware had acquired Bitnami (The leader of Packaged Open Source Applications) a bit over a year ago, it makes no surprise to anyone that Bitnami provides over ~130 Open Source Applications ready to serve to end tenants in the Cloud Director App Launchpad. That’s not all, as beyond the Bitnami, Cloud Director App Launchpad support another two sources for applications:
- ISV apps from VMware Cloud Marketplace
- In-house apps
This will allow our cloud providers to have a nicely populated catalog from day one as well the freedom to add their own custom offering to it. Before starting on the installation instructions, the below screenshot demonstrate what the marketplace of App Launchpad look like:

Now you know how nice it looks like, let’s get started on how to get it up and running.… Read More
…
Read MoreVMware Cloud Director App Launchpad 1.0 Step by Step Installation
Case study: How to quickly build and deliver a cloud-agnostic product
Cloud Case Studies, storageKonrad Lukasik is head of technical architecture at Objectivity. Sponsored Two years ago, Objectivity was asked to help deliver a new digital ecosystem. The client had a significant market share
Continue ReadingCase study: How to quickly build and deliver a cloud-agnostic product
Cloud computing: This new tool from AWS could make life easier for developers
CloudAppFlow has been designed to remove the need for manually-configured data transfers between AWS and services like Salesforce, Slack, Trend Micro and Zendesk. …
Read MoreCloud computing: This new tool from AWS could make life easier for developers
Using Existing AWS Security Groups with Cluster API
CloudI’ve written before about how to use existing AWS infrastructure with Cluster API (CAPI), and I was recently able to help update the upstream documentation on this topic (the upstream documentation should now be considered the authoritative source). These instructions are perfect for placing a Kubernetes cluster into an existing VPC and associated subnets, but there’s one scenario that they don’t yet address: what if you need your CAPI workload cluster to be able to communicate with other EC2 instances or other AWS services in the same VPC? In this post, I’ll show you the CAPI functionality that makes this possible.
One of the primary mechanisms used in AWS to control communications among instances and services is the security group. I won’t go into any detail on security groups, but this page from AWS provides an explanation and overview of how security groups work.
In order to make a CAPI workload cluster able to communicate with other EC2 instances or other AWS services, you’ll need to somehow use security groups to make that happen. There are at least two—possibly more—ways to accomplish this:
- You could add other instances or services to the CAPI-created security groups. The Cluster API Provider for AWS (CAPA) automatically creates a set of security groups for use by the EC2 instances and ELBs created when it bootstraps a workload cluster. You could manually add other instances or services into these groups to enable connectivity, but this approach is fraught with problems (not the least of which is coupling two things with different lifecycles).
- You could add CAPI-created instances to pre-existing security groups. This solves the problem of splitting the lifecycle of the workload cluster from the connectivity needs of workloads outside the cluster, but how do you have this happen automatically when new workload clusters are provisioned? Any sort of manual operations just won’t scale, and will become a potential source of failures and outages down the road.
Fortunately for us and for this blog post, CAPI provides an answer to how to take approach #2 above and automate it. In reviewing the API specification for CAPA v1alpha3, I noted that the AWSMachine spec includes a way to reference additional AWS security groups.
For example, in an AWSMachine spec, you could do the following:
spec: additionalSecurityGroups: - id: <existing_security_group_id>
This will result in an EC2 instance that joins the specified existing security group in addition to the CAPI-managed security groups. This provides exactly the behavior you’re seeking—the ability to have CAPI-managed resources automatically join an existing security group to enable access to other EC2 instances or AWS services.
Since the AWSMachine spec is also used by an AWSMachineTemplate, and since AWSMachineTemplates are used by both MachineDeployments (to manage groups of worker nodes) and the new v1alpha3 KubeadmControlPlane (to manage multiple control plane nodes as a single entity), this means we can leverage the same functionality for AWSMachineTemplates:
spec: template: spec: additionalSecurityGroups: - id: <existing_security_group_id>
Adding the snippet of YAML above to an AWSMachineTemplate definition would end up with multiple EC2 instances—managed by either a MachineDeployment or the KubeadmControlPlane—that automatically join the specified, pre-existing security group. To specify multiple security groups, you just add more entries under additionalSecurityGroups.
Therefore, if you have an existing VPC (and all associated subnets, gateways, routes, etc.) that you want to use with CAPI, and there are existing EC2 instances and other workloads in that VPC that need to communicate with the CAPI workload cluster, using this additionalSecurityGroups field in your AWSMachine and AWSMachineTemplate definitions can accomplish exactly what you need.
I hope this information is useful. If you have questions, I would love to try to help. Find me on Twitter, or hit me up on the Kubernetes Slack instance. I’m also open to hear any feedback or suggestions for improvement. Thanks!
…
Read MoreUsing Existing AWS Security Groups with Cluster API


