18
Feb, 2020
Setting up K8s on AWS with Kubeadm and Manual Certificate Distribution
CloudCredit for this post goes to Christian Del Pino, who created this content and was willing to let me publish it here.
The topic of setting up Kubernetes on AWS (including the use of the AWS cloud provider) is a topic I’ve tackled a few different times here on this site (see here, here, and here for other posts on this subject). In this post, I’ll share information provided to me by a reader, Christian Del Pino, about setting up Kubernetes on AWS with kubeadm but using manual certificate distribution (in other words, not allowing kubeadm to distribute certificates among multiple control plane nodes). As I pointed out above, all this content came from Christian Del Pino; I’m merely sharing it here with his permission.
This post specifically builds upon this post on setting up an AWS-integrated Kubernetes 1.15 cluster, but is written for Kubernetes 1.17. As mentioned above, this post will also show you how to manually distribute the certificates that kubeadm generates to other control plane nodes.
What this post won’t cover are the details on some of the prerequisites for making the AWS cloud provider function properly; specifically, this post won’t discuss:
- Setting node hotnames to match EC2’s Private DNS entry
- Creating and assigning the IAM instance profile to allow access to the AWS APIs
- Assigning the correct tags to AWS resources as required by the AWS cloud provider
I encourage you to refer to the link above for details on these specific areas.
Now, let’s take a look at the process that Christian tested and shared with me.
Bootstrapping the First Control Plane Node
To build the first control plane node, you need to setup a configuration file that will be used by kubeadm to perform an initialization. (Again, this assumes all necessary prerequisites described above have been satisfied.) Here is a sample configuration file:
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer: extraArgs: cloud-provider: aws
clusterName:
controlPlaneEndpoint: cp-nlb.elb.us-east-2.amazonaws.com
controllerManager: extraArgs: cloud-provider: aws configure-cloud-routes: "false"
kubernetesVersion: v1.17.0
networking: podSubnet: 192.168.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration: kubeletExtraArgs: cloud-provider: aws
The explanation of this configuration file is already pretty well-covered in this earlier post.
Once you have the configuration file set to your liking, run the
following command to setup your first control plane:
kubeadm init --config=init.yaml
(Replace init.yaml with the correct filename, of course.)
Assuming the process of initializing the first control plane node works as expected, then the next step is to install a CNI plugin, as documented here.
To confirm that your first control plane is up and ready, just run this command:
kubectl get nodes
If this command completes successfully and shows the first control plane node as “Ready,” then you are now ready to add more control plane nodes. However, since the --upload-certs flag was not specified during the kubeadm init command, the certificates generated on the first control plane node will not be shared across all the control plane nodes automatically. The next section explains how to manually copy the appropriate files across to additional control plane nodes.
Manually Distributing Certificates
There are a couple different reasons why a user might prefer to manually distribute certificates. First, the encryption key used by the --upload-certs flag expires after two hours. Therefore, it has been more than two hours, you need to either run kubeadm init phase upload-certs or manually copy the certificates across to other nodes. Second, you may want to use automation tools to deploy your cluster. One way to copy the certificates is to use ssh and scp, as described in the kubeadm documentation here.
You could also use S3 as a means of distributing certificates (more on that shortly).
Regardless of the method, here is the exact list of certificates that need to be copied over from the first control plane node:
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
If you’d like to use scp, then simply copy these files across to the EC2 instance(s) you’d like to use as additional control plane nodes.
Since this cluster will be using the AWS cloud provider, EC2 instances have to be booted with an IAM instance profile that enables access to the AWS APIs. If that IAM instance profile includes S3 permissions, and if you can install the AWS CLI tools on the EC2 instances, you can use the aws s3 cp command to upload the certificates to a bucket of your choice (just ensure the bucket permissions are appropriately secured). Once the certificates are uploaded to S3 (or copied over), you can proceed to add more control plane nodes.
Adding Additional Control Plane Nodes
Before adding more control planes, it is necessary to pull down the
certificates from the S3 bucket using the aws s3 cp command into the
/etc/kubernetes/pki directory (if you didn’t copy them across directly using scp). You must keep the same structure as in the first control plane:
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
Once the certificates are in place, you can build the configuration
file for your joining control planes:
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery: bootstrapToken: token: 123456.i2v5ub39hrvf51j3 apiServerEndpoint: cp-nlb.elb.us-east-2.amazonaws.com caCertHashes: - "sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"
nodeRegistration: name: ip-10-10-100-155.us-east-2.compute.internal kubeletExtraArgs: cloud-provider: aws
controlPlane: localAPIEndpoint: advertiseAddress: 10.10.100.155
Refer back to the earlier post for information on the parameters in this configuration file. The key difference here is that the configuration file does not include the certificateKey parameter since the original kubeadm init command did not include the --upload-certs flag.
Once the file is complete, proceed to run the following to join your
additional control plane nodes:
kubeadm join --config=cp-join.yaml
(Replace cp-join.yaml with the correct filename you used.)
Once run, the new control plane node should join the cluster. You can confirm that the additional control planes have joined the cluster by running kubectl get nodes and make sure the additional nodes are reporting as “Ready”.
Adding Worker Nodes
Adding worker nodes is simple. There are no certificates to copy and the
kubeadm configuration to join the cluster is similar to that of the
joining control plane nodes. Here is a sample:
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery: bootstrapToken: token: 123456.i2v5ub39hrvf51j3 apiServerEndpoint: "cp-nlb.elb.us-east-2.amazonaws.com" caCertHashes: - "sha256:082feed98fb5fd2b497472fb7d9553414e27ff7eeb7b919c82ff3a08fdf5782f"
nodeRegistration: name: ip-10-10-100-133.us-east-2.compute.internal kubeletExtraArgs: cloud-provider: aws
As outlined here, you’ll need to be sure your version of this file uses the correct bootstrap token, the correct API server endpoint, the correct CA certificate hash, and the correct node name. Once the file is complete, you will run the following to join your worker nodes:
kubeadm join --config=worker-join.yaml
(Substitute the filename you used for worker-join.yaml in the above command.)
Upon running this command, the worker node should join the cluster. Once again, you can use kubectl get nodes to verify that the node(s) have joined the cluster.
Summary
As you can see above, the process that Christian shared with me is extremely similar to what I shared here, with the exception of manually distributing the certificates (and Christian mentions a couple very good reasons above why manual certificate distribution may be preferred). This post also shows that the process for establishing an AWS-integrated Kubernetes cluster is consistent for versions 1.15, 1.16, and 1.17 (you just need to change the kubernetesVersion value in the configuration file you use to bootstrap the first control plane node).
I hope this information is helpful. Thanks to Christian for sharing this information with me and allowing me to publish it here. If you have any questions, feel free to hit me on Twitter or find me on the Kubernetes Slack instance.
…
Read MoreSetting up K8s on AWS with Kubeadm and Manual Certificate Distribution
17
Feb, 2020
VMware Cloud Provider Pod 1.5.0.x Update is here
VMware VariousProduct Versions Deployed with Cloud Provider Pod had been updated to include vCloud Director 10.0. Below is the full Version changes:
- VMware ESXi and vCenter Server are updated to 6.7 Update 3
- VMware NSX for vSphere is updated to 6.4.6
- VMware vCloud Director is updated to 10
- VMware vRealize Network Insight is updated to 5.0
- VMware vRealize Operations Manager is updated to 8.0
- VMware vRealize Operations Manager Multi-Tenant App is updated to 2.3
- VMware vRealize Operations Manager for NSX
- VMware vRealize Operations Manager – Cloud Pod Management Pack is updated to 3.0
· You must
get a new deployment guide from the Cloud Provider Pod designer and follow the
instructions included in it. The instructions are changed, now you must
manually upload the products.json in the Cloud Provider Pod
appliance to get the new product versions.
· There is a critical fix in the Drivers and
Tools section of the Cloud
Provider Pod page to resolve an issue with the download of required
deployment software. See the known issues for more details.
As there is plenty of improvements and new features in vCloud Director
10.0, if you were waiting to stand up your setup with CPOD, now you can do it
with vCloud Director 10.0 which has an enhanced CPOM, near parity HMTL5
Provider Portal, and many more features.… Read More
…
