Welcome to Technology Short Take #118! Next week is VMworld US in San Francisco, CA, and I’ll be there live-blogging and meeting up with folks to discuss all things Kubernetes. If you’re going to be there, look me up! Otherwise, I leave you with this list of links and articles from around the Internet to keep you busy. Enjoy!

Networking

• Networking guru Ivan Pepelnjak has migrated his online presence to AWS; read more here.

Servers/Hardware

• Interesting (but otherwise not terribly useful) article on how to turn a MacBook into a touchscreen. Lack of a touch screen remains the MacBook line’s second most egregious shortcoming against competing products (the first being the awful keyboard).

Security

• I came across this article on tools and methods for auditing Kubernetes RBAC policies, which had some new tools I hadn’t seen before.
• This web developer security checklist has a nice list of some “current recommended practices” for properly securing web applications/sites.
• Daniele Antonioli, Nils Ole Tippenhauer, and Kasper Rasmussen discuss the KNOB attack (a Bluetooth vulnerability).

Cloud Computing/Cloud Management

• My teammate John Harris has a great post on ensuring least privilege in Kubernetes using impersonation. Good stuff here.
• Alex Brand (also a teammate!) has a post on using Sonobuoy to test Kubernetes Network Policy enforcement (this is to ensure that your installed CNI plugin has the abilitiy to enforce Netowrk Policies).
• I’ve been using Pulumi quite a bit to help with managing infrastructure environments (spinning up and tearing down test environments). I then found this article by Lee Briggs on using Pulumi for Kubernetes configuration management, and it has me interested to continue to explore Pulumi’s capabilities. (Oh, and while we’re talking about Pulumi, here’s another article on using Pulumi for infrastructure as code.)
• Gary Stafford has a fairly in-depth article on managing AWS infrastructure using Ansible, CloudFormation, and CodeBuild.

Operating Systems/Applications

• Here’s how to create a snippet in VS Code to wrap text with some other text of your choosing. This is handy—I had some similar things set up for Sublime Text, but haven’t managed to do the same for VS Code yet.
• Network bonding in Linux—as outlined in this article—is something I may have to explore. Right now I have a custom shell script I wrote that “docks” and “undocks” my laptop by manipulating, among other things, some associated network settings. What I’d really like to have is something like ControlPlane (née Marco Polo for the longtime Mac users), but for Linux. Alas, it appears that even ControlPlane has been abandoned…
• Major Hayden explains how to use Fedora 30 on GCE.
• jq, awk, and sed. Learn them, use them, love them.
• Ivan Velichko journals his journey of learning about containerization and orchestration in this blog post. If nothing else, this post at least helps clarify the relationship between commonly-mentioned projects like runC, containerD, cri-o, and others.
• Youichi Fujimoto discusses running Kubernetes locally with KinD and Docker. I’ve found KinD to be easier to use and more flexible than Minikube, but I’m on Linux and have a native Docker installation.
• Tim Little shares some “lessons learned” regarding increasing resilience in Kubernetes, such as taking advantage of features like Pod Anti-Affinity, scaling up Deployments to have more than a single Pod, and configuring Readiness probes for all workloads.

Storage

• Via this blog post by Myles Gray, VMware has announced Cloud Native Storage (CNS), powered by a new CSI (Container Storage Interface) driver for Kubernetes (and other orchestration platforms that use CSI).
• When it comes to storage, Cormac Hogan is an invaluable resource. It’s great to see Cormac turning his attention to Kubernetes environments, and in a recent post Cormac tackled failure scenarios with Kubernetes storage on vSphere.
• Laurens van Dujin ferrets out a small detail regarding network connectivity while troubleshooting vSAN Witness Node isolation.

Virtualization

• Joey Ketels steps readers through how to set up RHEL7 desktops with Horizon Instant Clones.

Career/Soft Skills

• Via the vExpert Slack instance, I came across this article on the effect of the scarcity mentality in IT careers.
• Silvia Botros has a great piece of how we keep learning. There’s some good stuff in here, I highly recommend reading it.
• Although written for the creative industry, I think this article by Paul Jun on five things no one tells you about going full-time again probably holds equally true for IT freelancers moving back into corporate life, or even moving from startup culture to big company culture after an acquisition.

I guess that’ll have to do for now. It is my sincere hope that you’ve found something useful in this post, and if you have any feedback I invite you to contact me on Twitter. Thanks for reading!

…

Read MoreTechnology Short Take 118

As I mentioned back in May in this post on creating a sandbox for learning Pulumi, I’ve started using Pulumi more and more of my infrastructure-as-code needs. I did switch from JavaScript to TypeScript (which I know compiles to JavaScript on the back-end, but the strong typing helps a new programmer like me). Recently I had a need to create some resources in AWS using Pulumi, and—for reasons I’ll explain shortly—many of the “canned” Pulumi examples didn’t cut it for my use case. In this post, I’ll share how I created tagged subnets across AWS availability zones (AZs) using Pulumi.

In this particular case, I was using Pulumi to create all the infrastructure necessary to spin up an AWS-integrated Kubernetes cluster. That included a new VPC, subnets in the different AZs for that region, an Internet gateway, route tables and route table associations, security groups, an ELB for the control plane, and EC2 instances. As I’ve outlined in my latest post on setting up an AWS-integrated Kubernetes 1.15 cluster using kubeadm, these resources on AWS require specific AWS tags to be assigned in order for the AWS cloud provider to work.

As I started working on this, I found examples of Pulumi TypeScript code that could do part of what I needed, but not all of it:

• If you’re using the awsx module with Pulumi (part of their “Crosswalk” stuff) to create a VPC, it can handle creating subnets across multiple AZs automatically for you. Users have the ability to fine-tune some of the settings, but—here’s the kicker—tags you specify at the VPC level don’t get propagated. (There’s an issue open for this.)
• If you build it all manually, then you also have to manually deal with figuring out how many AZs are in a region, and how to loop through the AZs to create subnets in each one (which is stuff the awsx module handles for you).

Since I needed the tags—the AWS cloud provider for Kubernetes won’t work otherwise—I set out to take the second path (building it all out manually). I’m going to share the TypeScript code I used—and am still using—so that others who may find themselves in the same boat have an example upon which to base their code.

(I’m a relative newcomer to TypeScript, so be gentle!)

First, I needed to figure out how many AZs were in the given region. This snippet of code returned both the list of AZ names as well as the total number of AZs in the region:

const rawAzInfo = aws.getAvailabilityZones({ state: "available",
});
let azNames: Array<string> = rawAzInfo.names;
let numberOfAZs: number = azNames.length;

I gathered both the list of AZ names as well as the overall count because in the snippet of code for creating subnets, I’ll need the AZ names.

Next, I created a VPC (this part is straightforward and reasonably well-documented):

const vpc = new aws.ec2.Vpc("k8s-vpc", { cidrBlock: "10.1.0.0/16", enableDnsHostnames: true, enableDnsSupport: true, tags: { Name: "k8s-vpc", [k8sTagName]: "shared", },
});

The one part here that wasn’t immediately obvious to me was the [k8sTagName] syntax for referencing a variable named k8sTagName. This allow me to declare the tag (i.e., “kubernetes.io/cluster/cluster-name”) once in a variable instead of multiple times in the code.

Now we get to the more interesting stuff—iterating through the list of AZs to create a subnet in each. After stumbling around for a bit, I finally arrived here:

let subnets = [];
for (let i = 0; i < numberOfAZs; i++) { let subnetAddr: number = i*16; let netAddr: string = "10.1."; let cidrSubnet: string = netAddr.concat(String(subnetAddr), ".0/20"); subnets.push(new aws.ec2.Subnet(`subnet-${i+1}`, { availabilityZone: azNames[i], cidrBlock: cidrSubnet, mapPublicIpOnLaunch: true, vpcId: vpc.id, tags: { Name: `subnet-${i+1}`, [k8sTagName]: "shared", }, }));
};

So what does this code do?

1. First, it creates an array to hold the aws.ec2.Subnet objects I’m going to create in the loop.
2. Next, it jumps into a for loop that iterates over the total number of AZs in the region (hence why I collected that value earlier).
3. For each iteration, the code calculates a subnet address (as a multiple of 16), and then combines that subnet address with the network address and CIDR mask. Thus, the CIDR block for the first subnet is “10.1.0.0/20”, the second is “10.1.16.0/20”, and so on.
4. Finally, for each iteration, the push method adds (“pushes”) a new aws.ec2.Subnet object into the array with the appropriate properties (the VPC ID for the VPC created earlier, the calculated CIDR block, any other settings, and the necessary tags). Note the subnet-${i+1} syntax to calculate names like “subnet-1”, “subnet-2”, etc.

After this, the majority of the rest of the code is pretty straightforward (I won’t go into detail in this post since the examples and Pulumi API reference pretty much capture most everything needed):

• Create an Internet gateway in the VPC and tag it.
• Create a route table for the Internet gateway and tag it.
• Use a similar array and for loop construct to iterate across the subnets and associate each subnet (referenceable by subnets[i], assuming i is the loop variable) with the route table.

At this point, Pulumi has now created a VPC, subnets in each AZ, an Internet gateway, a route table to point to the Internet gateway, and a route table association linking each subnet to the route table. All you need now are security groups and EC2 instances, and you’re off to the races! All of the things that can be tagged are tagged, which means infrastructure provisioned with this code is ready for use with Kubernetes.

It’s more than fair to say that some of this is just me learning TypeScript; I can’t dispute that at all. If I’m searching for resources on how to do something, though, that’s probably a reasonable indicator that other folks are also looking for resources and examples, and why I wanted to publish this post.

I hope that this information is useful to someone; feel free to hit me up on Twitter if you have any questions, comments, corrections, or suggestions.

…

Read MoreCreating Tagged Subnets Across AWS AZs Using Pulumi

If you’ve used kubeadm to bootstrap a Kubernetes cluster, you probably know that at the end of the kubeadm init command to bootstrap the first node in the cluster, kubeadm prints out a bunch of information: how to copy over the admin Kubeconfig file, and how to join both control plane nodes and worker nodes to the cluster you just created. But what if you didn’t write these values down after the first kubeadm init command? How does one go about reconstructing the proper kubeadm join command?

Fortunately, the values needed for a kubeadm join command are relatively easy to find or recreate. First, let’s look at the values that are needed.

Here’s the skeleton of a kubeadm join command for a control plane node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \
--control-plane \
--certificate-key <certificate-key>

And here’s the skeleton of a kubeadm join command for a worker node:

kubeadm join <endpoint-ip-or-dns>:<port> \
--token <valid-bootstrap-token> \
--discovery-token-ca-cert-hash <ca-cert-sha256-hash> \

As you can see, the information needed for the worker node is a subset of the information needed for a control plane node.

Here’s how to find or recreate all the various pieces of information you need:

• The value for <endpoint-ip-or-dns>:<port> can be retrieved from the “kubeadm-config” ConfigMap in the cluster’s “kube-system” namespace. Just run kubectl -n kube-system get cm kubeadm-config -o yaml and then look for the value specified for controlPlaneEndpoint.
• For <valid-bootstrap-token>, there’s no way to find out the original token. It is, however, easy to create a new token. Just run kubeadm token create and note the output. Be aware that tokens have a default lifetime of 24 hours.
• For <ca-cert-sha256-hash>, see this blog post.
• For <certificate-key>, you can’t get whatever original value was output by kubeadm init. You can, though, generate another value. (This value is only good for two hours, so it’s far more likely you’d need to generate another value anyway.) Run kubeadm init phase upload-certs --upload-certs and make a note of the output of the command.

There you have it—how to easily reconstruct all the information needed to use kubeadm join to join a node (either control plane node or worker node) to a Kubernetes cluster. Note that this information also applies to gathering the information needed for a JoinConfiguration stanza in a kubeadm configuration file, like one used in this post to join control plane nodes or worker nodes to a cluster.

If I’ve missed anything here, feel free to contact me on Twitter. Thanks!

…

Read MoreReconstructing the Join Command for Kubeadm