• Search for:
    25 Nov, 2020
    Pausing Cluster API Reconciliation
    Cloud

    Cluster API is a topic I’ve discussed here in a number of posts. If you’re not already familiar with Cluster API (also known as CAPI), I’d encourage you to check out my introductory post on Cluster API first; you can also visit the official Cluster API site for more details. In this short post, I’m going to show you how to pause the reconciliation of Cluster API cluster objects, a task that may be necessary for a variety of reasons (including backing up the Cluster API objects in your management cluster).

    Since CAPI leverages Kubernetes-style APIs to manage Kubernetes cluster lifecycle, the idea of reconciliation is very important—it’s a core Kubernetes concept that isn’t at all specific to CAPI. This article on level triggering and reconciliation in Kubernetes is a great article that helps explain reconciliation, as well as a lot of other key concepts about how Kubernetes works.

    When reconciliation is active, the controllers involved in CAPI are constantly evaluating desired state and actual state, and then reconciling differences between the two. There may be times when you need to pause this reconciliation loop. Fortunately, CAPI makes this pretty easy: there is a paused field that allows users to pause the reconciliation loop (see here in the v1alpha3 CAPI reference).

    This field is optional, which you can observe using kubectl with an existing workload cluster. For example, here is the relevant output from kubectl get cluster workload1 against a CAPI management cluster; note the paused field is not present (i.e., reconciliation is currently active):

    spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 controlPlaneEndpoint: host: workload1-apiserver-1234567890.us-east-2.elb.amazonaws.com port: 6443 controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1alpha3 kind: KubeadmControlPlane name: workload1-control-plane namespace: workload1 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3 kind: AWSCluster name: workload1 namespace: workload1

    Reconcilation is active when the field isn’t present; setting the field to false is essentially the same as removing the field. To pause reconciliation, set it to true. One way of doing this is using kubectl patch, like this:

    kubectl path cluster workload1 --type merge \
-p '{"spec":{"paused": true}}'

    Naturally, you’d want to change workload1 to the name of the workload cluster for which you want reconciliation paused. After running the command, you can then run kubectl get cluster <cluster-name> -o yaml and verify the paused field is present:

    spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 controlPlaneEndpoint: host: workload1-apiserver-1234567890.us-east-2.elb.amazonaws.com port: 6443 controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1alpha3 kind: KubeadmControlPlane name: workload1-control-plane namespace: workload1 infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3 kind: AWSCluster name: workload1 namespace: workload1 paused: true

    If you check the logs for the CAPI controllers, you’ll see messages like this:

    I1125 17:31:40.287717 1 machine_controller.go:170] controllers/Machine "msg"="Reconciliation is paused for this object" "machine"="workload1-md-0-58c6df55bc-cqt86" "namespace"="workload1"

    When reconciliation is paused, the CAPI controllers will not make changes to the workload cluster in order to reconcile differences between actual state and desired state.

    When you’re ready to resume reconciliation again, just change the field to false:

    kubectl path cluster workload1 --type merge \
-p '{"spec":{"paused": false}}'

    This will remove the field from the spec, allowing the CAPI controllers to resume reconciliation of the CAPI-related objects (like Clusters, Machines, and MachineDeployments).

    As I mentioned at the start of this post, pausing CAPI reconciliation may be necessary in a few different situations. (Backing up Cluster API is one situation.)

    If anyone has any questions, or if you spot an error in this post, please let me know. Hit me on Twitter; I’d love to hear from you!

    …

    Read MorePausing Cluster API Reconciliation

    25 Nov, 2020
    Technology Short Take 134
    Cloud

    Welcome to Technology Short Take #134! I’m publishing a bit early this time due to the Thanksgiving holiday in the US. So, for all my US readers, here’s some content to peruse while enjoying some turkey (or whatever you’re having this year). For my international readers, here’s some content to peruse while enjoying dramatically lower volumes of e-mail because the US is on holiday. See, something for everyone!

    Networking

    Security

    • I’m glad to see this. Open source has become so critical to so many aspects of our computing infrastructure.
    • OpenCSPM looks like it could be quite a useful tool. I haven’t yet had time to dig in and get familiar with the details, but what I have seen so far looks good.
    • Uh oh…more hardware exploits.
    • The macOS OCSP fiasco generated quite a bit of attention and analysis (see here and here).

    Cloud Computing/Cloud Management

    Operating Systems/Applications

    • One of the things I love about Matt Oswalt is that he exemplifies the idea of a perpetual learner. The latest example is Matt’s post on the anatomy of a binary executable, in which he dives deep into what exactly it means to be a binary executable file. Good stuff!
    • You may have heard of eBPF, the Linux technology that is reshaping Linux applications (and in some ways reshaping Linux itself). Brendan Gregg discusses the future of BPF binaries, made possible through BTF and CO-RE. The idea of creating ELF binaries (don’t know what that is? See the previous bullet!) for BPF is pretty cool, in my opinion, and has the potential to unlock a lot of innovation in this space.
    • Something about Linux, and Fedora in particular, just keeps drawing me back. If you’re in a similar boat, and you’re looking for information on how to get Firefox on Fedora to play H.264 videos, Leo Chavez has some information that should help.
    • Here’s a bit of history on macOS, for those of us interested in such things.
    • This looks horribly confusing. It almost feels like we are well into Windows Registry territory here.
    • What a time to be alive: Microsoft has its own Linux distribution.
    • Whether it be dissatisfaction with macOS 11 “Big Sur,” or unhappiness at the direction of their hardware (there’s some discussion that the new M1 chips don’t support eGPUs), or concerns over privacy given the recent issues with OCSP and macOS “dialing home,” I’m seeing folks leaving macOS for other platforms (mostly Linux). Preslav Rachev shares his story here, and Juan Diego Caballero shares his story here.

    Storage

    Virtualization

    That’s all this time, but hopefully it’s enough! If you have suggestions for content to include in a future Technology Short Take, or if you’d just like to catch up and say hello, feel free to contact me on Twitter. Enjoy the rest of your week!

    …

    Read MoreTechnology Short Take 134