At GlueOps, we have been working on an internal tool to deploy and manage Kubernetes clusters across cloud providers and datacenters. During development, we ran into a few caveats. For example, if we modify a ‘prepare-node’ role to install an additional package or remove a package that seems unnecessary, it can indirectly affect the subsequent roles.
Example of one of the use cases, in the prepare-nodes playbook, we tend to remove a package that was thought unnecessary or change a package’s version, it can result in the subsequent roles like kubeadm-init to fail.
So we decided to add a couple of tests to help keep our work consistent and maintainable.

Some of the tests we created were:

• Scaling node(control-plane,data-plane)
• Rotate Etcd certs
• OS security patches
• Kubernetes versions upgrade

GlueKube Overview

As we deploy many production-grade clusters across numerous cloud providers (e.g. AWS, GCP, metal) for our customers, we wanted to make our deployment more agnostic. We took a look at a couple of existing tools like Kubespray, but we felt it may be a burden to maintain and modify the existing codebase in case we want to customize it.
Ultimately, we decided to build GlueKube: a platform to create kubernetes clusters agnostic to cloud providers with kubeadm, ansible.

GlueKube capabilities:

• Deploying a stacked Etcd cluster where the Etcd is embedded within control-plane nodes unlike the external-etcd where Etcd where it can in separate nodes.
• Supporting two load balancers (e.g. for the control plane and business applications)
• Scaling Up/Down nodes(control-plane and worker nodes)
• Tainting and Labeling nodes
• Rotating certs
• Applying OS security patches
• Upgrading cluster versions

Molecule as a Testing tool

We’re using ansible to configure our clusters, we needed a testing tool that’s compatible with ansible and supports managing Hetzner resources for testing.

As our case is more of integration test than unit test, we found Molecule a more suitable option than ansible-test, as it provides a structured way through sequences to create/test/destroy infrastructure.

After our research and experiments, we created this blog to help anyone else considering similar tradeoffs.

In this post, we’re focusing on our journey with molecule. If you’re interested in learning more about the project, check out the link.

Test Case I: Scaling Down Worker Nodes

We started with scaling down worker nodes because it is easier than working with control-plane nodes.

After implementing this workflow in GlueKube scaling down is as easy as removing the node from the inventory file hosts.yaml and applying the sync-resources.yaml file, like the demo below:

(demo)[https://drive.google.com/file/d/1RU4s2q8Gv-ef3Y7tRKoHvARPFxGUmAhF/view?usp=sharing]

Now how do we test that with Molecule ? after we created the scenario(test suite) using: molecule init scenario scaler-cluster.
We changed the scenario property in molecule.yaml to the following:

scenario:
  name: scale-cluster
  test_sequence:
    - create
    - converge
    - verify tests/test_cluster_healthy.yaml

To explain it :

• Create: for creating the required resources for testing, our cluster will usually consist of 3 master planes, 3 worker nodes, 1 loadbalancer (HAProxy in our case).
• Converge: it’s the file we will execute to transform resources into specific state
• Verify: a file to run some tests after the resources get converged. The simplest test will run at first is verifying cluster health.
• Destroy: at the end we need to cleanup the resources, this file will usually do the contrary of what create sequence does. One side note is this file will always run whenever the tests result (success || failed).

---
- name: Verify Kubernetes Nodes Status
  hosts: master-node-1
  become: true
  gather_facts: true
  tasks:
    - name: Get Kubernetes Nodes Status
      ansible.builtin.shell: |
        export KUBECONFIG=/opt/kubernetes/.kube/config
        kubectl get nodes --no-headers
      register: kube_nodes
      changed_when: false

    - name: Debug Node Status Output
      ansible.builtin.debug:
        msg: "{{ kube_nodes.stdout_lines }}"

    - name: Fail if any node is not Ready
      ansible.builtin.fail:
        msg: "One or more Kubernetes nodes are NOT in Ready state!"
      when: "' Ready ' not in item"
      loop: "{{ kube_nodes.stdout_lines }}"

GlueKube relies heavily on inventory/hosts.yaml to know the desired state of the cluster, think of it the same as terraform state and for our tests to run we will need one.

And this will lead us into Create.yaml that’s responsible for generating the hosts.yaml file from the created test resources. So we modified molecule.yaml to use the generated hosts.yaml with the following code:

provisioner:
 name: ansible
 inventory:
   links:
     hosts: inventory/hosts.yaml

Note: group_vars contains ansible configurations for each group in hosts.yaml.
For running a basic verification, we used the following command:

molecule test -s scale-cluster

This will trigger all the sequences we declared on molecule.yaml

Testing the scale down

To scale resources down, we need to remove the desired node from hosts.yaml, our initial thought of the process was creating a python script, give it the desired node, remove it from hosts.yaml and then refresh the inventory cache. However, we wanted to keep our test more Ansible oriented.
We found a better solution (at least for us) by creating two initial hosts.yamlfiles: the first one with all the nodes in and the second one without one of the worker nodes.
We used slicing to pick the [:1] from worker_nodes list, here is a code snippet from create.yaml:

- name: Store inventory(for scale down worker node)
      ansible.builtin.copy:
        content: |
          all:
            children:
              workers:
                hosts:
                  {% for node in worker_nodes.results[:1] -%}
                    worker-node-{{ node.item + 1 }}:
                      ansible_host: {{ node.hcloud_server.ipv4_address }}
                      ip: 10.0.0.3{{ node.item + 1 }}
                      ansible_user: cluster
                      ansible_ssh_private_key_file: keys/k8s_cluster
                      extra:
                        taints:
                          - node-role.kubernetes.io/control-plane:NoSchedule-
                  {% endfor +%}
        dest: "./inventory/scale-down-worker.yaml"
        mode: "0644"

Molecule has another sequence called side_effect, which we used to replace the hosts.yaml with scale-down-worker.yaml contents, refresh the inventory cache and do the syncing process, here is the code.

remove_worker_node.yaml

- name: Copy and rename a file
  hosts: localhost
  tasks:
    - name: Copy current host
      ansible.builtin.copy:
        src: ../inventory/hosts.yaml
        dest: ../inventory/hosts.old.yaml
        mode: "0644"
    - name: Add new host file
      ansible.builtin.copy:
        src: ../inventory/scale-down-worker.yaml
        dest: ../inventory/hosts.yaml
        mode: "0644"
        force: true


    - name: Refresh inventory to ensure new instances exist
      ansible.builtin.meta: refresh_inventory

- name: Sync Cluster
  ansible.builtin.import_playbook: ../../../playbooks/sync-resources.yaml

For Molecule to recognize the side_effect, we added it alongside the other sequences.

scenario:
  name: scale-cluster
  test_sequence:
    - create
    - converge
    - verify tests/test_cluster_healthy.yaml
    - side_effect side_effect/remove_worker_node.yaml
    ...

After the side_effect sequence gets executed, we should verify the expected state of the cluster, in our case the side_effect remove_worker_node should reduce the number of worker nodes by 1, so our test will count how many worker nodes we currently have.
Here is a code example:

---
- name: Verify Kubernetes Nodes Status
  hosts: masters[0]
  become: true
  gather_facts: true
  vars:
    desired_node: 1
  tasks:
    - name: Get Kubernetes nodes
      ansible.builtin.shell: |
        export KUBECONFIG=/opt/kubernetes/.kube/config
        kubectl get nodes --no-headers | grep worker
      register: kube_nodes
      changed_when: false

    - name: Count the number of nodes
      ansible.builtin.set_fact:
        node_count: "{{ kube_nodes.stdout_lines | length }}"

    - name: Assert that the number of nodes is equal to {{ desired_node }}
      ansible.builtin.assert:
        that:
          - node_count | int == desired_node | int
        fail_msg: "Expected {{ desired_node }} nodes, but found {{ node_count }}."
        success_msg: "Cluster has the expected number of nodes: {{ node_count }}"

Then we add the following test in molecule.yaml:

scenario:
  name: scale-cluster
  test_sequence:
    - create
    - converge
    - verify tests/test_cluster_healthy.yaml
    - side_effect side_effect/remove_worker_node.yaml
    - verify tests/test_scale_down_worker.yaml 
    ...

Summary

In this post, we shared our experience setting up integration tests for Kubernetes cluster management using Molecule and Ansible. We focused on a specific test case: scaling down worker nodes, illustrating how Molecule’s sequences like `create`, `converge`, `verify`, and `side_effect` can be orchestrated to achieve this. We also highlighted the importance of the Ansible inventory in defining the desired state of the cluster and how Molecule facilitates testing changes to this inventory. This approach allows us to maintain the reliability and consistency of our GlueKube platform as we continue to develop and enhance its capabilities for deploying and managing Kubernetes clusters across diverse environments.