Skip to content

Kubernetes


This Ansible role is designed to deploy a K3s Kubernetes cluster. After adding your configuration, the cluster will deploy and have a configured CNI (calico) and be in a state ready to use. This role can be used with our our playbooks or comes included, along with the playbook within our Ansible Execution Environment.

Features

  • CNI Setup

  • Configurable:

    • Container Registries

    • etcd snapshot cron schedule

    • etcd snapshot retention

    • Cluster Domain

    • Configure System reserved CPU, Storage and Memory.

    • Node Labels

    • Node Taints

    • Service Load Balancer Namespace

  • Encryption between nodes (Wireguard)

  • Firewall configured for kubernetes host

  • Multi-node Deployment

  • OpenID Connect SSO Authentication

  • Basic RBAC ClusterRoles and Bindings

  • ToDo-#5 Restore backup on fresh install of a cluster

  • Installs OLM for operator subscriptions

  • Install MetalLB

Role Workflow

For a more probable than not success this role first installs/configures prime master, other master(s) and worker nodes using the following simplified workflow:

  1. Download both install script and k3s binary to ansible controller

  2. copy install script and k3s binary to host

  3. Create required config files needed for installation

  4. (kubernetes prime master only) Add install required config files

  5. Install kubernetes

  6. (kubernetes prime master only) Wait for kubernetes to be ready. Playbook is paused until true

  7. Configure Kubernetes

If the playbook is setup as per our recommendation step 2 onwards is first done on master nodes then worker nodes.

Tip

If you prefer to manually restart the kubernetes service the following variables can be set to prevent a restart of the kubernetes service

nfc_kubernetes_no_restart: false
nfc_kubernetes_no_restart_master: false
nfc_kubernetes_no_restart_prime: false
nfc_kubernetes_no_restart_slave: false
See default variables below for explanation of each variable if it's not evident enough.

Default Variables

defaults/main.yaml
# Depreciated:
#      Calico is being migrated to use the calico operator.
#      in a near future release, this method of deploying calico
#      will be removed. use tag `operator_migrate_calico` to migrate
calico_image_tag: v3.25.0 # Depreciated
# EoF Depreciated
# SoF New Variables
nfc_kubernetes_calico_version: v3.27.0
nfc_kubernetes_tigera_operator_registry: quay.io
nfc_kubernetes_tigera_operator_image: tigera/operator
nfc_kubernetes_tigera_operator_tag: v1.32.3               # Calico v3.27.0
# EoF New Variables, EEoF Depreciated


nfc_kubernetes_enable_metallb: false
nfc_kubernetes_enable_servicelb: false


############################################################################################################
#
#                     Old Vars requiring refactoring
#
# ############################################################################################################
# KubernetesPodSubnet: 10.85.0.0/16
# KubernetesServiceSubnet: 10.86.0.0/16


Kubernetes_Prime: false                                    # Optional, Boolean. Is the current host the Prime master?
Kubernetes_Master: false                                   # Optional, Boolean. Is the current host a master host?

ContainerDioVersion: 1.6.20-1
KubernetesVersion: '1.26.2'                                # must match the repository release version
kubernetes_version_olm: '0.26.0'




KubernetesVersion_k8s_prefix: '-00'
KubernetesVersion_k3s_prefix: '+k3s1'

kubernetes_private_container_registry: []                  # Optional, Array. if none use `[]`

kubernetes_etcd_snapshot_cron_schedule: '0 */12 * * *'
kubernetes_etcd_snapshot_retention: 5

# host_external_ip: ''                                     # Optional, String. External IP Address for host.

kubernetes_type: k8s                                       # Mandatory, String. choice K8s | k3s

kube_apiserver_arg_audit_log_maxage: 2

kubelet_arg_system_reserved_cpu: 450m
kubelet_arg_system_reserved_memory: 512Mi
kubelet_arg_system_reserved_storage: 8Gi

# kubernetes_olm_install: true # optional, boolean. default=true


nfc_kubernetes:
  enable_firewall: true             # Optional, bool enable firewall rules from role 'nfc_firewall'

nfc_kubernetes_no_restart: false         # Set to true to prevent role from restarting kubernetes on the host(s)
nfc_kubernetes_no_restart_master: false  # Set to true to prevent role from restarting kubernetes on master host(s)
nfc_kubernetes_no_restart_prime: false   # Set to true to prevent role from restarting kubernetes on prime host
nfc_kubernetes_no_restart_slave: false   # Set to true to prevent role from restarting kubernetes on slave host(s)


k3s:
  files:

    - name: audit.yaml
      path: /var/lib/rancher/k3s/server
      content: |
        apiVersion: audit.k8s.io/v1
        kind: Policy
        rules:
        - level: Request
      when: "{{ Kubernetes_Master | default(false) }}"

    - name: 90-kubelet.conf
      path: /etc/sysctl.d
      content: |
        vm.panic_on_oom=0
        vm.overcommit_memory=1
        kernel.panic=10
        kernel.panic_on_oops=1
        kernel.keys.root_maxbytes=25000000

    - name: psa.yaml
      path: /var/lib/rancher/k3s/server
      content: ""
        # apiVersion: apiserver.conf0 */12 * * *ig.k8s.io/v1
        # kind: AdmissionConfiguration
        # plugins:
        # - name: PodSecurity
        #   configuration:
        #     apiVersion: pod-security.admission.config.k8s.io/v1beta1
        #     kind: PodSecurityConfiguration
        #     defaults:
        #       enforce: "restricted"
        #       enforce-version: "latest"
        #       audit: "restricted"
        #       audit-version: "latest"
        #       warn: "restricted"
        #       warn-version: "latest"
        #     exemptions:
        #       usernames: []
        #       runtimeClasses: []
        #       namespaces: [kube-system]
      when: "{{ kubernetes_config.cluster.prime.name == inventory_hostname }}"


#############################################################################################
# Cluster Config when stored in Inventory
#
# One required per cluster. recommend creating one ansible host group per cluster.
#############################################################################################
# kubernetes_config:                    # Dict. Cluster Config
#   cluster:
#     access:                           # Mandatory. List, DNS host name or IPv4/IPv6 Address.
#                                       # if none use '[]'
#       - 'my.dnshostname.com'
#       - '2001:4860:4860::8888'
#       - '192.168.1.1'
#     Name: earth                       # Mandatory, String. Cluster Name
#     group_name:                       # Mandatory, String. name of the ansible inventory group containg all cluster hosts
#     prime:
#       name: k3s-prod                  # Mandatory, String. Ansible inventory_host that will
#                                       # act as the prime master node.
#     networking:
#       encrypt: true                   # Optional, Boolean. default `false`. Install wireguard for inter-node encryption
#       podSubnet: 172.16.70.0/24       # Mandatory, String. CIDR
#       ServiceSubnet: 172.16.72.0/24   # Mandatory, String. CIDR
#                                       # Mandatory, String. Token to join nodes to the cluster
#   node_token: !vault |
#     $ANSIBLE_VAULT;1.2;AES256;kubernetes/cluster/production
#     {rest_of encrypted key}
#  oidc:                                                    # Used to configure Kubernetes with OIDC Authentication.
#    enabled: true                                          # Mandatory, boolen. speaks for itself.
#    issuer_url: https://domainname.com/realms/realm-name   # Mandatory, String. URL of OIDC Provider
#    client_id: kubernetes-test                             # Mandatory, string. OIDC Client ID
#    username_claim: preferred_username                     # Mandatory, String. Claim name containing username.
#    username_prefix: oidc                                  # Optional, String. What to prefix to username
#    groups_claim: roles                                    # Mandatory, String. Claim name containing groups
#    groups_prefix: ''                                      # Optional, String. string to append to groups

#     hosts: 

#       my-host-name:
#         labels:
#           mylabel: myvalue

#         taints:
#           - effect: NoSchedule
#             key: taintkey
#             value: taintvalue

About:

This page forms part of our Project Kubernetes Ansible Role.

Page Metadata
Version: ToDo: place files short git commit here
Date Created: 2023-10-24
Date Edited: 2024-01-30

Contribution:

Would You like to contribute to our Kubernetes Ansible Role project? You can assist in the following ways:

 

ToDo: Add the page list of contributors