Initialize First Master Node

There is a slight discrepancy between initializing the first master node and additional master nodes

Dependency

• Completed the guidance of Kubeadm Setup

A master node can be one, three, five, and so on, but always an odd number. Having an odd number of control plane nodes helps with leader selection in the case of a machine or zone failure. Creating Highly Available Clusters with kubeadm

kube-vip is an open-source project that aims to simplify providing load balancing services for Kubernetes clusters. Kube-VIP docs

The following procedures required root permission. Please switch to root account or execute command with sudo

Enable kubectl auto-completion for bash

Enable kubectl autocompletion https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#optional-kubectl-configurations-and-plugins

Ensure that the kubectl completion script gets sourced in all your shell sessions

kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
sudo chmod a+r /etc/bash_completion.d/kubectl

Extend shell completion to work with alias-k

echo 'alias k=kubectl' >> /etc/bash_completion.d/kubectl
echo 'complete -o default -F __start_kubectl k' >>  /etc/bash_completion.d/kubectl

Prepare Kube-Vip Configuration

Generate kube-vip manifest

Create a directory for storing the coming config files

mkdir ~/kube-vip-init

Scan and reserve the available IPs for virtual IP address (VIP)

nmap -sn 192.168.x.x/24

To examine the name of network interface

ip addr

Set the VIP to the available IP address that will be used for the control plane load balancer VIP

Please revise to your ip address

export VIP=192.168.9.88

Set the INTERFACE name to the name of the interface on the control plane(s) that will announce the VIP. In many Linux distributions, this can be found using the ip addr command.

Please revise to your own interface name

export INTERFACE=ens123

Get the latest version of the kube-vip release by parsing the GitHub API. This step requires the Linux packages jq and curl.

KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")

Create the alias of kube-vip docker container program

alias kube-vip="docker run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"

Create APR config file

This configuration will create a manifest that starts kube-vip, providing control plane VIP and Kubernetes Service management using the leaderElection method and ARP. When this instance is elected as the leader, it will bind the VIP to the specified INTERFACE. This is the same behaviour for Services of type LoadBalancer

kube-vip manifest pod \
    --interface $INTERFACE \
    --address $VIP \
    --controlplane \
    --services \
    --arp \
    --leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml

Customize kube-vip manifest

Edit the kube-vip.yaml to custom the config

vim /etc/kubernetes/manifests/kube-vip.yaml

The load balancing is provided through IPVS (IP Virtual Server) and provides a Layer 4 (TCP-based) round-robin across all of the control plane nodes. By default, the load balancer will listen on the default port of 6443 as the Kubernetes API server. The IPVS virtual server lives in kernel space and doesn't create an "actual" service that listens on port 6443. This allows the kernel to parse packets before they're sent to an actual TCP port. This is important to know because it means we don't have any port conflicts having the IPVS load balancer listening on the same port as the API server on the same host. Control Plane Load-Balancing

Disable svc_enable by set the value to false, to prevent conflict with other service load-balancer in future. Enable lb_enable, by append lb_enable and lb_port, then set to true and 6443, respectively.

spec:
  containers:
  - args:
    - manager
    env:
    - name: svc_enable
      value: "false"
    - name: lb_enable
      value: "true"
    - name: lb_port
      value: "6443"

Init Kubernetes cluster

Append master nodes and vip host into hosts file

cat <<EOF >> /etc/hosts
#Kube-vip
192.168.x.xx   kube-vip-lb
#Master nodes
192.168.x.xx   master1
192.168.x.xx   master2
192.168.x.xx   master3
#Worker nodes
192.168.x.xx   worker1
192.168.x.xx   worker1
EOF

Hot Fix: Command pre-kubeadm - Use super-admin.conf during kubeadm init

BUG: kube-vip requires super-admin.conf with Kubernetes 1.29 Issue #684​

sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
          /etc/kubernetes/manifests/kube-vip.yaml

Create kubeadm init configuration file

cat <<EOF > ~/kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: $VIP:6443
apiServer:
  certSANs:
  - kube-vip-lb    # Host name of kube-vip
  - master1        # Host name of master node 1
  - master2        # Host name of master node 2
  - master3        # Host name of master node 3
  - 192.168.x.xx   # IP address of kube-vip
  - 192.168.x.xx   # IP address of master node 1
  - 192.168.x.xx   # IP address of master node 2
  - 192.168.x.xx   # IP address of master node 3
EOF

Initialize the control plane with custom configuration file kubeadm-init.yaml

Stacked control plane and etcd nodes

kubeadm init --config=$HOME/kubeadm-init.yaml --upload-certs

Hot Fix: Command post-kubeadm - Revert the static pod back to admin.conf after successful kubeadm init

BUG: kube-vip requires super-admin.conf with Kubernetes 1.29 Issue #684

sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' \
          /etc/kubernetes/manifests/kube-vip.yaml

Save the initialized information into text file

vim ~/init-info.txt

Source out KUBECONFIG for root user

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /root/.bashrc
source /root/.bashrc

Examine master node connectivity

Debugging control plane load balancing

Display the load balancer configuration

sudo ipvsadm -ln

Watch things interact with the API server with auto-refresh the connections

watch sudo ipvsadm -lnc

Create regular user

Create user for k8suser

useradd -m k8suser

Set password for k8suser

passwd k8suser

Grant sudo privileges

visudo

Insert above line into the file

k8suser    ALL=(ALL)       ALL

Add user k8suser to the docker group

usermod -aG docker k8suser

Check which groups a user belongs to

groups k8suser

Switch to user k8suser to manage Kubernetes cluster

su k8suser
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Implement Calico Network (CNI)

Install Calico networking and network policy for on-premises deployments

Create directory to store the manifest file

mkdir -p $HOME/calico-CNI
cd $HOME/calico-CNI

Download and Apply Calico CNI

curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yaml -O
kubectl apply -f calico.yaml