Installing Kubernetes K3s on CentOS/RHEL Hosts

Here's the scenario: you need an on-premises container orchestrator. This could be for a variety of reasons: maybe going to the cloud is cost prohibitive or you simply want to install and maintain your infrastructure first hand. K3s to the rescue. Rancher's K3s is a lightweight Kubernetes that seeks to make installation and management easier. Kubernetes is notorious for being difficult to set up and administer, but has really become the standard in container orchestration.

Unfortunately, the RHEL (Redhat Enterprise Linux) version 7 OS and its close downstream deprecated cousin CentOS, aren't receiving the love from the Rancher folks as much as Ubuntu is. Documentation is a bit sparse, and there are gotcha's along the way. This "living" document will hopefully serve as a means of guiding and  reporting on experience with K3s on RHEL systems. It will be updated as time goes on (it already has seen updates several times).

Pre-Requisites

Obviously some knowledge of Docker or other container-based technology is critical. In addition, a high level of Linux system administration and networking competence with Redhat-based systems is assumed.

Pre-Installation Steps

Several sources recommend turning off firewalld and selinux and going with iptables only, but we'll try keeping them on for now, your mileage may vary.

K3s will complain (See "k3s check-config" below) about user namespaces and swap. To remedy these errors:

Enable namespaces at boot

Edit /etc/default/grub:

user_namespace.enable=1

Then run grub-mkconfig to save the changes:

grub2-mkconfig -o /boot/grub2/grub.cfg

Disable swap

swapoff -a
sed -e '/swap/s/^/#/g' -i /etc/fstab

Firewall

We need to allow traffic between interfaces. This requires bridged IP traffic and several modules (overlay, nf_conntrack, br_netfilter). Note: no longer needed - K3s appears to be handling this for us now.

cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

Lots of  potential ports need opening, depending on whether the node (host) is a "master" or "worker" node:

Master node(s):
TCP     6443*       Kubernetes API Server
TCP     10250       Kubelet API
TCP     10251       kube-scheduler
TCP     10252       kube-controller-manager
UDP     8285        flannel overlay network - udp backend

Worker nodes (minions):
TCP     10250       Kubelet API
TCP     30000-32767 NodePort Services
UDP     8285        flannel overlay network - udp backend

We can accomplish this with these commands on the master:

firewall-cmd --permanent --add-port=6443/tcp
firewall-cmd --permanent --add-port=10250-10252/tcp
firewall-cmd --permanent --add-port=8285/udp

And these on the worker:

firewall-cmd --permanent --add-port=10250/tcp
firewall-cmd --permanent --add-port=30000-32767/tcp
firewall-cmd --permanent --add-port=8285/udp

firewall-cmd --reload

Installation

On the master:

curl -sfL https://get.k3s.io | sh -

On the worker:

curl -sfL https://get.k3s.io | K3S_URL=https://[master host]:6443 K3S_TOKEN=
[Server token here: /var/lib/rancher/k3s/server/token]

Verify Operation

master node:

k3s check-config
systemctl status -l k3s
kubectl get nodes

worker:

systemctl status -l k3s-agent