{"id":496,"date":"2024-01-10T19:26:20","date_gmt":"2024-01-11T00:26:20","guid":{"rendered":"https:\/\/devopsny.com\/blog\/?p=496"},"modified":"2025-08-06T10:35:44","modified_gmt":"2025-08-06T14:35:44","slug":"a-four-node-kubernetes-cluster-for-your-home-lab","status":"publish","type":"post","link":"https:\/\/devopsny.com\/blog\/2024\/01\/10\/a-four-node-kubernetes-cluster-for-your-home-lab\/","title":{"rendered":"A Four Node Kubernetes Cluster for Your Home Lab"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\">While I have this on my mind, I decided to build a microk8s cluster from a bunch of raspberry pis that I have laying around. Why? In practice more useful than minikube which is also great for local development.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What we will be using<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Ubuntu 22.04.03 Linux distribution\n<ul class=\"wp-block-list\">\n<li>32 bit for contro pi 3b armhf<\/li>\n\n\n\n<li>64 bit for 4 arm64<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Ansible<\/li>\n\n\n\n<li>microk8s<\/li>\n\n\n\n<li>prometheus, grafana, node_exporter<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">Using five raspberry pi&#8217;s. One for an Ansible control to deploy software and four for our microk8s cluster<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Hardware Setup<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">IP Allocation<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I added address reservations on my router to automatically assign an IP to a known MAC address on a Raspberry pi. This ensures that the pi will have the same IP address each time it boots. No messing with the network configuration in the operating system every time I re-install the OS. Since this is technically a test environment where I will burn-and-build at will. I will not go through this excercise as there ae many ways to do this on a DHCP server.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To obtain the MAC aaddress, I first installed an OS on the pi and noted down the MAC.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">OS Installation<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Use raspberry pi imager. You can download the softwaere here https:\/\/www.raspberrypi.com\/software\/<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Ansible<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">for automation<\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\">Copy Ansible keys to other hosts from cloudletcmd<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">ssh-copy-id -i ansible_rsa.pub pi@192.168.1.200<br>ssh-copy-id -i ansible_rsa.pub pi@192.168.1.201<br>ssh-copy-id -i ansible_rsa.pub pi@192.168.1.202<br>ssh-copy-id -i ansible_rsa.pub pi@192.168.1.203<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">ansible-playbook update.yml &#8211;ask-become-pass<br>ansible all -m ping<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">For microk8s<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo nano \/boot\/firmware\/cmdline.txt<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">add the following to the file<code><mark style=\"background-color:#000000\" class=\"has-inline-color has-background-color\"><br><\/mark><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">cgroup_enable=memory cgroup_memory=1<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">then reboot<code><mark style=\"background-color:#000000\" class=\"has-inline-color has-background-color\"><br><\/mark><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo reboot<\/mark><\/code><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">MicroK8s<\/h2>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\"><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo snap install microk8s --classic<br>sudo apt install linux-modules-extra-raspi<br>sudo microk8s stop; sudo microk8s start<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-white-color\"><code>sudo microk8s.add-node<\/code> <\/mark>&lt;- have to run for each node<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">From the node you wish to join to this cluster, run the following:<br><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">microk8s join 192.168.1.200:25000\/cfd6f5addae6e0959a4c7f1de1f47a47\/583d9c2f5994<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Use the &#8216;<em>&#8211;worker<\/em>&#8216; flag to join a node as a worker not running the control plane, eg:<br><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">microk8s join 192.168.1.200:25000\/cfd6f5addae6e0959a4c7f1de1f47a47\/583d9c2f5994 --worker<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the node you are adding is not reachable through the default interface you can use one of the following:<br><code>microk8s join 192.168.1.200:25000\/cfd6f5addae6e0959a4c7f1de1f47a47\/583d9c2f5994<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">192.168.1.200 cloudlet01<br>192.168.1.201 cloudlet02<br>192.168.1.202 cloudlet03<br>192.168.1.203 cloudlet04<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">microk8s join 192.168.1.200:25000\/0c4812f17a41a0a6c9e0a931d263d0eb\/583d9c2f5994<br>microk8s enable ha-cluster on main node<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">kube config<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">to add to .kube\/config run sudo microk8s config and add output to config.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo usermod -a -G microk8s pi<\/mark><\/code><\/p>\n<\/div>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Prometheus<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Set up<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Manually setting up Prometheus node exporter<\/p>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\"><code><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo groupadd --system prometheus<br>sudo useradd -s \/sbin\/nologin --system -g prometheus prometheus<br>sudo apt update<br>sudo apt -y install wget curl vim<br>sudo apt autoremove<br>curl -s https:\/\/api.github.com\/repos\/prometheus\/node_exporter\/releases\/latest | grep browser_download_url | grep linux-arm64 | cut -d '\"' -f 4 | wget -qi -<br>tar xvf node_exporter-1.5.0.linux-armv7.tar.gz<br>cd node_exporter-1.5.0.linux-armv7\/<br>sudo mv node_exporter \/usr\/local\/bin<br>node_exporter --version<\/mark><\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A basic service config for node_exporter:<code><mark style=\"background-color:#000000\" class=\"has-inline-color has-background-color\"><br><\/mark><mark style=\"background-color:#000000\" class=\"has-inline-color has-white-color\">sudo vim \/etc\/systemd\/system\/node_exporter.service<\/mark><\/code><\/p>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"wp-block-paragraph\"><code>[Unit]<br>Description=Prometheus<br>Documentation=https:\/\/github.com\/prometheus\/node_exporter<br>Wants=network-online.target<br>After=network-online.target<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>[Service]<br>Type=simple<br>User=prometheus<br>Group=prometheus<br>ExecReload=\/bin\/kill -HUP $MAINPID<br>ExecStart=\/usr\/local\/bin\/node_exporter \\<br>--collector.cpu \\<br>--collector.diskstats \\<br>--collector.filesystem \\<br>--collector.loadavg \\<br>--collector.meminfo \\<br>--collector.filefd \\<br>--collector.netdev \\<br>--collector.stat \\<br>--collector.netstat \\<br>--collector.systemd \\<br>--collector.uname \\<br>--collector.vmstat \\<br>--collector.time \\<br>--collector.mdadm \\<br>--collector.zfs \\<br>--collector.tcpstat \\<br>--collector.bonding \\<br>--collector.hwmon \\<br>--collector.arp \\<br>--web.listen-address=:9100 \\<br>--web.telemetry-path=\"\/metrics\"<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>[Install]<br>WantedBy=multi-user.target<\/code><\/p>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">sudo systemctl daemon-reload; sudo systemctl start node_exporter; sudo systemctl enable node_exporter; sudo systemctl status node_exporter<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Remove microk8s<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">sudo microk8s reset<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">sudo snap remove microk8s<\/p>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Resources<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/microk8s.io\/docs\/install-raspberry-pi\" target=\"_blank\" rel=\"noopener\" title=\"\">https:\/\/microk8s.io\/docs\/install-raspberry-pi<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction While I have this on my mind, I decided to build a microk8s cluster from a bunch of raspberry pis that I have laying around. Why? In practice more useful than minikube which is also great for local development. What we will be using Using five raspberry pi&#8217;s. One for an Ansible control to [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":523,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[64,231,85,60,1],"tags":[248,247,249,33],"class_list":["post-496","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-kubernetes","category-monitoring","category-ubuntu-linux","category-uncategorized","tag-ansible","tag-k8s","tag-prometheus","tag-ubuntu-2"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/devopsny.com\/blog\/wp-content\/uploads\/2025\/08\/14150114293_e4132eb199_b.jpg?fit=768%2C1024&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3fcrv-80","jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/posts\/496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/comments?post=496"}],"version-history":[{"count":5,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/posts\/496\/revisions"}],"predecessor-version":[{"id":519,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/posts\/496\/revisions\/519"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/media\/523"}],"wp:attachment":[{"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/media?parent=496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/categories?post=496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devopsny.com\/blog\/wp-json\/wp\/v2\/tags?post=496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}