Category: DevOps

Posted on

A Four Node Kubernetes Cluster for Your Home Lab

Introduction

While I have this on my mind, I decided to build a microk8s cluster from a bunch of raspberry pis that I have laying around. Why? In practice more useful than minikube which is also great for local development.

What we will be using

  1. Ubuntu 22.04.03 Linux distribution
    • 32 bit for contro pi 3b armhf
    • 64 bit for 4 arm64
  2. Ansible
  3. microk8s
  4. prometheus, grafana, node_exporter

Using five raspberry pi’s. One for an Ansible control to deploy software and four for our microk8s cluster

Hardware Setup

IP Allocation

I added address reservations on my router to automatically assign an IP to a known MAC address on a Raspberry pi. This ensures that the pi will have the same IP address each time it boots. No messing with the network configuration in the operating system every time I re-install the OS. Since this is technically a test environment where I will burn-and-build at will. I will not go through this excercise as there ae many ways to do this on a DHCP server.

To obtain the MAC aaddress, I first installed an OS on the pi and noted down the MAC.

OS Installation

Use raspberry pi imager. You can download the softwaere here https://www.raspberrypi.com/software/

Ansible

for automation

Copy Ansible keys to other hosts from cloudletcmd

ssh-copy-id -i ansible_rsa.pub pi@192.168.1.200
ssh-copy-id -i ansible_rsa.pub pi@192.168.1.201
ssh-copy-id -i ansible_rsa.pub pi@192.168.1.202
ssh-copy-id -i ansible_rsa.pub pi@192.168.1.203

ansible-playbook update.yml –ask-become-pass
ansible all -m ping

For microk8s

sudo nano /boot/firmware/cmdline.txt

add the following to the file
cgroup_enable=memory cgroup_memory=1

then reboot
sudo reboot

MicroK8s

sudo snap install microk8s --classic
sudo apt install linux-modules-extra-raspi
sudo microk8s stop; sudo microk8s start

sudo microk8s.add-node <- have to run for each node

From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.200:25000/cfd6f5addae6e0959a4c7f1de1f47a47/583d9c2f5994

Use the ‘–worker‘ flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.200:25000/cfd6f5addae6e0959a4c7f1de1f47a47/583d9c2f5994 --worker

If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.200:25000/cfd6f5addae6e0959a4c7f1de1f47a47/583d9c2f5994

192.168.1.200 cloudlet01
192.168.1.201 cloudlet02
192.168.1.202 cloudlet03
192.168.1.203 cloudlet04

microk8s join 192.168.1.200:25000/0c4812f17a41a0a6c9e0a931d263d0eb/583d9c2f5994
microk8s enable ha-cluster on main node

kube config

to add to .kube/config run sudo microk8s config and add output to config.

sudo usermod -a -G microk8s pi

Prometheus

Set up

Manually setting up Prometheus node exporter

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus
sudo apt update
sudo apt -y install wget curl vim
sudo apt autoremove
curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest | grep browser_download_url | grep linux-arm64 | cut -d '"' -f 4 | wget -qi -
tar xvf node_exporter-1.5.0.linux-armv7.tar.gz
cd node_exporter-1.5.0.linux-armv7/
sudo mv node_exporter /usr/local/bin
node_exporter --version

A basic service config for node_exporter:
sudo vim /etc/systemd/system/node_exporter.service

[Unit]
Description=Prometheus
Documentation=https://github.com/prometheus/node_exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/node_exporter \
--collector.cpu \
--collector.diskstats \
--collector.filesystem \
--collector.loadavg \
--collector.meminfo \
--collector.filefd \
--collector.netdev \
--collector.stat \
--collector.netstat \
--collector.systemd \
--collector.uname \
--collector.vmstat \
--collector.time \
--collector.mdadm \
--collector.zfs \
--collector.tcpstat \
--collector.bonding \
--collector.hwmon \
--collector.arp \
--web.listen-address=:9100 \
--web.telemetry-path="/metrics"

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload; sudo systemctl start node_exporter; sudo systemctl enable node_exporter; sudo systemctl status node_exporter

Remove microk8s

sudo microk8s reset

sudo snap remove microk8s

Resources

https://microk8s.io/docs/install-raspberry-pi

Posted on

Setting up a new Windows Laptop with DevOps tools.

About

This guide is meant as a general guide to set up a Windows machine with a suite of DevOps tools. This guide assumes the user has a working knowledge of these tools. It is not meant for a novice.

This setup is being performed on a brand new Windows 11 Home Intel Core i9 with 32MB RAM.

Programming Languages

Python

Python is an open source programming language that can run on many different architectures. It has a syntax that is easy to learn and is notable as a learning language,

Head over to https://www.python.org/downloads/ and download the latest version of Python. As of this writing, version 3.10.1 is available.

Run the installer an make sure you add python to PATH. Open a command prompt and test with:

python –version

Coding IDE

My preference is VS Code. A good IDE is a MUST for any DevOps engineer.

Visual Studio Code

Visual Studio Code is a source-code editor made by Microsoft for Windows, Linux and macOS. Features include support for debugging, syntax highlighting, intelligent code completion, snippets, code refactoring, and embedded Git.

https://en.wikipedia.org/wiki/Visual_Studio_Code

Free and is very customizable. Head on over to https://code.visualstudio.com/docs/?dv=win to download the installer. Version 1.63 is available as of this writing.

PyCharm

alternative, Python specific

Source Code Management

The goal of any successful engineer is to have repeatable and reproducible methods. This goal can be achieved by storing all work in a repository. Git is the most common SCM at this point in time.

Store your source for free on github or gitlab. Download the git CLI from https://git-scm.com/download/win

Git is easy to learn and has a tiny footprint with lightning fast performance. It outclasses SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows.

https://git-scm.com/

Virtualization

VirtualBox

VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2. See “About VirtualBox” for an introduction.

https://www.virtualbox.org/

As of this writing, version 6.1.30 is available. Download the Windows installer here: https://www.virtualbox.org/wiki/Downloads

Infrastructure Automation

aws-cli

https://aws.amazon.com/cli/

vagrant

https://www.vagrantup.com/

Plugins (additional)

vagrant plugin install vagrant-disksize

terraform

download here https://www.terraform.io/downloads

Alternately install with chocolatey:

choco install terraform

Docker

https://www.docker.com/products/docker-desktop

uh oh for WSL 2

https://docs.microsoft.com/en-us/windows/wsl/install-manual#step-4—download-the-linux-kernel-update-package

chocolatey

Chocolatey is a software management solution unlike anything else you’ve ever experienced on Windows. Chocolatey brings the concepts of true package management to allow you to version things, manage dependencies and installation order, better inventory management, and other features.

https://chocolatey.org/
  1. First, ensure that you are using an administrative shell – you can also install as a non-admin, check out Non-Administrative Installation.
  2. Install with powershell.exe📝 NOTE: Please inspect https://community.chocolatey.org/install.ps1 prior to running any of these scripts to ensure safety. We already know it’s safe, but you should verify the security and contents of any script from the internet you are not familiar with. All of these scripts download a remote PowerShell script and execute it on your machine. We take security very seriously. Learn more about our security protocols.With PowerShell, you must ensure Get-ExecutionPolicy is not Restricted. We suggest using Bypass to bypass the policy to get things installed or AllSigned for quite a bit more security.
    • Run Get-ExecutionPolicy. If it returns Restricted, then run Set-ExecutionPolicy AllSigned or Set-ExecutionPolicy Bypass -Scope Process.Now run the following command:
    • Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString(‘https://community.chocolatey.org/install.ps1’))
  3. Paste the copied text into your shell and press Enter.
  4. Wait a few seconds for the command to complete.
  5. If you don’t see any errors, you are ready to use Chocolatey! Type choco or choco -? now, or see Getting Started for usage instructions.

Additional Tools

mRemoteNG

mRemoteNG is a fork of mRemote: an open source, tabbed, multi-protocol, remote connections manager for Windows. mRemoteNG adds bug fixes and new features to mRemote and allows you to view all of your remote connections in a simple yet powerful tabbed interface.

https://mremoteng.org/

This is a tool for all remote connections. Supports VNC, RDP, SSH, and others, all in a single GUI Interface. PuTTY is used in the backend for SSH connections. Download the latest version here: https://mremoteng.org/download

Posted on

HashiQube

What is this?

A colleague turned me on to this interesting project that pulls together some or all of HashiCorps offerings. They all run within a virtual instance on a local machine. This could prove useful for discovering how each component works.

You can find the project here: https://servian.github.io/hashiqube/

Getting it to work.

You will need Vagrant and Docker to get this up and running.

Posted on

Creating an HTPC Stack With Docker Things, Part 2

This is a continuation of my previous post: Create an HTPC Stack With Docker Things. I will introduce docker-compose to bring up an environment with both containers used previously, sabnzbd and sickbeard.

I installed docker-compose using the instructions found here: Install Compose. I am using Mac OS X.

First make sure boot2docker is running, boot2docker up. Then create the following file named docker-compose.yml.

sickbeard:
  image: maxexcloo/sickbeard 
  ports:
   - "8081:8081"
  name:
   - "sickbeard"
sabnzbd:
  image: maxexcloo/sabnzbd
  ports:
   - "8080:8080"
  name:
   - "sabnzbd"

This is setting up the environment with two images, exposing the ports the application will run on and naming the image to something meaningful. each directive in the yml file map to the same docker run command.

Once this file has been created, simply run docker-compose up, and the images will be started.

To-Do

  • create a github repo for application configs.
  • map a local directory for configs.
  • add the local config directory to the yml.
  • more things…
Posted on

Creating an HTPC Stack With Docker Things

Introduction

This will be a mess until I figure it all out. In this article, I will demonstrate using tools such as packer, docker client, docker-machine to build out a Docker host with an HTPC stack. The following applications will be used:

  • Sickbeard
  • Sabnzbd+
  • HTPC Manager
  • Plex
  • Transmission web interface
  • Couch Potato

I will be using the latest Ubuntu 14.04 server distribution. This is being developed on Mac OSX Yosemite 10.10.2.

This is a WIP and will be complete when it is complete. Some of these tools I have not used yet and am learning as I go!

Expect many changes as I figure this out!

Setup

You will need the following tools to start:

This article assumes you have installed, and are familiar with the above tools.

I have created a root directory named htpc where all project files will reside.

The Docker Host

First create a new Docker host with the docker-machine command:

docker-machine create --driver virtualbox htpc

Now, point your docker client at the new Docker host:

$(docker-machine env htpc)

Sickbeard container

Docker hub contains thousands of community made images. Searching the hub for a Sickbeard image returns many results. I chose maxexcloo/sickbeard for this example. It is Debian based. First, we need to pull the image.

docker pull maxexcloo/sickbeard

Now run it:

docker $(docker-machine config htpc) run -d -p 8081:8081 --name="sickbeard" maxexcloo/sickbeard

This command will expose port 8081 and run the image in daemon mode (detached).

the –name”” argument provides a nice name that will show up in the NAME column from the output of docker ps. If this argument is not suppled, a random name will be generated.

We are almost finished. To find the IP of the Docker host:

docker-machine ip

Finally, you can point your browser at http://<docker host ip>:8081.

Sabnzbd+

As with Sickbeard, this container will be run similarly.

docker pull maxexcloo/sabnzbd

docker $(docker-machine config htpc) run -d -p 8080:8080 –name=”sabnzbd” maxexcloo/sabnzbd

Running docker ps will show something like this:

CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS                    NAMES
b432e5cbf6df        maxexcloo/sabnzbd:latest     "/bin/sh -c /config/   5 seconds ago       Up 4 seconds        0.0.0.0:8080->8080/tcp   pensive_ritchie     
dbcd9f132908        maxexcloo/sickbeard:latest   "/bin/sh -c /config/   5 minutes ago       Up 5 minutes        0.0.0.0:8081->8081/tcp   suspicious_tesla

We now have a Docker host running two images!

Further Thoughts

If this all works out as intended, possibly use vagrant-managed-servers to build out a server that already has an OS on it.

Make my own images.

References

http://kappataumu.com/articles/creating-an-Ubuntu-VM-with-packer.html

https://www.packer.io/docs/builders/virtualbox-iso.html

https://registry.hub.docker.com/u/maxexcloo/sickbeard/

Posted on

Up and running chef-solo instance using Vagrant

This will describe a quick and easy way to get a chef-solo instance up and running using Vagrant. This document is intended for learning Chef and not intended for production use.

I am using Mac OS X for this tutorial. I am sure it is simiar for Windows and Linux.

Prerequisites

  • Oracle Virtual Box – Download
  • Vagrant – Download
  • Working knowledge of Vagrant

 Setting It Up

Create a directory for the Vagrantfile and bootstrap.sh. I named mine chef_solo. Change into that directory and create a Vagrantfile with the following:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision :shell, path: "bootstrap.sh"
end

This will spin up an Ubuntu server 14.04 64 bit instance and run bootstrap.sh.

Now edit bootstrap.sh and add the following:

#!/usr/bin/env bash

apt-get update
apt-get install -y curl
curl -L https://www.opscode.com/chef/install.sh | bash
chef-solo -v

Once the VM is up and running, bootstrap.sh will apply updates, install curl, download and install chef-solo using the Omnibus installer, and print out the version of Chef.

Chef-solo up and running

Now simply run vagrant up and your chef-solo instance will be installed.

Posted on

WMI and JMX with Icinga

Over the past week I have been flexing my monitoring skills again. Of course the landscape has changed since I worked on this a few years ago. Today, I am struggling with a commercial product, unnamed. Bloated, expensive, horrible GUI, and time consuming. I am sure it is a good product given that you have an entire staff dedicated to managing it.

So, I went back to my roots and set up an Icinga instance. I have it fully working on a Linux box monitoring Windows hosts. Agentless. No more nsclient installs. I am using check_wmi_plus plugin for everything Windows. You can supply a wmi query string to the plugin for anything WMI. Here are some URLs:

https://wiki.icinga.org/display/howtos/check_wmi_plus

some service definitions:

http://www.edcint.co.nz/checkwmiplus/?q=nagiosservicedefinitions

I am also monitoring out Java infrastructure. You need to make sure the java app is set up for remote jmx monitoring before this will work.

http://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details

and some nice service defs and commands:

https://kb.op5.com/display/HOWTOs/Jboss+Monitoring+with+check_jmx

Maybe I will get some time to write up a comprehensive faq.

Posted on

Connection problems to github.com

One of my colleagues had a small issue with pulling the latest from github for my site today. The error was:

ssh: connect to host github.com port 22: Connection refused
fatal: The remote end hung up unexpectedly

Of course the word “fatal” would freak out any non linux user. So a with a little searching the following is the solution.

Add the following to your ./.ssh/config:

host github.com
    hostname ssh.github.com
    port 443

This forces ssh over https for github. All fixed! Now I do not have to bother networking for a firewall rule.

Posted on

More Tools!

I love having insight into processes. One thing that we cannot live without are tools. I have been saving these up for a few months, but here is a list of three new (new to me) tools that I have come across.

nkrode – A real-time dashboard for redis. I have actually implemented this, but am not using it currently. We are using a windows port of redis (ugh…) and the dashboard does not seem to pick up certain counters like memory consumption. Using the windows port, not my idea!

Mod Rewrite Generator – Generates a mod_rewrite rule based upon your input. I have not used it yet, but some of my colleagues have used it with success.

Which loads faster – From the developer “web performance
matters” This tool will compare the load time of two sites. It is interesting to compare across browsers, and some friendly competition amongst colleagues! Thanks to Khalid for this one.