Cheap and Scalable Kubernetes Clusters with hetzner-k3s

The baseline cost for a managed Kubernetes cluster at Cloud providers is roughly 90$ to 210$ a month. At AWS and GCP, a good part of that cost is assigned to the control planes on which no workload is executed. If you're on the lookout for cheaper and scalable alternatives at less than 20$ with minimal maintenance, here is how to.

Learning more about kubernetes

If you are a french speaker and want to learn Kubernetes, I highly recommend Stéphane Robert's website.

Step 1: K3S

Many cannot afford to have a baseline cost in the hundreds for their cloud clusters. The most obvious solution to that problem is to use lightweight Kubernetes distributions to manage the cluster yourself, and to get rid of the high availability requirements.

The bottleneck will then become managing the cluster yourself, with the updates and setup involved. While we discuss automated cluster management solution later in the article, in practice, the annoyance can be reduced significantly by operating k3s, an edge-computing ready and lightweith Kubernetes distribution that all it takes to install is to run:

curl -sfL https://get.k3s.io | sh -

Eventually, to add a node, you can simply read a token from a file and run on a different machine:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

Need to update Kubernetes ? You can simply swap the binary and restart the service.

I actually have written lengthy articles about setting up single-node K3S for testing and home servers, you might want to check them out, but there are drawbacks to setting up the nodes yourself, namely:

• Managing persistent storage can get complicated, especially for network based storage.
• Automatically scaling up your cluster might prove daunting, especially given the load balancing requirements.
• Mental overhead of managing a fleet of servers and having to write automation for that.

Step 2: A Descent Cloud Provider

A cheap and reliable cloud provider is Hetzner, they have datacenters in Europe and America and offer cloud services like block devices with a Kubernetes CNI, and S3-like object storage (soon to be released). Their cloud offer includes Load Balancers and Private Networks, and this is pretty much all that one needs to do pretty much anything. On top of that, they have a lengthly list of instances in all price ranges, dedicated or with vCPUs.

Step 3: hetzner-k3s

So, to operate a scalable and eventually highly available fleet of K3S nodes, do you need to write hundreds of lines of Terraform or Ansible scripts ? The answer is no, thanks to Vito Botta, Lead Platform Architect at Brella.

Vito wrote a software called hetzner-k3s, which takes care of provisioning K3S clusters at Hetzner at any scale, highly-available or not, and that can scale automatically.

All it takes to effectively deploy a cluster is a yaml file that describes your cluster the following way:

cluster_name: my-cluster
kubeconfig_path: "./kubeconfig"
k3s_version: v1.30.6+k3s1

networking:
  ssh:
    port: 22
    use_agent: true
    public_key_path: "~/.ssh/id_ed25519.pub"
    private_key_path: "~/.ssh/id_ed25519"
  allowed_networks:
    ssh:
      - 0.0.0.0/0
    api:
      - 0.0.0.0/0
  public_network:
    ipv4: true
    ipv6: true
  private_network:
    enabled: true
    subnet: 10.0.0.0/16
    existing_network_name: ""
  cni:
    enabled: true
    encryption: false
    mode: flannel

schedule_workloads_on_masters: true

masters_pool:
  instance_type: cpx21
  instance_count: 1
  location: nbg1

worker_node_pools:
  - name: static-base
    instance_type: cpx21
    instance_count: 1
    location: nbg1
  - name: medium-autoscaled
    instance_type: cpx31
    instance_count: 2
    location: fsn1
    autoscaling:
      enabled: true
      min_instances: 0
      max_instances: 10

You Can Spread your Clusters Across Datacenters

Note that it's very easy to pick instances in different data centers for high-availability and resilience purposes.

In minutes, you can have your cluster ready. It will automatically setup the nodes with the default storageClass being Hetzner block device storage, that can be freely reassigned to different machines. Let's say that you create a service with a persistent volume with the default storage class, a matching volume will automatically be provisioned at Hetzner of the appropriate size.

You can also use the local-path storage if you need maximum IOps and have software side data replication.

On top of that, updating k3s and general maintenance procedures are all automated and require minimal oversight (read the documentation for instructions and examples).

So where does that takes us in terms of costs for non highly-available but scalable cluster ? You will have to pay for:

• The master node (minimum: CX22 at 5$ a month)
• Let's suppose, at least one worker node (minimum: CX22 at 5$ a month)
• Let's suppose, two devices of 20Gb of resilient block device storage (2$ a month)
• Let's suppose, a load balancer (7 $ a month)

Which, ingress and egress costs apart, is worth roughly 19 $ a month. Note that you could very well get rid of the block devices, and if you're smart with resource management, have the worker node scale down to 0 for a baseline cost of 12 $.

Conclusion

If you're a tiny business owner, or you're just starting up, you might find it an exciting opportunity to be able to both pull the cost down and retain the ability to scale. If you're a Kubernetes lover like me, or just a curious person, you might have found a nice playground. In any case, I hope you enjoyed reading the article, and I encourage you to support Vito by Sponsoring him on GitHub.