Casual Chat
Recently, I've been messing around with k8s and some other stuff, and I've found that it's a very deep rabbit hole (though it might just be my problem). Currently deployed things include:
- k3s
- nfs
- samba (this existed before)
- seaweedFS (deployed but for a specific service)
- MetalLB It's been quite a journey.
Architecture
The current setup is a bit of a clash between old and new architectures. An external Nginx directs some domains to k8s. Then it uses internal Traefik. Next is the internal k8s network. Some services go directly through LoadBalancer and then to Nginx. In short, it's quite messy lol.
Advantages
The advantages go without saying. Simply put, it has reduced some management difficulties. At least the problem of being too lazy to deploy on every single server has been solved. However, it comes with a corresponding massive learning cost.
For example, I hadn't learned Docker before, but it's required. If you don't want your images to be public, you can set up a Registry. And using http for the Registry is a problem in itself, but that's a side note for later.
In a way, it encourages learning. (I even want to set up CI/CD now. GitOps, here I come!)
Of course, there's also the fact that one deployment.yaml can handle a lot of things. Even though it can get very bloated. And then you get to learn even more new things (exhausting).
Actually, there's something interesting.
Since I couldn't find a Minecraft Server Manager for k8s before, I found while implementing it myself that you can make things stateless. Basically, the server just follows the k8s state to change its own stuff. It's surprisingly convenient and fits the k8s way of doing things, I guess. By the way, being able to mount Config-Maps as files reduces a lot of burden and makes changes easy. It's a win-win, really great.
Disadvantages
The disadvantages are quite obvious to me, mainly because I haven't had systematic learning. Basically, I learn as I go, so the foundation isn't solid. The learning curve is incredibly steep. While the basics are simple to understand, using it well is truly difficult.
There's also a pitfall, not exactly k8s-related. If you set up your own Registry using http, you'll run into a lot of trouble. In my case, I used Raspberry Pis to build the Cluster. You need to build images for arm64. In this situation, buildx ... --push will go on strike. Plus, you have to configure every node in the cluster to allow http. It's quite a waste of time lol.
Also, although it might be a rare case, if you want to achieve multi-node data loss prevention, you definitely need a third storage method independent of the Nodes. This is the only way to migrate data across Nodes, but that brings its own problems. For example, I set up an additional nfs. I still don't understand its permission issues. Though that's more of a Linux learning problem lol.