StopPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to destroy network for sandbox "055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708": failed to get network "cbr0" cached result: decoding version from network config: unexpected end of JSON input

As there has never been any associated impact with this that I could see I just ignored it... for a long time!

Well, fast forward to last Friday night where a sudden hyper-focus hit and I decided I wanted to know what was going on and how to stop it - here's how it went.

A little context

Before we dive into things I think it's worth providing a bit of context first so we all know what we're working with. The Kubernetes cluster in question is my "homelab" cluster that is running Talos Linux within a Proxmox cluster. The cluster has a single control plane node and 5 worker nodes. The worker nodes are where we'll be focussing. Beyond that there's nothing too special about this cluster. CNI is provided by Flannel with default configuration. The only thing that is kinda unusual is my use of SMB for my CSI driver - I only mention this because it's possible that a failure here might have been the thing that trigger the initial issue but I don't know for sure.

Identifying the issue

As I sad, I had ignored this for months and not thought much of it until Friday. I was cleaning up some unrelated stuff in my cluster and was checking my Kubelet logs stored in Loki to see if my cleaning up had been applied. This is how I (re-)noticed the high number of sandbox errors from various Kubelets. So I took a closer look with:

{service="kubelet"} |= "failed to get network \"cbr0\" cached result"

There was a LOT of these errors in the logs, almost 1000 per hour! 😳

Even though there was a lot of errors there was actually only a handful of sandbox IDs causing the problem so it wasn't widespread, just frequent.

Figuring out what was wrong

Because I'm a Professional Kubernetes Platform Engineer™ I knew exactly how to tackle this... ask the internet! So fired up DuckDuckGo, pasted in my error message and started wading through a LOT of useless results.

• I found a couple similar issues that people had experienced with Calico but didn't seem to fit what I was seeing.
• There was a closed issue against Containerd where this comment gave me some hope in solving it with a loopback CNI but turned out unrelated also.

These did give me enough info to start looking in the right places though. There was just one problem - Talos Linux doesn't have a shell you can SSH into, it's all API driven.

kubectl debug to the rescue!!!

Thankfully, it's possible to mimic an SSH session using kubectl debug against the node. As I knew I'd need a bunch of container-related tools to help with debugging I launched a debug container using the alpine-containertools image from Rory McCune. Due to how the debug command mounts the host filesystem I also had to override the CONTAINER_RUNTIME_ENDPOINT environment variable to point to the correct location.

kubectl debug -it --image raesene/alpine-containertools \
  --env CONTAINER_RUNTIME_ENDPOINT=unix:///host/run/containerd/containerd.sock \
  node/talos-192-168-1-15 \
  -- sh

Now we can start poking around on the node. First, lets see if we can force delete the sandbox pod ourself.

crictl rmp --force 055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708

Failed to stop sandbox before removing: rpc error: code = Unknown desc = failed to destroy network for sandbox "055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708": failed to get network "cbr0" cached result: decoding version from network config: unexpected end of JSON input

No luck! But we've been able to reproduce what Kubelet is hitting at least.

I spent a bit more time poking at crictl to get an understanding of the state of things. Doing so I was able to identify which pods were broken on each node.

# Get the full details about the pod in containerd
crictl inspectp --output=json 055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708
# List all pods known to containerd, and filter to those in a Notready state
crictl pods | grep NotReady

Eventually, after wading through all the unrelated search results one did seem relevant, a GitHub issue on containerd: "Empty cache files causes "KillPodSandbox" errors when deleting pods".

Cleaning things up

This final GitHub issue contained one important piece of information I was missing - the location of the network cache: /var/lib/cni/results. This is the directory where the CNI stores a JSON file for each network it creates. So let's go take a look:

> ls -la /host/var/lib/cni/results

total 148
drwx------    2 root     root          4096 Sep 27 05:48 .
drwx------    5 root     root            52 Jun 23  2022 ..
-rw-------    1 root     root          1647 Sep 27 05:47 cbr0-01d576d2f808a969bb7cb8da11d1ee3117ec4e9792d4aea33abec55318b74e01-eth0
-rw-------    1 root     root             0 Jun 15 21:28 cbr0-055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708-eth0
-rw-------    1 root     root          1608 Sep 27 05:48 cbr0-225486a2df0c2c8be5038245d099b4c48dc03ede648735a2b4676154650e7741-eth0
-rw-------    1 root     root          2045 Sep 27 05:46 cbr0-2e848e89543e7b6491d0b47eb2be7ec620865024d942012303ede16842aa1108-eth0
-rw-------    1 root     root          1638 Sep 27 05:48 cbr0-415cd18f24ec2740486a7bf254054f871bd8abe4c26bc6e27dbf1f5a10c29f69-eth0
-rw-------    1 root     root          1638 Sep 27 05:46 cbr0-47d021d26d2d1935025cee58941ff68ad28e09f1337b867e0865be56dde64a2a-eth0
-rw-------    1 root     root          1616 Sep 27 05:46 cbr0-5d40d2f86c6f4be7a85969f24e844ea049b41af14b97cfa3f5a655f52d4fc695-eth0
-rw-------    1 root     root          1616 Sep 27 05:48 cbr0-68f252ab057496036e8852a4f6103c1b6c564f6976d04676010858a5588a9a10-eth0
-rw-------    1 root     root          1596 Sep 27 05:48 cbr0-91d8805fe4b95c92f4c53e1321813b5fe9f71467c5692328bb7646649db57b22-eth0
-rw-------    1 root     root          1764 Sep 27 05:48 cbr0-a0eaa73bd701c9b1b1167efd49487b48074936211de1c6be37beec5184e42691-eth0

Notice anything suspicious? The file size of the file associated with our failing sandbox ID is 0. 🤔 That explains why the CNI is unable to load from the cache then - it's empty. Taking a quick look at one of the other files shows a valid JSON object with various network configuration stored within.

So lets delete this file - rm -rf /host/var/lib/cni/results/cbr0-055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708-eth0

With that gone, let's try again to delete the pod:

crictl rmp 055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708

Removed sandbox 055b221e44a28ce8d9120f771d5e1ef201f2457ce49c58999a0147104cca2708

SUCCESS! 🎉

Now we know what to clean up it's time to repeat the process for each filing sandbox ID on each node. Once that is done the kubelet logs are looking muuuucccchhhhhhh better!

Preventing this happening again

As I'm still unclear why or how these pods first failed I can't be sure it wont happen again. My current leading theories are it was either caused by a previous power failure that left the nodes in a bad state or was related to my NAS crashing as the few pods I took a close look at all had SMB volumes associated with them.

Regardless of what caused it I want to make sure I'm being more proactive going forward and leaving my cluster in a better state than it was before. I could set up a dashboard that includes the log query from Loki that we used at the start of this post but that requires me to check it often and... I'm lazy. What we really want is some alerts to annoy me when / if this happens again.

I already have quite a decent monitoring and alerting setup in my homelab made up of VictoriaMetrics, AlertManager and various metric exporters so I'm in a good starting point.

But what I am lacking is containerd metrics.

A quick search lead me to this great post by Abraham Dahunsi covering a lot of details about monitoring containerd. This gave the information on what metrics to work with and what I can do with them but I was still missing the actual metrics. In Talos Linux the containerd metrics aren't enabled by default but thankfully it's quite simple to enable, I just needed to add the following to my machine config for each of my nodes:

machine:
  files:
    - content: |
        [metrics]
          address = "0.0.0.0:11234"
      path: /etc/cri/conf.d/20-customization.part
      op: create

I then just need to instruct vmagent to scrape this new endpoint:

    scrape_configs:
    - job_name: containerd
      scrape_interval: 30s
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}:11234/proxy/v1/metrics
      - source_labels: [__meta_kubernetes_node_name]
        action: replace
        target_label: kubernetes_node

Now I have the metrics coming in, we can put them to use at identifying issues.

Identify failures starting a pod sandbox:

sum by(grpc_code, instance) (
  rate(grpc_server_handled_total{job="containerd", grpc_code!="OK", grpc_method="RunPodSandbox"})
) > 0

Identify failures stopping or removing a pod sandbox: (This is the issue we handled in this post)

sum by(grpc_code, grpc_method, instance) (
  rate(grpc_server_handled_total{job="containerd", grpc_code!="OK", grpc_method=~"StopPodSandbox|RemovePodSandbox"})
) > 0

Find all pods known to containerd that aren't known to Kubernetes:

(
  container_pids_current{namespace="k8s.io"}
    unless on (container_id)
    label_replace(
        kube_pod_container_info,
        "container_id", "$1", "container_id", "containerd://(.+)"
    )
    unless on (container_id)
    label_replace(
        kube_pod_init_container_info,
        "container_id", "$1", "container_id", "containerd://(.+)"
    )
)
unless on (container_id)
(
  container_memory_swap_limit_bytes > 0
)

This last one has a lot going on but it breaks down to:

1. List all pids counts for all containers in the k8s.io namespace in containerd
2. Unless the container_id is also found on the kube_pod_container_info metric (where we need to strip the containerd:// prefix)
3. Or the container_id is also found on the kube_pod_init_container_info metric
4. And finally only if the container_memory_swap_limit_bytes is greater than 0 which will filter out the sandbox containers (pause container) which aren't exposed to Kubernetes anyway.

These three then become new alerts that can notify me if any match.

- name: containerd
  rules:
  - alert: ContainerdPodStartFailed
    annotations:
      description: |
        Containerd on **{​{ .Labels.instance }}** has failed to start a pod sandbox due to **{​{ .Labels.grpc_code }}**.
    expr: |
      sum by(grpc_code, instance) (rate(grpc_server_handled_total{job="containerd",grpc_code!="OK", grpc_method="RunPodSandbox"})) > 0
    for: 5m
    labels:
      severity: notify

  - alert: ContainerdPodRemoveFailed
    annotations:
      description: |
        Containerd on **{​{ .Labels.instance }}** has failed to {​{ .Labels.grpc_method }} due to **{​{ .Labels.grpc_code }}**.
    expr: |
      sum by(grpc_code, grpc_method, instance) (rate(grpc_server_handled_total{job="containerd", grpc_code!="OK", grpc_method=~"StopPodSandbox|RemovePodSandbox"})) > 0
    for: 5m
    labels:
      severity: notify

  - alert: ContainerdHasLeftoverContainersRunning
    annotations:
      description: |
        Containerd on **{​{ .Labels.instance }}** has extra containers that aren't known to Kubernetes.
    expr: |
      (
        container_pids_current{namespace="k8s.io"}
          unless on (container_id)
        label_replace(
            kube_pod_container_info,
            "container_id", "$1", "container_id", "containerd://(.+)"
        )
        unless on (container_id)
        label_replace(
            kube_pod_init_container_info,
            "container_id", "$1", "container_id", "containerd://(.+)"
        )
      )
      unless on (container_id)
      (
        container_memory_swap_limit_bytes > 0
      )
    for: 15m
    labels:
      severity: notify

This I learnt

I'd never really used crictl much before this so it was good to be able to better understand it. One thing I didn't realise was it also has the concept of "pods" and "containers" which are different, but map to, those found in Kubernetes. A "pod" seems to refer to the sandbox container - that is, the container that is created to setup all the networking etc. that is needed for the workloads (this is the "pause container"). Pods can be listed with crictl pods and inspected with crictl inspectp [ID]. Containers can be listed with crictl ps and inspected with crictl inspect.

Before this I didn't really know what a CNI actually did in terms of things it created etc., and I still mostly don't, but I've at least seem some hints of it in the way of the network cache files that it creates.

Debugging Talos when it goes wrong at the host level is tricky as you have limited access to the node itself. This is often good as it prevents you from being able to make live changes that could be forgotten or reverted accidentally but when you actually need to fix something like this it just gets in the way. I'm very thankful for kubectl debug (and Rory's wonderful alpine-containertools image 💙) that makes this debugging possible and relatively painless.

2025-05-10: Added advice from Josh Clark

2025-05-03: Added some advice from Lobste.rs comments

2025-05-03: Added advice from Aaron Patterson about repeating questions back

2025-05-01: Added advice from Márk Sági-Kazár

2025-05-01: Added suggestion from Oriel about going to the toilet before speaking.

I've been giving talks at meetups and conferences for a few years now. I started off after the encouragement of my friends giving their own talk and looking so cool doing it! It's taken a while but I think I'm at a stage now where I'm not only good at it (at least I hope so 😅) but I feel confident and comfortable while doing it. I want everyone to have that same confidence and I want to hear ALL OF YOU giving talks too! You have stories to tell, lessons to share and experience to pass on. So here is my learnings on how I approach giving a talk in front of a crowd of techies, mainly focussed on technical talks but most of this should apply to most public speaking.

Before we get into it, I just want to set some expectations.

These are things that have worked for me, including advice I've been given by other speakers over the years, but that doesn't always mean they will work for you. We're all unique in weird and wonderful ways so if you find something doesn't work for you or isn't comfortable then ignore it and focus on the things that do help. 💙

I'm also not going to cover (much) about how to come up with a talk idea or how to get it accepted at a conference or meetup. I also have difficulty here and don't feel confident enough to offer advice on this.

I'm also not a "professional" speaker. This isn't my full-time job, even if I sometimes do it on behalf of my company. But that is also why I think all of you can also be giving talks and sharing your knowledge with the wonderful community.

So, let's dive into it shall we?

Coming up with the talk

As I said above, I can't really give much advice here. I struggle with this a lot of the time but there is one thing I can say for sure...

Talk about what you’re interested in and enjoy - if you don’t enjoy the subject your audience won’t either. It is also likely you'll have a terrible time giving the talk (not to mention writing the talk) if it's not something you enjoy.

One other thing I can say on this matter though is just because someone else has given a talk on a specific topic doesn't mean you cannot. We each bring a unique view to a topic and our delivery of the subject can resonate and connect with people differently.

Writing the talk

The language used

Make sure the language you use is inclusive and avoid words that trivialises actions. For example, avoid words like "guys" and "master" as much as you can to ensure you're not making anyone feel excluded - I like to use "y'all" when referring to a group of people. I also recommend limiting your use of words like "basically" and "simply" - these can feel alienating sometimes when the thing your talking about isn't seen as being basic or simple to the audience.

It's generally a good idea to avoid jokes unless you're sure they'll land and are designed to lift people up, not belittle anyone. A lot of the time jokes require some sort of pre-existing cultural knowledge that might not be the case for all your audience members. This is especially true at international conferences where you have people from different countries and languages in the audience.

Don't perpetuate stereotypes - avoid things like "so simple even my mum can use it" or similar comments.

Don't be afraid of saying "um" or pausing in your talk. In fact I suggest you make use of pauses in your speaking to add emphasis and break up sections.

Your slides

Design is not my strongest skill so I'm going to keep these brief and to the point:

• Make your font sizes BIG so that people at the back of a large room can see without struggling
• Avoid paragraphs - no one is there to read, they're there to listen
• Limit the amount of fonts and variations you use. Pick one main text font and one mono font. Use bold, italics and colour sparingly and to add emphasis where needed.

If you're going to be presenting your slides on a projector then your slides will always look very washed out compared to what you see on your computer screen. Plan ahead for this and make sure your slides have good contrast between the text and background. If you're worried about this I suggest speaking with the organisers at the start of the day and testing out your slides in the room you'll be presenting.

I recommend including your social media handle on each of your slides if you can. People take photos of slides to remind themselves of the content later and there's a good chance they would have forgot who gave the talk by then.

I also recommend having a final slide with links to get a copy of your slides if possible and where people can get in touch with you to ask questions after the talk.

Always remember that your slides are only there to reenforce what you're saying and are not the total content. They should enhance and summarise your points as you go so keep them brief and snappy.

On a similar note to avoiding paragraphs, stip points out on lobste.rs that:

you should limit code on slides as much as possible, ideally only to a few lines. In my opinion, most points can be made with just a function signature or a several-line toy function. Showing the audience more than that risks losing them while they try to figure out what is going on.

If you really need to put a long function on a slide, try and highlight the relevant section as you’re talking about it so the audience can follow along better.

This is great advice! Techies in the audience will automatically try and read all the code and figure out what it does rather than listen to you talking. For this I like to use contrast of text colour to highlight the bits I want people to focus on and "fade out" the rest of the code that is just boiler plate or will be relevant later. I will often have the same code on multiple slides but each slide highlights a different bit of the code so I can walk the audience through it rather than dumping it on them all in one go.

Technical

Most presentation software has the ability to write speakers notes. I don't personally use these myself as I can't read fast enough to make them useful but if you feel more confident with them then I totally recommend using them. The main thing to keep in mind though is not to use them as a script but rather just short notes to remind you of your place in the talk.

Ensure you know how to make your slides fullscreen before giving the talk. This may sound obvious but if your using Google Slides from Chrome on a Mac, for example, then you will actually need to enable to ability to go fullscreen in your browser first. I've seen this catch people out several times. If this is your setup then you want to go to View → Always Show Toolbar in Full Screen and make sure that is NOT checked. 😉

Demos go wrong and conference WiFi sucks. Be sure to have offline alternatives of your slides and backup options for demos going wrong. A video recording of your demo is always a good choice. There's rarely ever a need to do a live demo, I certainly never do - my typing skills and spelling are terrible 🤣.

Ending the talk

Always end the talk with a “thank you” and not “any questions”. When you ask for questions it causes confusion for the audience as to if they should clap or not. Let the conference / meetup organiser ask if there are questions as they also know if there’s time available and if there is a portable microphone for the audience to use.

On the day

Setting up your tech

Make sure you arrive prepared. This means having any adapters or dongles you need for your laptop to connect to the provided HDMI cable, that your laptop is sufficiently charged (at least 50%) and you have an offline copy of your slides available as we've already said the WiFi is unreliable.

Set your laptop and phone to Do Not Disturb mode. The last thing you want is an embarrassing notification to pop up for all to see.

Ensure your laptop is clear of clutter - arrange the windows you'll need during your talk, close or hide all others. It's never fun watching someone switch between multiple windows or tabs trying to find where their slides or demo went to.

Taking the stage

Just before you step up on to the stage I recommend removing your conference badge so it doesn't distract or get caught on anything and emptying your pockets to avoid things jingling or distracting you mid-talk.

Be sure to stay hydrated while giving your talk. If the organisers don't provide you with a bottle of water be sure to get your own drink to take up with you. Often the stage will be lit by very bright and very hot lights and you will feel your throat going dry in no time!

I can't believe I forgot this, so thank you to Oriel Jutty for reminding me about this one - make sure you go to the toilet before your talk! Especially withe the "stay hydrated" tip above you really don't want to get a few minutes into your talk and then be distracted by your bladder. I usually end up going to the toilet several times in the hour running up to my talk! 😅

Giving the talk

I like to use a presentation remote to control my slides as I like to move around the stage when I'm talking. This isn't required but I recommend it rather than hiding behind your laptop the whole time clicking the keyboard. My current clicker is this finger ring presentation remote. The added benefit of using a remote is it gives you something to do with (one of) your hands.

On a similar note, don't bother with a laser pointer. They're rarely visible to all attendees and are completely lost on the recording of the talk (if there is one). If you do want to point to things on the slides I recommend looking into software based laser pointer or just using your mouse cursor.

A quick word on microphones

When it comes to giving a talk to a medium to large sized crowd you'll almost certainly be using a microphone to amplify your voice. You often don't get a choice in what microphone you can use but if you do here are my thoughts on the different styles:

• Podium - These are usually fixed to the podium and you need to always face that direction when talking. Remember not to talk while pointing at the screen behind you as people won't hear you.
• Handheld - These should be held right up to your mouth, closer than you initially think. If you start to drift it away from your face the sound quality degrades very quickly. Be aware that these can be awkward if you plan to do live demos where typing is needed as one of your hands will need to be holding the mic.
• Lapel - These clip on to your clothes, usually near the collar, so avoid loose or noisy clothing that could cause noise on the mic
• "Britney" style - Not sure the correct name for these but they're the ones fitted over your ear with a small arm mic on them. These are my favourite as you don't need to hold them and you can move around with the, just be careful if coughing as the sound isn't nice.

On a related point, try to avoid needing to play any sound from your laptop during your slides or demo. If you absolutely need sound for your talk then please discuss it with the organisers before the event so they can ensure it is catered for.

Questions

As said above, end your talk with a "thank you" and let the organiser initiate questions if there is time.

Be aware that you don't need to be all knowing - if you don't have an answer for a question it is totally acceptable to say "I don't know".

If you think an answer might require considerable time or be confusing to explain from the stage then suggest that the person asking the question meets you after the session to discuss it further.

One thing Aaron Patterson pointed out on Bluesky that I hadn't included here is:

during Q&A try to repeat the questions. Audience members don't always have a mic so repeating will help everyone, it makes sure you understand the Q, and buys you time to think of an answer

After the talk

So, you did it! You gave the talk! Congratulations! 🎉

Now what?

Well, first things first - it's always good to make yourself available immediately after your talk as it's possible people will want to come ask you questions or possibly give you feedback or thanks. I also like to take this time to "unwind" a little after my talk and make any notes on things that I think might need changing or improving for next time.

On that note - yes, there should be a next time. If your talk was a success and you enjoyed giving it then why stop there? There are always more people that will want to hear what you have to say so don't be afraid of reusing talks at other events. Just be sure not to simply copy and paste each time but work to improve and update as you learn and grow.

My most important tip

I forget who gave me this advice, many years ago, but it's stuck with me all this time - Celebrate every talk - I like to have a “little treat” after (doughnut, muffin, something tasty) but do whatever is nice and rewarding for you. You put a lot of work into it and regardless of how it went on the day you deserve to celebrate that achievement.

If you have any tips of your own I'd love to hear them! Let me know on Bluesky or Mastodon.

Finally I'd like to leave you with a collection of resources I have learnt from over the years...

Resources

Books

• Resonate: Present Visual Stories That Transform Audiences by Nancy Duarte
• Slide:Ology: The Art and Science of Creating Great Presentations by Nancy Duarte
• Demystifying Public Speaking by Lara Hogan

Websites

• speaker-starter-kit
• speaking.io
• Self-Defined

Blog posts

• I can’t read your slides
• Preparing a talk - Part 1, Part 2, Part 3
• Public Speaking for Beginners
• Inclusionary and Exclusionary Language
• A white-label slide deck

Videos

• The art of slide design

Advice from Friends

After posting this I started asking for other people's tips and advice. I will update below as I get more to share.

Márk Sági-Kazár

I see folks sometimes struggling with coming up with the content. They have the title and the topic, they have the thank you slide, but they have trouble with the slides in between. I usually recommend writing a draft first instead of thinking in terms of slides. Number one rule of writing is that you don't write and edit at the same time.

Personally, I use pen and paper, because it slows my thoughts and I don't bounce between ideas.

I can totally see how this would be useful for many folks. Myself I actually prefer the opposite, I throw together a bunch of slides with my various points on them and keep rearranging and refining them until they fit the story I want to tell. The key thing is, try various approaches and see what works for you. It might also not be the same approach for each talk you build - creativity can be fickle sometimes.

Another common mistake is time: less experienced speakers can't estimate how long they are going to talk with a given set of slides. One reason for that is they don't rehearse (I'm not saying it's a must, but it definitely helps). Another reason (even when they rehearse) is they lose focus and they stretch out some of the slides. Rehearse and practice focusing on the key ideas on each slide.

I should have included this in my post initially. Rehearsing a talk is a very personal thing I've found and everyone does it differently but it is very important to have a good idea of how long your talk is going to last. My top tip for this is to open a Google Meet (or your video conference of choice), start the recording and then present the entire talk as though you were giving it in front of a crowd. You don't need anyone else in the call with you, just let the recoding go until your done and then once complete you can look at the timestamp and see how long your talk takes. You don't even have to watch yourself back if you don't want to. 😅

But my number one tip (or rather the one I keep repeating the most): don't look at the flippin' screen behind you. 😄 If you are not wearing a mic (eg. at a small meetup) noone is going to hear you. If you are wearing a mic, it just looks bad, especially if there is a recording.

😅 Yeah - I'm at fault for this quite often to be honest. I like to point at things and be quite physically expressive.

Zoom in on your terminal windows.

Bigger is always better here. Assume the person sat at the back of the room also has the worst possible eyesight.

Disable true tone (sometimes it makes screen recording yellowish)

Solid practical tip if you use true tone on your machine. Not something I've ever thought of before.

tomhukins on lobste.rs

In one of my talks, I completely forgot what I wanted to say. I noticed myself panicking, so took a deep breath, paused and looked at the audience. When I watched the video of my talk, it looked like a natural pause in the presentation. Although saying “um” as a pause is okay, silence works better.

I’m a native English speaker and I often present to people who aren’t. I try to speak slowly and clearly, and to use simple or common words whenever possible. I want to communicate an idea, not to sound clever.

Sharing links and contact details on a final slide makes sense: include a QR code for the audience to photograph.

Josh Clark on LinkedIn

In demos, make sure the relevant info (e.g. terminal commands) is in the middle third of the screen. The top of the projector screen can often be washed out or reflecting the overhead lights in the room, and the bottom is often cut off by people's heads.

This is actually really good advice for all your slides. Having padding around the edges of your presentation theme will help you deal with this issue and any wonky projector issues that you might face.

And when I'm juggling multiple windows (slides, terminal, wireshark, etc) in a presentation, I like to set them up in individual virtual desktops on my mac. In rehearsals, I make sure the order of those desktops is conducive to swiping between them.

This is a great practical tip for how to handle the "cleaning your desktop" advice I have above! Much slicker than moving around windows or finding the appropriate tab or whatever.

Intro to the bots

Before getting into the details, lets take a little look at what bots I actually run.

• Blog posts - Whenever I publish a new blog post I automatically create a new post on my personal Bluesky, Mastodon and LinkedIn profiles.
• CloudNative.Now - A monthly newsletter that I run that posts to Bluesky and Mastodon whenever a new issue is released.
• Conf.Party - Similar to above, this is a site I run to collect parties happening around conferences. It posts to Mastodon and Bluesky when new parties are added.
• Kubernetes Releases - I have a bot post to Mastodon and Bluesky whenever there is a new Kubernetes release available on GitHub.
• Mastodon Releases - I have a bot post to Mastodon whenever there is a new Mastodon release available on GitHub.
• Dropout.Tv Releases - A bot that posts to Mastodon and Bluesky whenever there is a new video on the Dropout.TV new releases page.

The bots each post a customised message template that remains the same for each post.

How this looks in Node-RED

Almost all the bots follow roughly the same flow:

1. They are triggered by a 15 minute interval inject node.
2. An RSS feed with the source data is fetched (this is something like my blog RSS feed or the GitHub releases RSS feed).
3. Parse the RSS feed XML.
4. Iterate through each item in the feed and for each one check if the guid of that item already exists within a bot-specific context data store.
5. If guid is not found in existing list include that item in the results then add the guid to the list and store into the context data store.
6. Split the returned list of items into individual items.
7. Build a message template for each social media platform.
8. Send that message to a platform-specific subflow that handles the actual API call needed for each platform. (More on this below)

The exception to this is the Dropout TV bot. There isn't an RSS feed for the new releases so I had to do some web scraping instead and it took a bit of trial and error. I wont go into detail for this as it's very specific to this bot.

Lets take a look at each step in a bit more detail...

1. Inject Node

Nothing special here, it's an Inject Node with a repeat interval set to 15 minutes. Set this to whatever makes most sense to your bot but be sure to be respectful of the datasource you're fetching from and not to call too often.

2 & 3. Fetch RSS Feed

Again, nothing special here, just a HTTP request node that fetches the URL of the RSS feed and passes it on to an XML node to parse it to the JSON payload.

4. Check for updates

This is where we start to implement some logic. We use a Function Node with the following custom JavaScript:

// Grab our bots context data store or initialise it as an empty array
let prev = flow.get("kubernetes_releases") || [];
// Grab all the guids from our incoming RSS feed data
let guids = msg.payload.feed.entry.map(a => a.id[0])
// Create an array to store new posts in to return at the end
let newPosts = [];

// For each of the guids check if our existing context data contains it already
// If not, add the guid to the existing and add the whole post to the newPosts array
guids.forEach((guid, i) => {
    if (!prev.includes(guid)) {
        prev.push(guid)
        newPosts.push(msg.payload.feed.entry[i]);
    }
});

// Make sure we update our context data store with our updated list of entries
flow.set("kubernetes_releases", prev);

// Return all the new posts
return {
    payload: newPosts
};

5. Split

Use a Split Node to split the array returned from the previous function into single calls - Array split using fixed length of 1.

6. Generate Post Message

This will likely be different, more or less, for each bot as you'll want to build something that makes sense for the context of your bot.

The result of this function will also vary depending on the platform as different platforms require different values for authentication and similar. I'll cover these in more detail below when I look at each provider individually.

As an example, here's the function I use for the Mastodon release bot:

// Grab the title of the Release from the incoming RSS item.
// Depending on the feed structure this might be different.
let title = msg.payload.title[0];

// Handle some special cases where the release version contains "rc"
// as we want to make it clear that this is a Release Candidate release.
let variant = "";
if (title.includes('rc')) {
  variant = " Release Candidate";
}

// Build the post text
let postBody = `New Mastodon${variant} Release

:mastodon: ${title} :mastodon:

${msg.payload.link[0]['$'].href}

#Mastodon #Mastoadmin
`;

// Return the payload that will be passed into the Mastodon subflow
return {
  // The Mastodon instance the bot belongs to
  instance: "mastodon.social",
  // The dev access token for the bot
  password: "XXXXXXXXXXXXXXX",
  // The post text
  body: postBody
}

7. Post to Platform

This is different for each platform so we'll look at those individually below...

Posting to Mastodon

Posting an update to Mastodon is fairly simple. You'll first need to generate a new application access key (Settings -> Development -> New application) and make sure it has the write:status scope. Make a note of the access token as we'll be using this in our payload.

This subflow expects 3 properties on the input message:

• instance - This is the Mastodon instance the bot belongs to and to which the API request will be made.
• password - This is the access token of our development applications.
• body - The actual text to post as the status update

With these 3 values we then first format the API payload in a function node using the following JavaScript:

return {
  payload: {
    status: msg.body,
    visibility: "public"
  },
  url: `https://${msg.instance}/api/v1/statuses`,
  headers: {
    "Authorization": "Bearer " + msg.password
  }
}

This is then passed to an http request node with the method set to POST which will perform the API call and then finally we output it to a debug node so we can check for any errors returned from the Mastodon API.

Posting to Bluesky

The Bluesky API is somewhat more complex to work with compared to Mastodon.

First off, we'll need to generate a new App Password to authenticate out bot (no need to direct message access).

We also need the Decentralized Identifier (DID) of our bot user. There's a couple ways to get the DID, you can either use the ATProto Browser website and enter the link to the user profile or you can make a CURL request to get the profile details, e.g.:

HANDLE='averagemarcus.bsky.social'
DID_URL="https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile"
curl -G --silent --data-urlencode "actor=$HANDLE" "$DID_URL"

The DID will look something like did:plc:mtepw4cvbmdvu7zygmm5xbop.

This subflow expects 3 properties on the input message:

• appPassword - The password we just generated for our bot.
• did - The DID of our bot.
• body - The actual text to post as the status update

Creating an API session

The first thing we need to do when working with the Bluesky API is to create a new session, using our DID and app password. In a function node we format these into the required payload:

msg.payload = {
  "identifier": msg.did,
  "password": msg.appPassword
};

return msg;

We then pass this to an http request node with the following values set:

• Method = POST
• URL = https://bsky.social/xrpc/com.atproto.server.createSession
• Headers = Content-Type: application/json

And finally have this node output to a json node to parse the response into the payload for us to make use of in the next stage.

Building the post message

This is where things get really complicated. The ATProto isn't as user-friendly as the Mastodon API. You can't just pass it a block of text and it'll handle it and turn it into a nicely processed status post. Instead, you need to send you text along with all the details of the hashtags, links, media and mentions that are contained in your post.

To be able to do this, we need a couple extra node modules to be made available to our Node-RED setup. In your Node-RED config directory make sure the following packages are also installed:

• node-fetch
• @atproto/api

Now, in out Format post function node we need to reference these two packages in our "Setup" tab:

The function itself then looks something like this:

const getBlueskyAgent = async () => {
  const agent = new atproto.AtpAgent({
    service: "https://bsky.social",
  })

  await agent.login({
    identifier: msg.did,
    password: msg.appPassword,
  })

  return agent
}

const getUrlMetadata = async (url) => {
  const req = await fetch(`https://cardyb.bsky.app/v1/extract?url=${url}`)
  const metadata = await req.json()

  return metadata
}

const getBlueskyEmbedCard = async (url, agent) => {
  if (!url) return

  try {
    const metadata = await getUrlMetadata(url)
    const blob = await fetch(metadata.image.replaceAll('&', '%26')).then(r => r.blob())
    const { data } = await agent.uploadBlob(blob, { encoding: "image/jpeg" })

    return {
      $type: "app.bsky.embed.external",
      external: {
        uri: url,
        title: metadata.title,
        description: metadata.description,
        thumb: data.blob,
      },
    }
  } catch (error) {
    console.error("Error fetching embed card:", error)
    return
  }
}


const agent = await getBlueskyAgent()
const rt = new atproto.RichText({ text: msg.body })
await rt.detectFacets(agent)

let url = rt.facets.find(f => f.features && f.features.length && f.features[0].uri);
if (url) {
  url = url.features[0].uri
}

return {
  payload: {
    "collection": "app.bsky.feed.post",
    "repo": msg.did,
    "record": {
      "createdAt": (new Date),
      "$type": "app.bsky.feed.post",
      text: rt.text,
      facets: rt.facets,
      embed: await getBlueskyEmbedCard(url, agent),
    }
  },
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer " + msg.payload.accessJwt
  }
};

There's a lot going on here so I'm going to break it down...

First we have a function that generates a new Bluesky client for us to use to handle most of the actual work. We use our DID and app password as the login credentials to be able to perform any actions tat need authorisation.

const getBlueskyAgent = async () => {
  const agent = new atproto.AtpAgent({
    service: "https://bsky.social",
  })

  await agent.login({
    identifier: msg.did,
    password: msg.appPassword,
  })

  return agent
}

Next we have a couple helper functions to help with URL previews. First calls a service that generates metadata about an URL:

const getUrlMetadata = async (url) => {
  const req = await fetch(`https://cardyb.bsky.app/v1/extract?url=${url}`)
  const metadata = await req.json()

  return metadata
}

This returns something along the lines of the following:

{
  "error": "",
  "likely_type": "html",
  "url": "https://marcusnoble.co.uk/index.html",
  "title": "Blog",
  "description": "The blog of Marcus Noble, self-described tinkerer, platform engineer and all round average guy!",
  "image": "https://cardyb.bsky.app/v1/image?url=https%3A%2F%2Fopengraph.cluster.fun%2Fopengraph%2F%3FsiteTitle%3DMarcus%252BNoble%26title%3DBlog%26tags%3D%26image%3Dhttps%253A%252F%252Fmarcusnoble.co.uk%252Fimages%252Fmarcus.jpg%26bluesky%3D%2540averagemarcus.bsky.social%26fediverse%3D%2540marcus%2540k8s.social%26github%3DAverageMarcus%26website%3Dwww.MarcusNoble.co.uk%26bgColor%3D%2523ffffff%26fgColor%3D%2523263943"
}

This output is used in the next helper function to get our preview card details:

const getBlueskyEmbedCard = async (url, agent) => {
  if (!url) return

  try {
    const metadata = await getUrlMetadata(url)
    const blob = await fetch(metadata.image.replaceAll('&', '%26')).then(r => r.blob())
    const { data } = await agent.uploadBlob(blob, { encoding: "image/jpeg" })

    return {
      $type: "app.bsky.embed.external",
      external: {
        uri: url,
        title: metadata.title,
        description: metadata.description,
        thumb: data.blob,
      },
    }
  } catch (error) {
    console.error("Error fetching embed card:", error)
    return
  }
}

This builds an "embed" record for us to use with our post and will cause Bluesky clients to show a preview card of an URL contained in our post message.

With the helper functions defined we can start building our message. First off, we initialise a new client and parse our message body for any "facets". In the ATProto "facets" are things like hashtags, urls and mentions - the things that and up being links when rendered. We need to define these manually in our payload to the API so the @atproto/api module helps us generate these.

const agent = await getBlueskyAgent()
const rt = new atproto.RichText({ text: msg.body })
await rt.detectFacets(agent)

With the facets generated we then want to check if they contain any URLs and if so grab the first one found. We will use this to generate the preview card using our helper functions above.

let url = rt.facets.find(f => f.features && f.features.length && f.features[0].uri);
if (url) {
  url = url.features[0].uri
}

Finally we build our actual API payload. There's a few things going on here that I don't want to go into too much detail about but if you're interested then take a look at the Bluesky docs or the ATProto SDK docs.

return {
  payload: {
    "collection": "app.bsky.feed.post",
    "repo": msg.did,
    "record": {
      "createdAt": (new Date),
      "$type": "app.bsky.feed.post",
      text: rt.text,
      facets: rt.facets,
      embed: await getBlueskyEmbedCard(url, agent),
    }
  },
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Bearer " + msg.payload.accessJwt
  }
};

The main thing to be aware of here is the "collection" of app.bsky.feed.post tells the API that this is a status post, the "repo" is our bot user and then the "record" contains our post text, the "facets" we generated with the @atproto/api module and an "embed" containing details of an URL preview card if an url was found in our message.

Finally, the Authorization header makes use of the accessjwt of the session we created in the previous stage.

With the message now, finally, build we can pass this on to an http request node that sends a POST request to https://bsky.social/xrpc/com.atproto.repo.createRecord and then finally sends the returned output to a debug node so we can check for errors.

Posting to LinkedIn

I actually wrote about this last week - Posting to LinkedIn via the API. The Node-RED implementation is just formatting the payload and then sending the HTTP request to the LinkedIn API.

let body = msg.body;
let urn = msg.urn;
let access_token = msg.access_token;

return {
  payload: {
    "author": "urn:li:person:"+urn,
    "lifecycleState": "PUBLISHED",
    "specificContent": {
      "com.linkedin.ugc.ShareContent": {
        "shareCommentary": {
          "text": body
        },
        "shareMediaCategory": "ARTICLE",
        "media": [
          {
            "status": "READY",
            "originalUrl": msg.link
          }
        ]
      }
    },
    "visibility": { "com.linkedin.ugc.MemberNetworkVisibility": "PUBLIC" }
  },
  url: `https://api.linkedin.com/v2/ugcPosts`,
  headers: {
    "Authorization": "Bearer " + access_token,
    "LinkedIn-Version": "202210",
    "X-Restli-Protocol-Version": "2.0.0"
  }
}

Twitter

It's dead. The API isn't accessible anymore. Move on.

✨ Bonus ✨ - Archiving to Wayback Machine

As several of my bots are triggered from new posts being created on either this blog or new newsletter posts on CloudNative.Now I realised I could leverage the same flow to also ensure that the Wayback Machine on Archive.org grabs a copy of the new page to cache it.

The subflow to handle the Wayback Machine calls isn't too complex:

The subflow splits the incoming call into two flows - one that caches the new page and another that re-caches the homepage as this is also updated with the new post included.

Formatting the URL for the new page is very simple, we just take the URL of the page from the incoming message and prepend the Wayback Machine URL - https://web.archive.org/save/:

return {
    url: `https://web.archive.org/save/${msg.payload.link[0]}`
};

This then output to an http request node that calls the Wayback Machine save URL with out new page URL that triggers it to create a new cached copy for us.

The homepage flow is slightly more work, we just need to grab the homepage link from the new post URL:

let link = msg.payload.link[0].replace('https://', '').split('/')[0]

return {
    url: `https://web.archive.org/save/https://${link}`
};

And then again pass this on to the http request node to trigger the cache process.

Wrap Up

Hopefully this is helpful to some of y'all. Even if you don't end up using Node-RED for your bots there should be enough info here on how to make use of the different APIs to help you along.

I'd love to hear about what bots y'all are making - let me know on Bluesky or Mastodon!

Creating a new App

I'm going to assume you already have a LinkedIn account, if not you will need to create one or this whole process is pretty pointless. 😅 Once logged in, head over to https://www.linkedin.com/developers/apps/new and fill out the details.

Note: For this you require a "company page" that the app will be associated with. You can create one easily enough that can just be used as a placeholder for the app.

Once your app is created the first thing you'll need to do is head to the "Settings" tab and follow the process to verify your app. Not sure why this isn't part of the app creation flow but whatever. 🤷

With your app verified you now need to enable the "products" you'll be using so switch to the Products tab. For sharing posts to our LinkedIn profile we're going to need to enable the following:

• Share on LinkedIn
• Sign In with LinkedIn using OpenID Connect

Finally, switch over to the "Auth" tab and scroll down to the "OAuth 2.0 scopes" section, you should hopefully be seeing something similar to the following:

Creating an Access Token

Now that we have an app created we need to generate a new access token to use when making API calls. Thankfully, this is now very simple as LinkedIn provide a tool to generate a token whereas previously users were required to run a server to handle callbacks, etc.

Navigate to https://www.linkedin.com/developers/tools/oauth/token-generator, select your app from the dropdown, check all the scopes available and click the "Request access token" button.

You will likely need to login again here but once done you should be presented with an access token that will expire in 2 months.

Keep this safe and DO NOT SHARE IT.

Using the API

Now that we have an access token we can finally make calls to the API!

First, lets get our own user profile details as we're going to need our URN to use when posting status updated.

curl --silent -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  "https://api.linkedin.com/v2/userinfo"

{
    "sub": "XXXXXXX",
    "email_verified": true,
    "name": "☁️Marcus Noble",
    "locale": {
        "country": "US",
        "language": "en"
    },
    "given_name": "☁️Marcus",
    "family_name": "Noble",
    "email": "",
    "picture": ""
}

Replace the ${ACCES_TOKEN} with your newly generated token. The response you get back should look similar to the above JSON but obviously with your LinkedIn details. Make a note of the sub value as we'll be using this next.

To post a status update to your profile:

ACCESS_TOKEN="xxx"
URN="xxx"
POST_BODY="This is a test post"

curl -X POST \
  -H "LinkedIn-Version: 202210" \
  -H "X-Restli-Protocol-Version: 2.0.0" \
  -H "Authorization:  Bearer ${ACCESS_TOKEN}" \
  --data '{"author": "urn:li:person:'${URN}'","lifecycleState": "PUBLISHED","specificContent": {"com.linkedin.ugc.ShareContent": {"shareCommentary": {"text": "'${POST_BODY}'"},"shareMediaCategory": "NONE"}},"visibility": {"com.linkedin.ugc.MemberNetworkVisibility": "PUBLIC"}}' \
  "https://api.linkedin.com/v2/ugcPosts"

Fill in the environment variables with your appropriate values and run the curl command to post a new status update to your profile.

Note: If you include any " in your post body you will need to escape them or it will cause the JSON to be invalid.

For full details on what is available when sharing to LinkedIn check out the official documentation.

2025-01-17: Added details about Cloudflare

Last week I announced the launch of a new project I've been working on - ✨ Cloud Native Now ✨ - a new monthly newsletter that will provide a roundup of all the happenings in the cloud native world. This newsletter is my attempt at keeping myself, and others, up-to-date on all the latest news, tools and events happening in the cloud native world. A new issue will be published each month on the last Friday of that month and contain a roundup of articles, announcements, tools, tutorials, events and CFPs relating to cloud native technologies and the community.

I hope y'all will subscribe to the email newsletter or add the RSS feed to your favourite feed reader to make sure you don’t miss anything! Or, if you prefer, you'll be able to view the entire archive directly on the website.

There is also a Bluesky and Mastodon account if you prefer to follow those. If you have any suggestions for items to include or any improvements / changes that you think would be good then please do reach out. I'd love to hear from you!

Technical Details

Now, I'm sure y'all are interested in all those tasty technical details!

Infrastructure

There's nothing too wild going on, the majority of things are running on a Civo Kubernetes cluster along with a MySQL database and object store. The infrastructure is managed using OpenTofu and applied by a GitHub action when changes are merged into the main branch.

OpenTofu also handles installing Flux into the cluster which points back to the same repo and handles managing Kubernetes resources via GitOps. The setup is mostly the same as outlined in my previous post - Bootstrapping a Civo cluster with OpenTofu and Flux. Along with installing Flux into the cluster, OpenTofu also creates a few Kubernetes Secrets containing various credentials that will be needed by applications (Civo creds for cluster-autoscaler, database credentials, object store credentials).

Flux is then setup to handle some core applications (sealed-secrets, metrics server, etc.) and then applies some extra Kustomizations that handles installing apps and extra-config (these are things that require apps to be installed first so have a dependency).

Applications

There's not many applications that make up Cloud Native Now, the main is a self-hosted Ghost powered site that handles the website, newsletter and RSS feed. This provides a very nice content management system for creating posts with very little effort. For the actual sending of the newsletter, Mailgun is the service of choice here. The posts and content for Ghost is all stored in a Civo-managed MySQL database and object store. For the theme, I forked the Alto theme and applied some custom tweaks in my own repo.

Within the cluster there is also ingress-nginx (for ingress), cluster-autoscaler (for dynamic worker node sizing) and cert-manager (for SSL certificates).

Backups are very basic right now - a couple CronJobs that run regularly to perform a MySQL dump and a copy of the object store content into another backup object store (with some versioning in place).

To handle the collation of entries for the newsletter throughout the month I've put together a custom Grist project that I use to record links and data as I find them. Unfortunately, this isn't public and I don't have anything to share. To make this easier to work with I built a custom url-to-grist tool that allows me to share an URL to it and it'll automatically import it into Grist and attempt to grab some metadata about the page to help with putting together the newsletter.

I'm also re-using my existing Node-RED setup to have new posts (based on the RSS feed) automatically posted to both the Mastodon and Bluesky accounts. Again, this is unfortunately not public but would be happy to chat about it if anyone is interested.

One area that still needs improvement is monitoring. Currently there is some blackbox monitoring from my local infrastructure to ensure pages are reachable and I have some (temporary) MySQL monitoring to keep an eye on the state of the database. This is an area I'll be focussing more on over the next few months to ensure uptime is maintained as much as possible.

Issues along the way

It hasn't been all smooth sailing so far. I had some trial-and-error with OpenTofu and getting the infra set up how I wanted. This was mostly due to me having not worked with it for a long time and just bumbling through it.

One major issue I have experienced a few times other the past week or so is database connectivity problems. It's still unclear exactly what happened and I've been working with Civo support to try and figure this out but it looks like somehow the database was being overwhelmed. I've now got monitoring in place to try and catch this happening again but as of now it hasn't yet occurred again.

It looks like I had also misconfigured the email sending at some point between launching and publishing this blog post. 😅 This is now fixed and sign ups should be working again now.

The Future

There's several things I already have planned as well as some vague ideas for things I might do, depending on how popular this ends up becoming.

Besides improving the monitoring as mentioned above I plan to also implement Cloudflare as a cache in front of the website to ensure the database doesn't get overloaded too much (see below).

Eventually, after a couple issues or so to get a feel for what people want, I'm hoping to also set up an accompanying Podcast where I talk with a different guest each month about all the happenings in that months newsletter. I want to use this as a way to talk with all the ✨ wonderful ✨ humans in our cloud native community and get a wide and diverse array of thoughts and opinions to complement (and maybe contrast) with my own. I'm really hoping to not have this just become "another white guy with a podcast" but something that can really show the amazing individuals and efforts without our community.

If things go well and this ends up taking off, I have some loose plans on how to cover the costs as things grow. I guarantee that all newsletter editions will always be freely available without an account but I'm hoping to later introduce paid membership that comes with some extra goodies. Not sure what exactly yet but I'm thinking along the lines of being able to add comments to the posts, the warm good feeling of helping keeping things running and maybe some physical rewards (postcards, stickers, etc.) if I can figure out how to send to the EU. 😅 There's also a possibility to sponsored posts and job listings in the future but I need to come up with some requirements that these would need to meet first to ensure quality.

And who knows... maybe I'll go wild and look at putting together a conference that I've always wanted too!

Updates

I have now switched to using Cloudflare for the nameserver and proxying requests through their edge servers with a 2 hour cache added to most of the site. This should allow the site to remain up, and read-only, during any periods where the database (or other) is having trouble. I did have a small issue when setting this up that resulted in infinite redirects - turns out I needed to switch from "Flexible" to "Full" in the Cloudflare SSL configuration, otherwise nginx kept trying to perform a redirect to the SSL port as Cloudflare was sending traffic to port 80.

For those interested, this is the specific expression I'm using for the caching in Cloudflare with an Edge TTL of 2 hours:

(http.request.full_uri wildcard "https://cloudnative.now/*" and not starts_with(http.request.uri.path, "/ghost/"))
or
(http.request.full_uri wildcard "https://www.cloudnative.now/*" and not starts_with(http.request.uri.path, "/ghost/"))

If you'd rather skip this blog post and just see the final result you can find all the code in the repo bootstrap-civo-cluster-blog-post-example.

Pre-Requisites

For the purposes of this post the following are assumed:

• You have an account with Civo (if not, you can sign up now and get $250 credit to play with)
• You have your Civo API Token set as the CIVO_TOKEN environment variable in your terminal
• You have the OpenTofu CLI installed
• You have a GitHub repo created to contain all your GitOps source (this can be private, if you prefer)
• A GitHub personal access token

Repo Structure

First we'll setup our repo file structure so that we have the directories we need for all of our different GitOps resources.

From the root of your new repo:

mkdir -p {apps,flux,infra}

You should end up with something like the following:

📂 .
├── 📂 apps
├── 📂 flux
├── 📂 infra

Infrastructure

The first thing we'll want to do is get our infrastructure setup using OpenTofu. All our infra code will live within our new infra directory so lets go ahead and create the files we'll be working with:

cd infra
touch main.tf # This is our main file containing our resources
touch providers.tf # This contains the config for the providers we will be using
touch variables.tf # This will describe any input variables we plan to use
touch output.tf # This contains all the output variables we will generate from our new infrastructure

With our new project ready lets setup start setting up the providers we will be using. For this we will be making use of two providers: Civo and Flux. In our providers.tf lets define each of these providers (along with their current versions at time of writing this).

terraform {
  required_providers {
    civo = {
      source  = "civo/civo"
      version = ">= 1.1.3"
    }
    flux = {
      source  = "fluxcd/flux"
      version = ">= 1.2"
    }
  }
}

We also need to configure each of the providers with various details. For now we're just going to add placeholders as they will reference some resources we haven't yet created so we'll come back to fill these in later. For now, add the following placeholders to the bottom of your providers.tf file:

provider "civo" {
}

provider "flux" {
}

With our providers defined we can now initialize our project and have the providers downloaded so we can start working.

tofu init

Once this is done you'll notice some new files have been created in our infra directory:

📂 .
├── 📂 .terraform
│  └── 📂 providers
├── 📄 .terraform.lock.hcl
├── 📄 main.tf
├── 📄 output.tf
├── 📄 providers.tf
└── 📄 variables.tf

The .terraform.lock.hcl is our dependency lock file that we will want to commit into git but the .terraform file contains the downloaded providers and ideally we would leave that out of our repo and generate it as needed. So lets take a moment to create a .gitignore in the root our of repo to ensure we are only going to commit the files we care about.

**/.terraform/*
*.tfstate
*.tfstate.*
*.tfvars
*.tfvars.json
.terraform.tfstate.lock.info
.terraformrc
terraform.rc

Great! Now lets start creating some infrastructure!

Let's move over to our main.tf where we're going to define three resources - a Civo Network, a Civo Firewall and a Civo Kubernetes Cluster. In all of my examples, I will be using the imaginative name of example so replace it with whatever makes sense for you.

resource "civo_network" "example" {
  label = "example-network"
}

resource "civo_firewall" "example" {
  name                 = "example-firewall"
  create_default_rules = true
  network_id           = civo_network.example.id
}

resource "civo_kubernetes_cluster" "example" {
  name        = "ExampleCluster"

  firewall_id = civo_firewall.example.id
  network_id  = civo_network.example.id

  cluster_type       = "k3s"
  kubernetes_version = "1.30.5-k3s1"
  cni                = "flannel"

  write_kubeconfig = true

  pools {
    size       = "g4s.kube.small"
    node_count = 2
  }

  # This allows us to make use of cluster autoscaler.
  # If you don't plan to use cluster autoscaler you can remove this `lifecycle` block.
  lifecycle {
    ignore_changes = [
      pools["node_count"],
    ]
  }
}

It's looking good so far but before we attempt to create our infra we need to specify which Civo region we'll be using. Rather than specifying this on every resource we can configure it on our Civo provider which will then apply to all resources it creates if not set.

To make things more configurable, let's create a variable that we can use to set the region at runtime. In our variables.tf lets add a new region variable and give it a default value so we don't always need to provide it:

variable "region" {
  description = "The region to create all our Civo resources in"
  type        = string
  default     = "LON1"
}

With this in place we can return back to our providers.tf and set the region on our Civo provider by updating the civo placeholder we added earlier:

provider "civo" {
  region = var.region
}

We're now at a point where we could create some infrastructure. Lets first confirm everything is looking good by running tofu plan to get an overview of what will be created. You should see something similar to the following:

✨ tofu plan

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # civo_firewall.example will be created
  + resource "civo_firewall" "example" {
      + create_default_rules = true
      + id                   = (known after apply)
      + name                 = "example-firewall"
      + network_id           = (known after apply)
      + region               = (known after apply)
    }

  # civo_kubernetes_cluster.example will be created
  + resource "civo_kubernetes_cluster" "example" {
      + api_endpoint           = (known after apply)
      + cluster_type           = "k3s"
      + cni                    = "flannel"
      + created_at             = (known after apply)
      + dns_entry              = (known after apply)
      + firewall_id            = (known after apply)
      + id                     = (known after apply)
      + installed_applications = (known after apply)
      + kubeconfig             = (sensitive value)
      + kubernetes_version     = "1.30.5-k3s1"
      + master_ip              = (known after apply)
      + name                   = "ExampleCluster"
      + network_id             = (known after apply)
      + num_target_nodes       = (known after apply)
      + ready                  = (known after apply)
      + region                 = (known after apply)
      + status                 = (known after apply)
      + target_nodes_size      = (known after apply)
      + write_kubeconfig       = true

      + pools {
          + instance_names      = (known after apply)
          + label               = (known after apply)
          + node_count          = 2
          + public_ip_node_pool = (known after apply)
          + size                = "g4s.kube.small"
        }
    }

  # civo_network.example will be created
  + resource "civo_network" "example" {
      + cidr_v4        = (known after apply)
      + default        = (known after apply)
      + id             = (known after apply)
      + label          = "example-network"
      + name           = (known after apply)
      + nameservers_v4 = (known after apply)
      + region         = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Looking good! But, before we go ahead and apply this there's one more thing we should do first - define some outputs!

We would need at least a couple output values defined so we can access the new cluster. Lets create some outputs that report the cluster name, it's API endpoint and the KubeConfig file we can use to access it with kubectl. We'll define these in our outputs.tf file:

output "kubernetes_name" {
  value       = civo_kubernetes_cluster.example.name
  description = "The name of the Kubernetes cluster"
}

output "kubernetes_api_endpoint" {
  value       = civo_kubernetes_cluster.example.api_endpoint
  description = "The API endpoint of the Kubernetes cluster"
}

output "kubernetes_kubeconfig" {
  value       = civo_kubernetes_cluster.example.kubeconfig
  description = "The KubeConfig for the Kubernetes cluster"
  sensitive   = true
}

Note that for the kubernetes_kubeconfig we've set sensitive = true. This will prevent the contents from being printed out into any logs and leaking our credentials but will still be saved into the state so be careful.

Ok, I think we are ready to create some infra! You can choose to skip this and apply everything together at the end but if you want to check that things are working as expected you can run tofu apply and then follow the instructions output.

If all goes well you should see OpenTofu creating the new Civo Kubernetes cluster.

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

You may notice that a new file has also been created - terraform.tfstate. Assuming you correctly created the .gitignore earlier this file should be ignored by git and not committed into our repo as it will contain sensitive values!

Now is a good time to commit in your changes so far before moving on to the next section!

Installing Flux

Now that we have a Kubernetes cluster created we can now have Flux installed into it to handle our GitOps'd workloads.

Go back to our providers.tf where we'll configure our Flux provider to make use of our new Kubernetes cluster. For this we are going to reference output values from our Civo Kubernetes Cluster and make use of two built in OpenTofu functions - yamldecode and base64encode.

provider "flux" {
  kubernetes = {
    host = civo_kubernetes_cluster.example.api_endpoint

    client_certificate = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).users[0].user.client-certificate-data
    )
    client_key = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).users[0].user.client-key-data
    )
    cluster_ca_certificate = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).clusters[0].cluster.certificate-authority-data
    )
  }
}

As you can see above, we're making use of the api_endpoint output value for the host and we're using the kubeconfig output value with our two functions to parse the specific values we need from the yaml file and then encode them into base64 for use by Flux.

Flux also needs a git repo configured that it will use for all it's gitops source. To make this dynamic, lets define some variables in our variables.tf that we can use to configure this at runtime. We're going to add three new variables - one will be to pass in a GitHub Token and the other two will define the repo.

variable "github_token" {
  description = "GitHub token"
  sensitive   = true
  type        = string
  default     = ""
}

variable "github_org" {
  description = "GitHub organization"
  type        = string
  default     = "AverageMarcus"
}

variable "github_repository" {
  description = "GitHub repository"
  type        = string
  default     = "bootstrap-civo-cluster-blog-post-example"
}

Here you can see I've set default values that point to the example code repo for this blog post.

Now, lets go back to our Flux provider config and add the git config, using our variables:

provider "flux" {
  kubernetes = {
    host = civo_kubernetes_cluster.example.api_endpoint

    client_certificate = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).users[0].user.client-certificate-data
    )
    client_key = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).users[0].user.client-key-data
    )
    cluster_ca_certificate = base64decode(
      yamldecode(civo_kubernetes_cluster.example.kubeconfig).clusters[0].cluster.certificate-authority-data
    )
  }

  git = {
    url = "https://github.com/${var.github_org}/${var.github_repository}.git"
    http = {
      username = "git"
      password = var.github_token
    }
  }
}

Great! We're almost there! The last thing we need to set up Flux is define a flux_bootstrap_git resource. Back in our main.tf add the following to the bottom:

resource "flux_bootstrap_git" "example" {
  embedded_manifests = true

  # This is the path within our repo where Flux will watch by default
  path               = "flux"
}

Now, let's install Flux!

Before we do so, this is a good place to commit and push your latest changes before Flux adds files into your repo.

We're going to run apply again but this time we need to pass in our GitHub personal access token so that it has access to write to our repo.

tofu apply -var="github_token=${GITHUB_TOKEN}"

Once done you should see a new directory created in your repo on GitHub containing the flux files - flux/flux-system. Let's pull these down into our local copy before we continue - git pull.

Confirming Our Cluster Is Setup

Now that we've got our Civo infrastructure created and Flux setup and installed I think it might be a good time to take a look in our cluster and confirm everything looks as we expect.

We can extract our cluster's KubeConfig by grabbing it from our outputs:

tofu output -raw kubernetes_kubeconfig > ~/.kube/civo-example.yaml
export KUBECONFIG=~/.kube/civo-example.yaml

With our new KubeConfig set, lets take a little look at our cluster, feel free to skip over this section if you are happy with your cluster running.

✨ kubectl get nodes
NAME                                                  STATUS   ROLES    AGE   VERSION
k3s-examplecluster-94d9-fc72b3-node-pool-910d-4znod   Ready    <none>   23m   v1.30.5+k3s1
k3s-examplecluster-94d9-fc72b3-node-pool-910d-esen4   Ready    <none>   23m   v1.30.5+k3s1

✨ kubectl get namespaces
NAME              STATUS   AGE
default           Active   24m
flux-system       Active   6m29s
kube-node-lease   Active   24m
kube-public       Active   24m
kube-system       Active   24m

✨ kubectl get gitrepo --all-namespaces
NAMESPACE     NAME          URL                                                                             AGE     READY   STATUS
flux-system   flux-system   https://github.com/AverageMarcus/bootstrap-civo-cluster-blog-post-example.git   6m57s   True    stored artifact for revision 'main@sha1:d1f3a4272b26ebc97b2698dd59dbbc4f462485fa'

✨ kubectl get kustomizations --all-namespaces
NAMESPACE     NAME          AGE     READY   STATUS
flux-system   flux-system   6m46s   True    Applied revision: main@sha1:d1f3a4272b26ebc97b2698dd59dbbc4f462485fa

Installing Workloads

What's the point of having a cluster setup if we don't install anything on it? Finally, we're going to define some workloads in our git repo and have Flux automatically pick them up and install them into our cluster for us.

First thing we're going to do is define a new Kustomization that will point to our apps directory in our cluster. Create a new file at flux/apps.yaml with the following contents:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 5m0s
  path: ./apps
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

This will create a new Kustomization in our cluster that will use the already-existing flux-system GitRepository that is pointing to our GitHub repo.

Before we commit that, lets first add an application into our /apps directory for it to pick up. For our example, we're going to install ingress-nginx as a Helm chart. Create the file apps/ingress-nginx.yaml with the following:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: nginx
  namespace: default
spec:
  url: https://kubernetes.github.io/ingress-nginx
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: nginx
  namespace: default
spec:
  interval: 5m
  targetNamespace: kube-system
  chart:
    spec:
      chart: ingress-nginx
      version: "4.11.3"
      sourceRef:
        kind: HelmRepository
        name: nginx
        namespace: default
  values: {}

This will create two new resources in our cluster - a HelmRepository and a HelmRelease. These two together will install ingress-nginx into our cluster with the version 4.11.3 and the default values for the chart. If you want to provide any values, you can do so now.

Commit those changes and push them up to GitHub.

It may take a few minutes but eventually Flux will pick up the new resources and start processing them. You can watch for the changes with the following:

kubectl get kustomizations --all-namespaces --watch

Once the Kustomization is there and showing as ready we can then check for ingress-nginx being installed:

✨ kubectl get pods --namespace kube-system
NAME                                                          READY   STATUS    RESTARTS        AGE
civo-ccm-7967db4cfc-md7jg                                     1/1     Running   2 (6m49s ago)   38m
civo-csi-controller-0                                         4/4     Running   0               38m
civo-csi-node-b29h9                                           2/2     Running   0               38m
civo-csi-node-h9b8p                                           2/2     Running   0               38m
coredns-7b98449c4-t5zz4                                       1/1     Running   0               38m
kube-system-nginx-ingress-nginx-controller-7ff945b767-tc5fz   1/1     Running   0               72s

🎉

There's our app, managed by GitOps with Flux, running in our Civo cluster created as Infrastructure as Code with OpenTofu!

Wrapping Up

We've now got a cluster setup that we can easily install and manage workloads via git. From here you can add more applications under the apps directory and they will be automatically picked up and processed by Flux.

Want to take things further?

• How about setting up Renovate to handle automatic version upgrades?
• Maybe you need more infra for your workloads - how about a Database or an Object Store?
• What about using Remote State and leveraging GitHub Actions to automatically apply changes in the infra directory?

These are out of scope for this post but if I get the time they may feature in a future post. 😉

Hopefully this is helpful for someone! I'd love to hear how you get on and any feedback you have, let me know on Mastodon at @Marcus@k8s.social or Bluesky at @averagemarcus.bsky.social!

2024-06-24: Added links to video channels DevOpsToolkit, You Choose and Enlightning. Added link to KubeHuddle conference.

Recently, a friend of mine asked me what resources I'd recommend to start learning about Kubernetes. He was a victim of the layoffs that seem to be so prevalent right now and has experience as a classic SysOps / SysAdmin engineer but no expose to Kubernetes yet and wanted to learn to help improve his job-hunting prospects.

I wasn't sure what to recommend at first, it's been a long time since I was learning Kubernetes for the first time and wasn't sure what was still useful and relevant but what follows is what I ended up sharing with him, and now with all of you.

If you have suggestions for more resources to include here please reach out to me on Mastodon at @Marcus@k8s.social!

(I have also previously wrote about my recommended Go resources which also might be of interest.)

📚 Books

• The Kubernetes Book by Nigel Poulton
• Docker Deep Dive by Nigel Poulton (if you also need to get up to speed with Docker / Containers)
• Quick Start Kubernetes by Nigel Poulton
• Understanding Kubernetes in a visual way by Aurélie Vache
• The Illustrated Children’s Guide to Kubernetes - don't let its child-focussed format fool you, this is a great (free) book! And theres a whole series of books on related topics available.

The following might not be quite beginner friendly and should be picked up after you understand the basics:

• Kubernetes Best Practices by Brendan Burns, Eddie Villalba, Dave Strebel & Lachlan Evenson
• Kubernetes Patterns by by Bilgin Ibryam, Roland Huβ Ph.d.

🧰 Services, Tools and Libraries

• Civo - you can get $250 worth of credit when you sign up to play with their managed Kubernetes offerings.
• Kind - local testing / experimenting with Kubernetes running inside a container.
• Talos - My favourite OS for running Kubernetes on. This is what powers my homelab cluster.
• k3s - a great, lightweight Kubernetes distrobution that you can even run on a Raspberry Pi!
• k9s - an interactive terminal tool for working with a Kubernetes cluster

🔗 Websites / Blog Posts / Tutorials

• Official Kubernetes tutorials
• Civo Learn - Tutorials
• Civo Academy - A series of workshops to help beginners learn Kubernetes
• Kube Academy from VMWare has a lot of tutorials to follow
• Pluralsight - if you have a subscription there's several good classes on here related to Kubernetes. I can recommend the ones by Nigel Poulton.
• I've previously written a blog post, Managing Kubernetes without losing your cool, that has some tools and tips I recommend for everyone working with Kubernetes.

📺 Videos

• Rawkode Academy - David covers more than just Kubernetes and has a lot of useful videos about all things cloud native (and more)!
• Kunal Kushwaha - Kunal is a powerhouse when it comes to making super helpful videos. There a lot of stuff here, not just Kubernetes.
• Anaïs Urlichs - Lots of great, beginner friendly video about Kubernetes and related tools. Specifically the Full Kubernetes tutorial on Docker, KinD, kubectl, Helm, Prometheus, Grafana which might be helpful.
• CNCF - Collections of conference talks from previous KubeCon, Kubernetes Community Days and other conferences.
• DevOpsToolkit - Viktor is great at introducing complex topics in an approachable way
• You Choose - a series of videos going through lots of the CNCF Landscape tools and pitting them against each other.
• Enlightning - Whitney does a great job of going through lots of CNCF technologies with the help of her lightboard.

📽️ Conferences

• KubeCon + CloudnativeCon is great for meeting people in the community if you have a chance to attend. There are events available in North America, Europe, China and India
• I'm a huge fan of Kubernetes Community Days! These are smaller, local Kubernetes conferences hosted all over the world.
• Cloud Native Rejekts is recent favourite of mine and is set up as the "b-side" conference to KubeCon. This great, community focussed conference runs for 2 days before the main KubeCon conference.
• KubeHuddle - a wonderful community focussed conference

📰 Newsletters

• KubeWeekly from CNCF

Hopefully these are helpful for someone! I'd love to hear what your favourite resources are, let me know on Mastodon at @Marcus@k8s.social!

What is it?

For those that don't know, the event is described as "Electromagnetic Field is a non-profit camping festival for those with an inquisitive mind or an interest in making things". What this basically means is it's a multi-day camping festival for nerds of all kinds! The event takes places from Thursday to Monday (if you want to stay that long) in a large deer park in Eastnor in a lovely part of the UK. Throughout the long weekend you will be able to attend talks and workshops on a large range of diverse and interesting topics, dance and party the night away at a couple different music areas, play some fun and geeky games around the site and marvel at the creativity and imagination of the other attendees and their installations.

The event is 100% run by volunteers and a lot of love and effort goes into making it successful for everyone. All attendees are encouraged to volunteer for a shift helping out at various things around the site from bar shifts to infrastructure support.

The run-up

As this was my first time I was quite anxious in the run up to it as I'm not really one for camping or for festivals but I was very excited that several people I knew were also going, including several old friends from my JSOxford days! 💙

I was also helping Sam Machin with some of the phone infrastructure for the event by setting up and configuring a Kubernetes cluster that ran on-site and handled almost all of the applications that allowed the various telephone networks to work and be built on top of. As I'm sure you can imagine, I really enjoyed helping out with this and already have a bunch of ideas on how to make it better and more reliable for the next event! We did have to deal with a couple bugs during the weekend but for the most part it went well! 😅

On-site

The main thing you'll notice while walking around the site is that there is no shortage of interesting things to look at or interact with. Some of the things are organised by the event and others are just cool things that people have brought along with them for the enjoyment of others.

Some highlights include...

... plus many more cool, interesting and striaght up weird and wonderful things.

One thing EMF Camp has that you likely won't see at many other festivals is a very impressive amount of network infrastructure that serves the whole site. Almost the entire site is covered in WiFi backed by a 40Gb fibre connection to the internet (into the middle of a field!!! 🤯) that is useable by all attendees. The site is also covered in an impressive network of phone systems operated by the POC (Phones Operations Centre) that includes DECT coverage on the whole site, the ability to use SIP clients, a plain old telephone service and a 2G GSM network. It was also possible for attendees to build apps using the phone network that other attendees can then use over the weekend. Along with all of this was a fantastic network of electric hookup points across the site for all to use that ended up leading to my most millennial camping experience..

While talking about food it's worth noting that the food and drink options on site were amazing. There was several fantastic food stalls open most of the time and there was two bars available to get drinks from.

While I didn't get to see as many talk as I might have liked (the seats weren't very comfortable) I did manage to get to see some of my wonderful friends speak!

I also managed to catch a showing of "An Evening Of Unnecessary Detail" which I very much enjoyed.

Things I think could be improved

On the whole everything I experienced was great! There's only some very minor annoyances that I'd like to see improved next time around.

While the toilets on site were quite nice (besides the fetilizer ones that grossed me out) I found them quite tight to squeeze into. Not a big deal compared to camping I've done in the past though.

There was a "night market" on site where some vendors were set up selling some funky, geeky crafts. I loved this idea but the room they were in was so small and crowded that I couldn't handle being in there for very long at all. It was a shame as I'd have liked to have chatted with some of the vendors while I was there.

As a non-(alcoholic)-drinker I would have liked to have seen more non-alcoholic options available in the bars. The selection they had was perfectly fine, it just would have been nice to have some 0% beers or similar.

The programmable badges had issues. This was kinda to be expected as they were brand new and very ambitious but it was still frustrating not really being able to use them right away. I'm looking forward to re-using mine in 2026 when I'm sure the software will be rock solid!

Not something for the event itself to improve on but something for me next time - I'd like to take part more in the various activities around the site. I was taking things steady this year as I didn't want to get overwhelemd by everything but I do now wish I took part in the Clippy Murder Mystery or the capture the flag style game with the badges. There's so much going on all over the site that you're bound to miss out on something you wish you'd managed to do.

Summary

If it wasn't obvious by now I had a fantastic weekened at EMF Camp and will certainly be attending the next one in 2026 if possible. I hope to see even more of my friends (new and old) there so we can hang out and nerd out on all the cool experiences that EMF Camp has on offer.

Before we dig in to the Renovate configuration, lets take a look at what exactly we're trying to achieve...

The Goal

All of the infrastructure for k8s.social is managed in git and available on GitHub. The Civo Kubernetes cluster is managed using Pulumi, using Go, and applied from a GitHub Action. The Kubernetes version is included in the main.go file:

k8SCluster, err := civo.NewKubernetesCluster(ctx, "k8sCluster", &civo.KubernetesClusterArgs{
    Name:       pulumi.String("k8s.social"),
    KubernetesVersion: pulumi.String("1.27.1-k3s1"),
})
if err != nil {
    return err
}

(Lines removed for clarity)

So as you can see above, we're currently using the version 1.27.1-k3s1 but I'm aware that 1.28.2-k3s1 is now available to use. Rather than me having to update that value myself each time a new release is published (and to prevent me forgetting) lets have Renovate update that string for us automatically when it detects a new version.

Getting the versions

The first thing we need, before we can even start on the Renovate configuration, is some way of getting a list of available versions. As Renovate supports using a JSON file from an URL (spoiler!) it would be best if we have an API we could call to get the versions.

As it turns out, Civo does have an API you can call to get a list of versions. Unfortunately, this requires you to be authenticated when calling it and I'd rather not provide Renovate with my API token just to get a list of available versions.

Instead I put together a very quick API app, civo-versions, that calls the Civo API with authentication and returns the versions as a JSON object. If y'all want to make use of it yourself it is available at https://civo-versions.cluster.fun/. (Different endpoints are available for different filtering, take a look at the README.md for more info).

Creating the custom datasource in Renovate

ℹ️ Note: As I write this, Renovate is currently in the process of renaming / migrating the regexManagers to be customManagers (see PR). This made the documentation currently quite confusing and hard to follow as things aren't quite completed yet. For the purpose of this post I shall be using the old regexManagers which should still work for a while as there is a migration step built in to Renvoate to convert it to the new customManagers.

Now that we have somewhere that lists the available versions we can configure renovate!

For this to work we'll be making use of two properties in the Renovate configuration: customDatasources and regexManagers.

{
    "customDatasources": {

    },
    "regexManagers": [

    ]
}

According to the documentation the resulting JSON must match the following format (with the version being the only required field):

{
  "releases": [
    {
      "version": "v1.0.0",
      "isDeprecated": true,
      "releaseTimestamp": "2022-12-24T18:21Z",
      "changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md#v0710",
      "sourceUrl": "https://github.com/demo-org/demo",
      "sourceDirectory": "monorepo/folder"
    }
  ],
  "sourceUrl": "https://github.com/demo-org/demo",
  "sourceDirectory": "monorepo/folder",
  "changelogUrl": "https://github.com/demo-org/demo/blob/main/CHANGELOG.md",
  "homepage": "https://demo.org"
}

While Renovate does have capabilities to perform transformations on the JSON is retrieves I decided to make things easier and have civo-versions return the JSON in a compatible format to start with. Because of this, our customDatasources will look quite sparse:

{
  "customDatasources": {
    "civo-k3s": {
      "defaultRegistryUrlTemplate": "https://civo-versions.cluster.fun/k3s/"
    }
  }
}

Here we're giving our custom datasource the name of civo-k3s and the defaultRegistryUrlTemplate points to the URL where we can get a list of the current k3s versions available from Civo. If our returned JSON wasn't already in the required format we could use transformTemplates to perform some manipulation of the data - see the documentation for more details if needed.

Now that we have our new custom datasource we can make use of it just like we would with the built in datasources.

We'll be using the regex manager to parse main.go and extract the line that contains our Kubernetes version string, then we'll then use our new custom datasource to replace the version if a newer one is found.

{
  "customDatasources": {
    "civo-k3s": {
      "defaultRegistryUrlTemplate": "https://civo-versions.cluster.fun/k3s/"
    }
  },
  "regexManagers": [
    {
      "fileMatch": ["main.go"],
      "matchStrings": ["KubernetesVersion: pulumi.String\\(\"(?<currentValue>\\S+)-k3s1\"\\)"],
      "datasourceTemplate": "custom.civo-k3s",
      "depNameTemplate": "civo-k3s",
      "extractVersionTemplate": "^(?<version>.*)-k3s1$"
    }
  ]
}

In the above you can see the following:

• fileMatch tells Renovate which files we want to try and replace versions within.
• matchStrings is a list of regex strings to match on. The (?<currentValue>\\S+) capture group is used to extract the current version number that Renovate will replace.
• datasourceTemplate is the name of the datasource we want to use for this manager. For custom datasources it is the name we defined prefixed with custom..
• depNameTemplate is the name to use for the dependancy. This will be used in the PR title / body for example.
• extractVersionTemplate is a regex string to perform on the new version.

Putting it all together

With all this in place and committed into git we should now have Renoate creating PRs to bump our Kubernetes version. 🎉

Bonus

To make the use of this Civo version datasource easier for others I've added it to my renovate-config repo that contains reusable Renovate configuration presets.

My Civo preset contains multiple datasources covering both k3s and Talos cluster types as well as limiting the versions just to stable releases. To make use of the presets, add the following to your Renovate configs extends property:

github>averagemarcus/renovate-config:civo

Resources

• Renovate custom datasource documentation
• Renovate customManager documentation
• Civo API documentation
• My personal Renovate presets
• civo-versions

Not good!

To improve on this situation I decided to switch to making use of Redis Sentinel to take advantage of its failover capabilities. I wanted to keep the same configuration for everything else so I was sure it would behave the same with Mastodon. This mainly meant I wanted to keep a single endpoint that Mastodon connected to Redis with and that Redis was configured with persistence in Append Only File (AOF) mode.

In most situations where I’ve used Redis it’s been as a caching layer so to switch to a new setup all I would need to do is point my app at the new endpoint. Mastodon, however, makes use of persistence in Redis and uses it as a data store for various things including users home feeds. So I needed to migrate across the existing data.

Migration Steps

After hours of trial and error I finally got a set of steps to follow that allowed me to migrate the data safely with only a small windows of time where I missed out on new updates from the live server. This was acceptable for me in this instance but might not always be suitable so please keep that in mind.

Before you plan to follow these steps yourself, please make sure you also take a look at the Gotchas below!

Assumptions

• You are using the Redis Helm chart by Bitnami
• You’re Redis instance is running in a Kubernetes cluster and it has enough resources available to run another instance alongside
• You have Redis configured in master/replica mode with Append Only File persistence.
• Your current Redis instance is deployed into the redis namespace.
• Your data is in the Redis database 1

Not all of these are required to follow these steps (I wasn’t actually using the Helm charts) but it helps to establish some common groundwork.

Steps

1. Setup Redis Sentinel alongside old setup (ideally in a new namespace) with the following configuration changes:

1. `appendonly` config set to `no` (we’ll change this later)
2. Replicas set to `1`

2. Once new PVC has been created and Redis has started scale down Redis Sentinel’s StatefulSet replicas to 0
3. On old Redis master, snapshot the current state:

1. `kubectl exec -n redis -it redis-master-0 -- bash`
2. `redis-cli -a ${REDIS_PASSWORD} -n 1 SAVE`

4. On your local machine copy backup: kubectl cp redis/redis-master-0:/data/dump.rdb ./dump.rdb

5. Launch a temporary pod that mounts the new PVC to upload dump into:

   kubectl run --namespace redis-sentinel -i --rm --tty temp --overrides='
     {
         "apiVersion": "v1",
         "kind": "Pod",
         "metadata": {
             "name": "temp"
         },
         "spec": {
             "containers": [{
                "command": [
                     "tail",
                     "-f",
                     "/dev/null"
                ],
                "image": "bitnami/minideb",
                "name": "mycontainer",
                "volumeMounts": [{
                    "mountPath": "/mnt",
                    "name": "redisdata"
                 }]
             }],
             "restartPolicy": "Never",
             "volumes": [{
                 "name": "redisdata",
                 "persistentVolumeClaim": {
                     "claimName": "redis-data-redis-node-0"
                 }
             }]
         }
     }' --image="bitnami/minideb"
   

6. On your local machine upload backup to the temporary pod:

1. `kubectl cp ./dump.rdb redis-sentinel/temp:/mnt/dump.rdb`

7. Delete the temporary pod (kubectl delete pod -n redis-sentinel temp) and restore the Redis Sentinel’s StatefulSet to 1 replica
8. Once the new Redis Sentinel is running, instruct it to switch to appendonly:

1. `kubectl exec -n redis-sentinel -it redis-node-0 -- bash`
2. `redis-cli -a $REDIS_PASSWORD -n 1 bgrewriteaof`

(👆 This step seemed to be missing from all the backup/restore guides I found for Redis and was required to continue making use of the Append Only File.)

9. Update the Redis Sentinel’s config to set appendonly back to yes
10. Restart the Redis Sentinel pod
11. Once Redis is up and running scale the StatefulSet back up to 3 (or however many replicas you plan to run)
12. Deploy redis-sentinel-proxy to act as a single entrypoint to Redis that points to the current leader. (You can take a look at my configuration if you’re not sure how it needs to be set up)
13. Update your applications to point to the new redis-sentinel-proxy Service.

Gotchas

CoreDNS Changes

Redis Sentinel makes use of a headless Kubernetes service to discover and connect to the Redis replicas. For this to work correctly, CoreDNS must be configured with endpoint_pod_names within the kubernetes block (see the CoreDNS documentation for more info). This allows pods to be resolved by hostnames such as redis-node-0.Redis-headless.redis.svc instead of 1-2-3-4.redis-headless.redis.svc.

If the endpoint_pod_names properly already exists in your CoreDNS config you don’t need to do anything but it wasn’t default in my cluster.

Example:

kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    endpoint_pod_names
    fallthrough in-addr.arpa ip6.arpa
}

Re-enabling appendonly

It turns out that simply re-enabling appendonly yes and performing a server restart will lead to all your previous data (in the dump.rdb) being ignored and Redis starting fresh with the Append Only File mode.

To get around this issue you must first inform Redis that you are enabling appendonly by performing redis-cli -a $REDIS_PASSWORD -n 1 bgrewriteaof.

This wasn’t mentioned in any of the backup / restore instructions I found for Redis until I came across “Migrating to AOF from RDB with Redis” by Laurent. I went through two attempts where I lost the data before I realised this was needed so I’m very thankful for discovering this post!

No write endpoint

it turns out that Redis Sentinel, and more specifically the Bitnami Hem chart, doesn’t have a service specifically for the leader instance. This means if you attempt to use the redis.redis-sentinel.svc Service for both reads and writes you’ll eventually receive an error telling you that Redis is in read only mode as the service load balances across all the pods, not just the leader. Instead you’re expected to have your app query Sentinel to request the endpoint of the current leader. This was a problem for me as I couldn’t make changes to Mastodon to support this. Instead I deployed redis-sentinel-proxy that acts as a small proxy to the current write-capable pod and updates as it fails over to a new pod.

There is now a slight concern that this has become the single point of failure. The proxy is fairly lightweight and launches quickly so I’m pretty confident that if errors do occur a replacement will be available quickly. I’ve made sure it is running with at least 2 replicas with a pod anti-affinity to hopefully avoid node failures taking it down.

Migration window

I didn’t set up any replication from the old Redis to the new, instead I was making use of a snapshot from the old and restoring it into the new. Because of this there was a small window of time where new data was being created in the old Redis that wouldn’t be available in the new as I was in the process of switching over. For my purposes this was acceptable and I scheduled the move during a period of time I know saw less activity but I know this approach wont be suitable for all situations.

To minimize this window I worked with only a single Redis Sentinel replica during the migration and I set sentinel.getMasterTimeout to a very small value to ensure that first pod came up quickly so I could move traffic over. Just remember to put that timeout back up once you’re done!

The scripts in the Helm chart didn’t work correctly

It’s entirely possible I did something wrong here but the prestop scripts within the Bitnami chart didn’t work correctly and weren’t actually triggering the failover on shutdown. I only discovered this while I was testing out redis-sentinel-proxy and noticed there was a couple minutes where it was still attempting to connect to the old leader that had shutdown.

From what I could tell, the main thing missing was the prestop script for the Sentinel container didn’t trigger the failover command. In the end I decided not to use the Helm chart but instead use it to generate manifests to use directly. You can see the scripts I use on GitHub.

Summary

In the end, after a few hours of trial and error, the actual migration process was fairly smooth. I think I had a migration window of about a minute or less where new data was lost.

I was pretty annoyed that none of the backup / restore guides I came across actually handled the AOF issues. They all stated that it needed disabling at the start to generate the dump.rdb but there was then no mention about it for the resulting Redis instance upon restoring. Hopefully this post will help someone else avoid that issue in the future.

I found the Bitnami Helm chart to be a pain to work with. It has so many different configurations it was hard to know for sure what the result would be, plus the above issue about the scripts not working. I feel better having used it as a base and generating the yaml manifests to then further work with.

My only outstanding concern now is the redis-sentinel-proxy being a single point of failure. It’s only a small concern though, it has been running smoothly for the past few days now and it’s configured to be resilient enough to handle failures but I’ll be keeping an eye on this going forward.

If you’ve found this useful, or found issues with it, I’d love to hear! You can find me on Mastodon at @marcus@k8s.social.

Resources & References

• https://docs.bitnami.com/kubernetes/infrastructure/redis/administration/backup-restore/
• https://github.com/bitnami/charts/tree/main/bitnami/redis
• https://hub.docker.com/r/flant/redis-sentinel-proxy / https://github.com/flant/redis-sentinel-proxy
• https://pellegrino.link/2020/06/10/2020-redis-persistence-migrating-rdb-to-aof.htm

2022-07-15: Added tweet from Ian Coldwater with nsenter example

This post is based on a webinar i've previously given where I go through some of my favourite tips for working with Kubernetes clusters all day long. The goal of all of these techniques is to make my life easier and (hopefully) less error prone. I start off with the first 5 tips being applicable to anyone working with Kubernetes and can be picked up right away. From there I move on to a couple that would benefit from having some old-skool Linux sys-admin experience. Finally I finish of with some more advanced techniques that require some previous programming experience.

#0 - Pay someone else to do it

Yeah, ok, this one is a little tongue-in-cheek as this is what we provide at Giant Swarm but it is worth thinking about.

If you have dozens or even hundreds of clusters, on-top of standard development work, you’re going to be stretched thin. Getting someone else to manage things while you focus on what makes your business money can often be the right choice.

#1 - Love your Terminal

It doesn’t matter if your shell of choice is Bash, ZSH, Fish or even Powershell you’ll benefit greatly from learning the ins-and-outs of your terminal of choice.

Many shell implementations have some variation of “rc” files - often referred to as “dotfiles”. In bash, it’s a file called .bashrc, in ZSH it’s .zshrc. Generally these are found in your users home directory. These files are interpreted by your shell each time a new session (tab/window) is opened and allows you to set up defaults and customisations. One of my favourite uses for these session files is to create custom aliases for functions (or groups of functions) that I use often to help me (try to) avoid typos and “fat fingering” when typing out long commands.

Many people share their custom “dotfiles” on GitHub or similar and are a great source of neat tricks and handy shortcuts. My dotsfiles can be found on GitHub at AverageMarcus/dotfiles.

#2 - Learn to Love kubectl

Everything you might want to do with a Kubernetes cluster can be acomplished with the help of kubectl. Learning how to master this handy little CLI can go a long way to mastering Kubernetes cluster management.

The official documentation offers a single page view of all built in commands that is a great place to start if you’re not sure of a command: kubectl-commands.

Building on tip #1, my most used command is k, an alias I have set up for kubectl to save on keystrokes. Add the following to your relevant ‘rc’ file:

alias k='kubectl'

Once this is in your terminal session, you can do something like the following:

k get pods

(You’ll see more of this alias in the other examples used in this post)

kubectl even has a very handy command to find out everything you need to know about the properties of any Kubernetes resource - kubectl explain

# k explain pods.spec.containers
KIND: Pod
VERSION: v1

RESOURCE: containers <[]Object>

DESCRIPTION:

I find this most useful when you can’t access the official docs or your know exactly the field you’re interested in.

#3 - Multiple Kubeconfigs

When dealing with multiple Kubernetes clusters your kubeconfig file can very quickly become an unwieldy mess.

Thankfully there are some tools available to make it more manageable by making it quick and easy to switch between cluster context.

• kubectx and kubens - https://github.com/ahmetb/kubectx
• kubeswitch - https://github.com/danielfoehrKn/kubeswitch
• kubie - https://github.com/sbstp/kubie

kubeswitch is my current favourite as it supports loading kubeconfigs from a directory of yaml files, allowing me to export the kubeconfig of each cluster to their own file. I had previously had this same multi-file setup with just kubectl by combining multiple files in the KUBECONFIG environment variable with a : (see below) but I found that many tools didn’t support this and attempted to treat the value as a single file path.

Multi-file kubeconfig hack, expecting files to be in the ~/.kube/clusters directory:

## Merge multiple kubeconfigs
function join_by { local d=$1; shift; echo -n "$1"; shift; printf "%s" "${@/#/$d}"; }
KUBECONFIG="$HOME/.kube/config"
OTHER_CLUSTERS="$(join_by :$HOME/.kube/clusters/ $(echo $HOME/.kube/clusters/$(/bin/ls $HOME/.kube/clusters)))"
export KUBECONFIG=$KUBECONFIG:$OTHER_CLUSTERS

I recommend using one of the tools listed above rather than this shell hack.

#4 - k9s

k9s is one of my favourite tools when working with clusters. It provides an interactive terminal that supports all resource types and actions, lots of keybinding and filtering. Once you learn all the shortcuts and features, this handy tool will save you a lot of time when working with problematic clusters.

#5 - kubectl plugins

Just like Kubernetes itself, kubectl can be extended to provide more powerful functionality via a plugin mechanism, there’s even a package manager available to help discover plugins - Krew.

But you don’t need a package manager, or really anything too fancy, to add plugins to kubectl. Any executable in your $PATH that is prefixed with kubectl- becomed a kubectl plugin.

For example, assume we wanted to make a simple “hello, world” style plugin, we just need to create an executable somewhere in our path and name it something like kubectl-hello.sh and then we automatically have the kubectl hello command available to us.

# kubectl-hello.sh
echo "Hello from kubectl"

---

> kubectl hello
"Hello from kubectl"

This example uses a shell script but your plugins can be built in any language you like as long as it’s executable and has the needed prefix.

One thing to note with plugins though - if you use tabcompletion / autocomplete in your shell with kubectl it generally wont work with the plugin commands.

If you want to get started with some kubectl plugins you can take a look at the Krew plugin directory to see if a plugin already exists for your needed task. Some of my favourite plugins are:

• stern - Multi-pod/container log tailing
• tree - Show hierarchy of resources based on ownerReferences
• outdated - Find containers with outdated images
• gs - Giant Swarm’s plugin for working with our managed clusters

#6 - kshell / kubectl debug

There are many times where I need to launch a temporary pod in a cluster to aid with debugging, usually related to networking issues. To help with this I have a handy Bash alias to easily launch a temporary pod and drop me right into its shell.

alias kshell='kubectl run -it --image bash --restart Never --rm shell'

Now I can run kshell to get a new debugging pod running the bash container image. Need more tools then available in the bash image? Replace bash with something like ubuntu.

# kshell
If you don't see a command prompt, try pressing enter.
bash-5.1# nslookup google.com
Server: 1.1.1.1
Address: 1.1.1.1:53

Non-authoritative answer:
Name: google.com
Address: 142.250.187.206

This is great for more general debugging of a cluster, especially with networking issues such as DNS resolution or network policies.

This is great for more general debugging but what about debugging a specific pod? Well we’ve got a couple tools we can use depending on the situation - kubectl exec or kubectl debug.

kubectl exec lets up drop into the shell of a running container, just like we do with kshell above, but this isn’t always possible.

# kubectl exec my-broken-pod -it -- sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec……

The container needs to have a shell we can drop into which isn’t always the case and we’ll be presented with something similar to the error above.

Similarily, if our pod is CrashLooping we’ll only be able to have access to the shell until the pod is restarted again, at which point we’ll be kicked out back to our local shell session.

Thankfully, if running Kubernetes 1.23 or later we can make use of kubectl debug to potentially investigate these problem pods.

# kubectl debug -it --image bash my-broken-pod
Defaulting debug container name to debugger-gprmk. If you don't see a command prompt, try pressing enter.
bash-5.1#

This injects an Ephemeral Container into our chosen pod that can then run alongside our broken container.

kubectl debug has a few different modes:

• The default launches an “ephemeral container” within the pod you’re debugging

  kubectl debug
  

• The copy-to flag creates a copy of the pod with some values replaced (e.g. the image used)

  kubectl debug –copy-to
  

• Specifying the node launches a pod in the nodes host namespace to debug the host machine

  kubectl debug node/my-node
  

There are some limitations though, for example kubectl debug can’t access the whole filesystem of our failing pod, it’ll only be able to access mounted volumes if specified.

When to use what:

#7 - kube-ssh

Sometimes investigating the cluster isn’t enough to find the cause of a problem. Sometimes the host machines hold the answer. If we have ssh access to our machines, this is a simple task, but that’s not always the case. It’s a common practice to disable ssh and similar services on a cluster and treat all machines as immutable and replaceable. This is great from a security perspective but not so much when trying to debug a problem that seems limited to a single (or subset) of nodes.

If our cluster is running Kubernetes 1.23 or later then we can make use of the kubectl debug command as described above with the node we want to debug.

But if we aren’t yet running 1.23 there are some alternatives.

I have a project called kube-ssh that gives “ssh-like” access to a nodes underlying host machine and Giant Swarm has a similar kubectl plugin, kubectl-enter.

kube-ssh uses a few of the tips already suggested in this post to launch a new pod in the cluster with elevated permissions and makes use of nsenter to switch to the hosts Linux namespace.

sh -c "$(curl -sSL https://raw.githubusercontent.com/AverageMarcus/kube-ssh/master/ssh.sh)"
[0] - ip-10-18-21-146.eu-west-1.compute.internal
[1] - ip-10-18-21-234.eu-west-1.compute.internal
[2] - ip-10-18-21-96.eu-west-1.compute.internal
Which node would you like to connect to? 1

If you don't see a command prompt, try pressing enter.
[root@ip-10-18-21-234 ~]#

(Please don’t blindly execute a script from the internet like that, examine it first at least)

There are some caveats though:

• The underlying host needs a shell just like we mentioned above with kubectl exec. (So this wont work if you’re running Talos for example)
• You require enough permissions to launch pods with privileged securityContext - RBAC, PSPs and Admission Controllers could all potentially block this. (This could also be considered a benefit to this approach over traditional SSH)
• Not a real SSH session

Shortly after I posted this blog post, Ian Coldwater tweeted out a version of this technique that's small enough to fit in a tweet! 🤯

#8 - Admission Webhooks

Kubernetes has two types of dynamic admission webhooks (there are also CRD conversion webhooks but we’re not talking about those here):

• ValidatingWebhook - Ability to block actions against the API server if fails to meet given criteria.
• MutatingWebhook - Modify requests before passing them on to the API server.

These webhooks allow you to implement more advanced access control that is possible with RBAC, PSPs, etc. as well as the ability to modify requests to the Kubernetes API on-the-fly. With this you can take more control over your clusters in an automated way, for example you could:

• Add default labels to resources as they’re created.
• Prevent the latest tag being used on images.
• Enforce all pods have resource requests/limits specified.
• “Hotfix” for security issues (e.g. mutating all pods to include a LOG4J_FORMAT_MSG_NO_LOOKUPS env var to prevent Log4Shell exploit).

With these webhooks you’re also able to do subtractive access control - take away a users ability to perform a certain action - something that isn’t possible with RBAC which is only addative. I have actually wrote about this previously where we needed to remove a specific permission from the cluster-admin role: Restricting Cluster Admin Permissions

There’s two general ways to go about implementing webhooks in your cluster, either you build something bespoke yourself with custom logic or you leverage a third-party tool that abstracts away from of the low-level details. I recommend starting with a third-party solution to start and if that doesn’t fulfil your needs then look at a custom implementation.

The two current most popular options are Kyverno and OPA Gatekeeper.

Example Kyverno policy taken from my Restricting cluster-admin permissions blog post:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-bulk-certconfigs-delete
  annotations:
    policies.kyverno.io/description: Block delete all bug in CLI
spec:
  rules:
  - name: block-bulk-certconfigs-delete
    match:
      any:
      - resources:
          kinds: [CertConfig]
    preconditions:
      any:
      - key: ""
        operator: Equals
        value: ""
    validate:
      message: |
        Your current kubectl-gs version contains a critical bug
      deny:
        conditions:
        - key: ""
          operator: In
          value: [DELETE]

There are some things to be aware of though. It’s completely possible (and frustratingly common) to take down a cluster with incorrectly configured webhooks.

Where possible, always try and exclude applying webhooks to the kube-system namespace to ensure critical components aren’t accidentally blocked by a webhook.

For example, i’ve dealt with cases where a validating webhook applied to all pods, in all namespaces, and during a cluster upgrade the pods that provided the logic for the webhooks weren’t yet running when the apiserver pod was being created. The webhooks pod was also blocked from launching because the webhook couldn’t run against the pod launch. The launching of this pod being blocked cascaded to more and more failures in the cluster until eventually manual intervention was needed.

On a similar note, be aware of the FailurePolicy property on the webhook resources. By default this is set to fail which will prevent whatever action is being targeted if the webhooks service is unavailable.

There is also a reinvocationPolicy property that can be set if changes made by a MutatingWebhook need to trigger a previously executed webhook to re-run with the latest changes. Use this sparingly as you don’t want to be continuously running webhooks all the time.

The last thing to be aware of - the order webhooks are executed isn’t guaranteed. The MutatingWebhooks are run before the ValidatingWebhooks but within those two phases there’s nothing to ensure a specific running order. In practice, they are run in alphabetical order but that shouldn’t be counted on.

Unfortunately these recommendations are rarely followed by the various open source project that make use of them so you need to be careful about how these applications are configured in your cluster. Oh, and if you have the bright idea like I did to implement a webhook that enforces these recommendations you can stop now. Webhooks aren’t performed against other webhook resources.

#9 - Kubernetes API

All Kubernetes operations are done via the API - kubectl uses it, in-cluster controllers use it, the Kubernetes scheduler uses it and you can use it too! ✨

Many agree that the API is what makes Kubernetes so powerful and successful. It provides a solid foundation with some core resources and functionality, then let’s others extend it and build on top more advanced and powerful features.

The API can be extended by either:

• the creation of Custom Resource Definitions (CRDs) and controllers.
• implementing an Aggregation Layer (such as what metrics-server implements). - I’ll not be talking about this in this post.

It’s easy to start experimenting with the Kubernetes API, kubectl actually exposes a --raw flag that we can use to perform an action against a raw API endpoint without having to worry about handling authentication as this will be pulled from your KUBECONFIG details. For example, the following is equivalent to kubectl get pods -n default:

# kubectl get --raw /api/v1/namespaces/default/pods
{"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":...

The following table shows which kubectl commands to use for the different HTTP method verbs.

There is even a kubectl command to list out the available APIs in a given cluster in case you weren’t sure.

# kubectl api-resources
NAME                              SHORTNAMES         APIVERSION                             NAMESPACED   KIND
apiservices                                          apiregistration.k8s.io/v1              false        APIService
applications                      app,apps           argoproj.io/v1alpha1                   true         Application
applicationsets                   appset,appsets     argoproj.io/v1alpha1                   true         ApplicationSet
appprojects                       appproj,appprojs   argoproj.io/v1alpha1                   true         AppProject
bindings                                             v1                                     true         Binding
certificatesigningrequests        csr                certificates.k8s.io/v1                 false        CertificateSigningRequest
clusterinterceptors               ci                 triggers.tekton.dev/v1alpha1           false        ClusterInterceptor
clusterrolebindings                                  rbac.authorization.k8s.io/v1           false        ClusterRoleBinding
clusterroles                                         rbac.authorization.k8s.io/v1           false        ClusterRole

Given the above results, the API endpoints are formatted like so:

/{API_VERSION}/namespace/{NAMESPACE}/{RESOURCE_KIND}/{NAME}

If a resource isn’t namespaced (if its cluster-scoped), or if your wanted to perform the request across all namespaces, then you can remove the namespace/{NAMESPACE}/ from the URL.

Similarly, if you’re wanting to list resources rather than retrieve a single resource then you remove the /{NAME} from the end of the URL.

Be aware that the {API_VERSION} has a bit of a gotcha! If the APIVERSION column in the kubectl output just says v1 then this is one of the “core” resources and the API path will start with /api/v1/. But, if the resource is not one of the “core” ones, for example deployments are apps/v1, then the endpoints starts slightly differently: /apis/apps/v1. Take note that this path starts with apis (with an “s”) followed by the api version, whereas the core start with api (without and “s”).

You may be wondering why we’d want to use the API directly rather than making use of kubectl, well there’s at least a couple good reasons:

• the API allows us to work with Kubernetes in any programming language we want and without the need for additional applications.
• the kubectl tool is designed to be used by humans primarily meaning most of the output is best suited for a person to read, rather than a computer. Similarly, the commands available in kubectl don’t always map to a single API call, often there are several calls made “behind the scenes” before the result is returned to the user.

Generally its unlikely we’ll make raw HTTP requests to the Kubernetes API server and instead make use of a client library in our programming language of choice. For example:

• kubernetes/client-go - the official Golang module for interacting with the Kubernetes API
• Kubernetes Provider for Terraform (actually uses the above Go module under the hood)
• kubernetes-client organisation on GitHub has many official clients in different languages

#10 - Custom Resources and Operators

Custom Resource Definitions (CRDs) and Operators give us the ability to implement abstractions and higher-order representations of concepts. You can define your own resource types with CRDs that then can be handled just like any other resource in a Kubernetes cluster and you can implement the operator pattern to add logic to those custom (or default, built-in) resources.

This is a whole topic on its own and not something I can do justice here so I highly recommend taking a look at the Kubernetes Operators Explained blog post from Container Solutions. Below is a diagram from that blog post explaining the process of an operator.

An operator implements the standard reconciliation loop logic, as found in almost all parts of Kubernetes, to watch for changes to resources or external factors and make adjustments accordingly - either to said resources current state or some external system in response to a change to a resource.

There are several popular frameworks available to handle most of the boilerplate logic needed to create an operator:

• Kubebuilder is pretty much the most commonly used, especially if working with any of the cluster-api providers.
• Operator Framework
• Kudo
• Metacontroller

To learn more on this subject I can recommend the following YouTube videos:

• Writing a Kubernetes Operator from scratch
• Tutorial: From Zero to Operator in 90 Minutes!
• Smooth Operator: A Rough Guide to Kubernetes Operators

Wrap-up

Hopefully there has been at least something from these points that you can take away and make use of today. If you have any tips you rely on when working with Kubernetes I'd love to hear them, reach out to me on Twitter @Marcus_Noble_ or on Mastodon @marcus@k8s.social.

Generally, and by default, operators of the cluster are assigned to the cluster-admin ClusterRole. This gives the user access and permission to do all operations on all resources in the cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-admin
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

There's very good reason for this, an admin generally needs to be able to setup and manage the cluster, including the ability to define and assign roles. But what if we need to block an action performed by cluster admins? We can't do it with RBAC, it only allows for adding of permissions, not taking them away.

Well, recently at Giant Swarm we had an issue in one of our CLIs used by cluster admins that incorrectly deleted more resources than intended. This was causing issues authenticating with the cluster and we needed to get a fix in place fast. The problem was we had no control over when people would update their CLI even once we'd released the fix.

We couldn't hot-fix this by updating RBAC rules as we couldn't subtract the specific permission, but what we could do was leverage an admission controller to block the request to the API.

At Giant Swarm we're pretty big users of Kyverno (which deploys an admission controller) and use it for a lot of validation and defaulting of resources in all our clusters. Not only can Kyverno validate and block based on a resource's values but it can also validate the operation being performed, and by who.

With this functionality available we were able to deploy a cluster policy that would block cluster-admin (and anyone else) from performing the specific delete all action that was causing us issues.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-bulk-certconfigs-delete
  annotations:
    policies.kyverno.io/title: Block bulk certconfig deletes
    policies.kyverno.io/subject: CertConfigs
    policies.kyverno.io/description: >-
      A bug in an old kubectl-gs version causes all certconfigs to
      be deleted on performing a login, this policy blocks that
      action while still allowing the single delete.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: block-bulk-certconfigs-delete
    match:
      any:
      - resources:
          kinds:
          - CertConfig
    preconditions:
      any:
      - key: ""
        operator: Equals
        value: ""
    validate:
      message: "Your current kubectl-gs version contains a critical bug, please update to the latest version using `kubectl gs selfupdate`"
      deny:
        conditions:
        - key: ""
          operator: In
          value:
          - DELETE

There's a few things going on here so I'm going to explain each bit individually.

First up we specify the Kind of resource we want this policy to apply to, in our case that's CertConfig but it can be anything within the cluster. It's also possible to target specific groups and API versions if needed.

match:
    any:
    - resources:
        kinds:
        - CertConfig

Next up we're setting a precondition that allows us to do some filtering based on the details of the request. In this instance we're only interested in requests that don't have a name specified (this is the case when operating on a list of resources rather than a single resource) which allows us to target only those "delete all" requests.

preconditions:
    any:
    - key: ""
    operator: Equals
    value: ""

Finally we have the actual validation rule. We're specifying a deny rule that'll block requests (matching the previous match and preconditions) with the DELETE operation. We're also able to define a message that'll be returned to the client as the reason for why the admission controller rejected the request. We're using this to inform users about the bug and encouraging them to upgrade.

validate:
    message: "Your current kubectl-gs version contains a critical bug, please update to the latest version using `kubectl gs selfupdate`"
    deny:
    conditions:
    - key: ""
        operator: In
        value:
        - DELETE

With this single policy deployed to our clusters we've been able to block the bug in our CLI, even when the user performing the action has cluster-admin level permissions.

Other Use Cases

While we needed this to work around a bug in our client there are other situations where this approach could be useful.

• A policy that prevents deletion of any resource that has a do-not-delete: "true" annotation on it to prevent accidental deletion of critical resources (such as persisten volumes or secrets).
• A policy that prevents fetching the details of secrets in a specific namespace while still allowing in every other namespace (including those that may not yet exist).
• A policy that blocks deletes, updates or patches from everyone except a specific user that can be used to prevent others "cleaning up" resources that you may be trying to debug.

So, if you haven't already, I recommend taking a look at Kyverno and especially taking a look at the example Policies that they have for an idea of what can be done.

📚 Books

My favourite book for learning Go is - ✨ The Go Programming Language ✨ by Alan A. A. Donovan & Brian W. Kernighan. This was the book I used to pick Go up in the first place. If you already have previous programming experience it's quite a nice book to first read about the specifics of the go language then flip through as needed to the various topics (such as Goroutines) that you might not know about.

Some 🆓 books I've seen recommended:

• Go With The Domain by Three Dots Labs
• Go Design Patterns by Mario Castro Contreras (Free with trial)

I've also got the following books but haven't yet had a chance to read any of them so can't comment on their usefulness just yet:

• Let's Go by Alex Edwards
• Let's Go Further by Alex Edwards
• Ultimate Go Notebook by Ardan Labs

There's also Everyday Golang by Alex Ellis which I haven't had a chance to pick up yet but if it's anything like Alex's other writing it'll be a great resource!

🧰 Tools and Libraries

In terms of useful resources, tools and libraries I use a lot:

• The Go Playground is a fantastic tool for learning and experimenting.
• Fiber - My favourite web server framework. Very similar to the Express framework available for NodeJS.
• Viper - Config management (env vars, config files, cli flags).
• Cobra - Framework for building amazing CLI tools.
• Afero - Filesystem framework (very useful for tests).
• ZeroLog - Very useful logging framework. Support log levels, metadata attributes and different log output styles.
• GORM - ORM database library with official support for MySQL, PostgreSQL, SQLite and SQL Server.

🔗 Websites / Blog Posts

• Awesome Go - List of all things Go.
• Embedding Static Files in Golang - Embedding static files into a Go binary.
• Go Recipes - a collection of handy Go commands you can run to help with things like module updates.
• Multi-arch Dockerfile - Example multi-stage Dockerfile for building tiny, multi-arch container images for Go applications.
• Go Weekly Newsletter - A fantastic weekly newsletter on all things Go (RSS feed also available).
• Writing Unit Tests - Writing Go unit tests.

🧑‍💻 Suggested by others

• https://github.com/quii/learn-go-with-tests - Learn Go with TDD.
• https://gobyexample.com/ - Common program examples.
• https://changelog.com/gotime - Good podcast on the Go community.
• My friend Michael Cade has shared his Journey To Learning Golang that has some great tips included!

If anyone has any other resources they recommend I'd love to hear about them and update this list. Please feel free to reach out to me on Twitter at @Marcus_Noble_.

2021-09-01: Added note about socket bugfix PR

2021-09-01: Added troubleshooting section about port forwarding bug

2021-09-04: Added note about switching to Podman on Windows

2021-09-04: Added update about port forwarding PR

2021-09-04: Added note about M1 Mac support

2021-09-04: Added volume mount limitation

2021-09-04: Added podman-macos utility

Docker has recently announced that Docker Desktop will soon require a subscription and, based on the size of your company, may require a paid subscription. (It remains free for personal use)

There has been quite a bit of reaction to this news:

Depending on which side your opinions lie, you might be looking for alternatives. Well it just so happens that Podman posted this well-timed tweet:

So, lets give it a whirl...

Replacing Docker with Podman (on Mac)

Note: This currently doesn't work for Macs with an M1 CPU. I've come across this post in my search - Running Podman Machine on the Mac M1 - but I've not confirmed if it works or not.

1. brew install podman

2. Wait while brew downloads, builds and installs...

3. Create a virtual machine for Podman to run from:

   ✨ podman machine init
   
   Downloading VM image: fedora-coreos-34.20210821.1.1-qemu.x86_64.qcow2.xz: done
   Extracting compressed file
   
   🕙 took 2m44s
   

4. Start the virtual machine and set up the connection to Podman:

   ✨ podman machine start
   
   INFO[0000] waiting for clients...
   INFO[0000] listening tcp://0.0.0.0:7777
   INFO[0000] new connection from  to /var/folders/x_/bfc7v6kn4fs0rl9k77whs0nw0000gn/T/podman/qemu_podman-machine-default.sock
   Waiting for VM ...
   qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2]
   
   🕙 took 34s
   

5. alias docker=podman (Add this to your .bashrc (if using Bash), .zshrc (if using ZSH) or whatever the correct file for your shell is)

6. 🎉

Replacing Docker with Podman (on Windows)

I don't currently have access to a Windows machine where I can test this out but Frank sent me this on Twitter that covers the process needed for those on Windows with WLS2 - How to run Podman on Windows with WSL2.

Troubleshooting

Ok, so it's not all completely pain free, there are a few issues you might hit...

Failed to parse config

Error: failed to parse query parameter 'X-Registry-Config': "n/a": error storing credentials in temporary auth file (server: "https://index.docker.io/v1/", user: ""): key https://index.docker.io/v1/ contains http[s]:// prefix

Podman seems more strict than Docker when parsing the config file, check the ~/.docker/config.json file for the key with the https:// prefix (as mentioned in the error message) and remove it.

Sock already exists

✨ podman machine start
ERRO[0000] "/var/folders/x_/bfc7v6kn4fs0rl9k77whs0nw0000gn/T/podman/qemu_podman-machine-default.sock" already exists
panic: interface conversion: net.Conn is nil, not *net.UnixConn

This seems to happen (for me at least) when I've previously run podman machine stop. It looks like the sock file isn't correctly being removed. Doing an rm on that file mentioned in the error message will be enough to get you going again.

UPDATE: Looks like this will be fixed in an upcoming release. - PR

Volume mounts

✨ podman run --rm -it -v $(pwd):/usr/share/nginx/html:ro --publish 8000:80 docker.io/library/nginx:latest
Error: statfs /Users/marcus/web: no such file or directory

Podman machine currently has no support for mounting volumes from the host machine (your Mac) into the container on the virtual machine. Instead, it attepts to mount a directory matching what you specified from the virtual machine rather than your Mac.

This is a fairly big issue if you're looking for a smooth transition from Docker Desktop.

There's currently a fairly active issue about this limitation but as of right now there doesn't seem to be a nice workaround or solution.

Automatic published port forwarding

✨ podman run --rm -it --publish 8000:80 docker.io/library/nginx:latest &
✨ curl http://localhost:8000

curl: (7) Failed to connect to localhost port 8000: Connection refused

The current latest version of Podman (v3.3.1) has a bug where the automatic port forwarding from host to VM when publishing a port with the -p / --publish flag doesn't work.

There's currently a couple workarounds for this:

The first is passing in the --network bridge flag to the podman command, e.g.

✨ podman run --rm -it --publish 8000:80 --network bridge docker.io/library/nginx:latest

The other, more perminant option is to add rootless_networking = "cni" under the [containers] section of your ~/.config/containers/containers.conf file.

To follow the progress of this bug, please refer to the issue. UPDATE: This has now been merged and is expected to be released in v3.3.2 in the next few days or so.

short-name resolution

Error: error creating build container: short-name resolution enforced but cannot prompt without a TTY

Ok, this is the big one and the major issue you'll likely hit making the switch today from Docker to Podman. Lets dive into it in a bit more detail...

First we need to understand what a short-name is in this context. It refers to container images that don't have a full domain name prefixed. You've likely come across these quite a lot before - e.g. alpine:latest, ubuntu:12, giantswarm/pause:latest, etc.

When using Docker, these images are actually first prefixed with docker.io (or docker.io/library for those official images without a namespace) before being pulled.

Podman doesn't have this as a default. It can work in the same way as Docker but needs a bit of configuring.

It's worth briefly pausing here to explain why this behavior is different. Podman takes the secure by default attitude to configuration and installation, and this difference is a prime example of that mindset. You've likely heard in the news over the past few years about some of the supply chain hacks that have had a big impact on some companies and projects. One of the common attack vectors is tricking users into installing what they think is a legitimate package but actually contains malicious code. The use of short names for images opens up the risk of accidentally pulling the wrong image from the wrong registry.

To mitigate this risk Podman has a feature where it will prompt you asking which registry you'd like to pull the shot named image from and will then save that choice to speed things up later. (On a side note, there's a repo where the community is trying to collate a collection of some of the most widely used shortcodes - https://github.com/containers/shortnames)

So, Podman has this handy feature to help out with security so why are we seeing an error? Well, when running Podman on MacOS (or Windows) we're actually running it in a Linux VM and remotely connecting to Podman running in that machine. Because of this we don't have an interactive terminal with the underlying Podman engine so it is unable to receive our choice if it asked us which registry to use.

Fix

There's a couple of solutions for this:

1. Instead of using short names we could switch to using fully prefixed images (this includes updating any FROM commands in our Dockerfiles also).

2. The other approach is to reduce this security feature to be on-par with the experience we're used to with Docker.

As the first solution really just relies on you changing the image names you're referencing, which will depend on how you're working, I'll focus on the second solution.

With our machine created and started (as outlined above) we need to access the machine to make a small configuration change. Thankfully Podman makes this quite easy:

podman machine ssh

This will drop you into an SSH session within the virtual machine created for Podman. Once in this machine we want to make a change to the /etc/containers/registries.conf file. If we take a look at the file contents we'll see the final lines of it (at time of writing this) as followed:

# Enforcing mode for short names is default for Fedora 34 and newer
short-name-mode="enforcing"

The short-name-mode property has 3 possible values:

• enforcing: If no alias is found and more than one unqualified-search registry is set, prompt the user to select one registry to pull from. If the user cannot be prompted (i.e., stdin or stdout are not a TTY), Podman will throw an error.
• permissive: Behaves as enforcing but will not throw an error if the user cannot be prompted. Instead, Podman will try all unqualified-search registries in the given order. Note that no alias will be recorded.
• disabled: Podman will try all unqualified-search registries in the given order, and no alias will be recorded. This is pretty much the same behavior of Podman before short names were introduced.

If we want Podman to perform more like Docker we'll want to change this value to permissive:

sudo sed -i 's/short-name-mode="enforcing"/short-name-mode="permissive"/g' /etc/containers/registries.conf

There's one more property in this file that it's worth at least being aware of.

unqualified-search-registries = ["registry.fedoraproject.org", "registry.access.redhat.com", "docker.io", "quay.io"]

This property contains a list of all the registries that will be checked (in order) when looking up a short name image. Be sure the values in here are ones that you trust!

With that change made we can exit from the virtual machine and Podman should then search for any short name images using these registries from now on.

GUI Replacement

For those that like to have a graphical UI to manage / monitor their running containers Victor has released podman-macos that provides a tiny taskbar utility for Podman.

Wrap Up

I'm sure there's many more inconsistencies but so far I'm pretty impressed. I'm plan to try using Podman instead of Docker for a while and see how I get on. I'll try and update this post with anything more I find out.

If anyone wants to share their experiences with Podman please reach out to me on Twitter at @Marcus_Noble_.

2021-09-02: Added resources

I’ve recently accepted a new job offer and wanted to share a little about my experience job hunting during a global pandemic. Even after more than a year it was clear that some companies still hadn’t adjusted or taken the time to stop and look at the approach they were taking and the effect it had on the interview candidate. Hopefully some of this will be of use to someone looking to hire during similar situations.

Most of what’s to come is aimed at those doing the hiring rather than the hunting but I think it could be of benefit for both sides to be aware of what might come up.

In no particular order…

Be clear about remote positions

In a world where we’ve all been forced to work remotely it’s good to know what a role is actually going to be like when some normalcy returns to the world.

Is the role you’re advertising a permanent remote position? If it is, is it fully remote or are some days in the office expected / required?

Be clear about where you can hire remotely. Are there any legal or financial reasons why you can or can’t employ someone in a given country? If so, state those up front and save everyone some time.

I’d say it’s also good to mention any “core hours” or the time zones the majority of the team work in. Not everyone will be comfortable either with the asynchronous communication or having to shift their working day earlier or later. On a similar note, make sure to mention if contact with external people is part of the role and if there are any time constraints around that.

Longer interview process

One thing that became clear to me fairly early on was how much more spread out the whole interview process was.

It’s not uncommon for a company to want to perform a few different interviews (e.g. technical, behavioural, etc.) or with a few different interviewers. In face-to-face interviews these normally tend to happen all in the same day, one after another. In a remote approach this was much more dragged out with interviews spread over multiple weeks. As an example, I had one application spread over a month and half with a total of 7 hours of interviews plus time taken on a take-home programming exercise.

This all amounted to dragging out all the usual stress/anxiety that comes with interviewing. If the candidate is currently unemployed (not uncommon during this pandemic) then any delay to the process could have a serious impact on their life and their family.

Pants & T-shirt

Ok, so it’s not all bad.

While I don’t recommend actually sitting in your pants (or underwear for the American readers) and a t-shirt while on an interview call I did notice there was much less worry or expectation about what to wear. Where previously I’d feel compelled to wear a shirt and smart trousers to an interview I could instead wear shorts and a t-shirt and feel much more comfortable in what I was wearing.

This may seem like a small thing but that extra level of comfort adds wonders to confidence during the interview.

Audio is most important

One thing that really struck me was how important it was that the audio worked, rather than the video.

While the video does allow both parties to pick up on non-verbal cues, it’s much more important that both can hear each other clearly so neither misunderstand.

Google Meet is especially bad in this area from my experience. When the connection degrades it’s both the video and the audio that suffers, leading to several "sorry, can you repeat that please?”. In contrast I’ve found Zoom to fare quite well in this area as in a default setup the audio is prioritised slightly higher than video and screen share traffic.

If free / open source is more your thing then Jitsi Meet prioritises audio, even to the point where it’ll stop the video if required.

On a similar note, make sure you have a stable internet connection and try to avoid tethering from a mobile where possible. If unavoidable make sure to let the other person know at the start and suggest switching to audio only.

Include the company name in email subjects

When I received emails and calendar invites only about half of them included the company name in the subject, instead most were titles something like "Interview with Marcus Noble". I couldn’t see at a glance which company it was related to and as I’d applied to a lot of different places it was quite a chore to keep things organised.

When sending calendar invites I much prefer to know the company and the type of interview (if relevant) than the person I’m interviewing with. Information about the interviewer and anything else relevant can be contained in the main body content but when looking at the overview of my calendar I want to know quickly which interviews I need to prepare for that day.

Explain the process

This is something that almost all the companies I spoke with did and it’s super valuable for the candidate.

Knowing exactly what to expect and when means the candidate can just focus on showing their best self during the interview process and not having to get anxious about all the worst case scenarios that flood their mind. If you can include this information along with the initial job specification then that’s even better!

Have a single contact person

I noticed that there was the vast number of different people that were involved in the whole interviewing process. It wasn’t uncommon to interact with a technical recruiter, the hiring manager and multiple different people from various teams. Having a single person to contact if needed and knowing that from the start goes a long way to relieving some stress on the candidates side.

Scheduling is hard

Pretty much every company I interviewed with used something similar to Calendly to make it easier for them to arrange the best time for the interview.

The problem though is the interview candidate has to update their availability on all of these. So once one company has scheduled a call you then need to go through all the others you have given your availability to and update them.

Ideally it’d be better if the candidate could use a single service for their availability but I can’t see that happening in reality.

Competitive salary

(Ok this one is more of a general rant)

If your salary is so competitive then brag about it! Put it in the job spec and save wasted time on both sides.

Not wiling to share your salary? Then ask the candidates what they’re looking for, if it’s more than you can afford then thank them for their interest and let them know you don’t currently have any roles suitable. DO NOT use this as a way of underpaying people. If someone is asking for a value below what you think the role should be don’t give them that low salary to save money. You’ll get a much happier and productive worker if you pay them fairly.

LinkedIn

I’ve not had much luck with LinkedIn but your mileage may vary.

Most of the people contacting me on there were just automatic messages and largely irrelevant to what I was looking for.

I forget where I first saw this suggested but adding an Emoji into your name is a really good way of filtering out the automatic messages from the personal ones. Anything that was sent to "☁️ Marcus" was immediately ignored as they were mostly completely irrelevant.

Ghosting / Delay in Responses

One thing that was especially frustrating was the companies that didn't seem like they had the time or interest to actually take part in the hiring process. I had a few companies that didn't even respond to an application and a good few others that were slow (over a week) to respond at each stage.

Not only does this add to the longer process mentioned above it also makes candidates either worry about the position or move on to something else.

If you don't plan to proceed with an application then let that candidate know. Even a basic boilerplate email would be enough so the candidate can focus elsewhere.

All-in-all the process wasn’t too bad. Being able to do all the interviews in a setting that I was comfortable was a big positive but I also understand that I’m in a very privileged position where I have a quiet space to do that and not currently need a new job.

I'm not sure how different the situation would be if I wasn't in this privileged position but I can't imagine it being pleasant. It's worth keeping in mind when you're hiring that you may be automatically excluding some quality candidates due to either dragging out the whole process or not being flexible enough. A "we embrace diversity" message at the bottom of your job spec isn't enough.

If anyone is currently in the process of job hunting (or hiring) and wants to chat about any of this or just get some general advice I'm always happy to help where I can (though I'm far from an expert). Feel free to message me on Twitter (DMs open).

Bonus: Resources for those job hunting

• De-Coding the Technical Interview Process by Emma Bostian fantastic book (although examples are web dev based) outlining a lot of what to expect in the interview process.
• Tech Interview Handbook - Wish I'd come across this sooner. Lots of great information (especially liking the Questions to ask page) but does seem fairly USA centric.

2021-08-08: Added latency details

I've been using Scaleway's Kapsule managed Kubernetes offering for my personal projects for a while now (this blog is running on it) so I was pretty excited when they announced a new managed Kubernetes offering dubbed Kosmos. What makes Kosmos really interesting is it's sold as a multi-cloud Kubernetes offering.

Kubernetes Kosmos is the first-ever managed Kubernetes engine, allowing you to attach an instance or dedicated server from any cloud provider to a Scaleway’s Kubernetes control plane.

With Kubernetes Kosmos, you can:

• Deploy clusters across multiple providers
• Attach nodes from any cloud providers
• Benefit from scalability & stability of cross-provider infrastructure
• Access a fully managed & redundant Control Plane

This is quite a powerful approach as it allows the user to take advantage of the best functionality of each cloud provider on offer. But on top of that you are able to use a managed control plane with your on-premise instances.

This is all made possible thanks to the Kilo project that's built on top of WireGuard to provide multi-cloud networking.

Let's give it a try

Creation of a Kosmos cluster is a fairly easy process via the Scalway web console. Select the cluster type you want to create (Kosmos), the region to deploy into and the version of Kubernetes you want to use.

You also have the option of giving your cluster a name, description and some tags to differentiate it from any other clusters you may have.

During the initial creation you have the option of adding a Scaleway managed node pool. You can choose to skip this and instead set up your node pools later.

Once the cluster has finished creating and is ready to use you can add more node pools. One thing I did find interesting when adding a second Scaleway managed node pool was I could pick a different region to what my cluster was deployed into.

Above you can see two node pools, the one called default I created along with the Kosmos cluster, the other was created after in a different region to the control plane. So far we have a multi-region Kubernetes cluster.

Multi-cloud

Now that we have Kosmos set up it's time to give the multi-cloud functionality a try. The process for this isn't much more that adding another node pool to the cluster but this time we need to select the "Multi-Cloud" type instead of "Scaleway" for our new pool.

Once the pool is created you can pick to "Add a node" which will provide you with instructions on how to register an external node with your Kosmos cluster.

It's worth noting here that currently only Ubuntu 20.04 is supported as the host OS (20.10 currently fails with an apparmor_parser error) and the setup script needs to be run as root (I find it best to run the setup script from the init userdata if your cloud provider supports it).

Adding an external node

For our first multi-cloud nodes I'm going to be using Civo.

The multicloud-init.sh script provided by Scaleway doesn't provide much in the way of progress updates and the external node can take quite a while before it shows up in kubectl so to keep an eye on anything going wrong I recommend taking a look at journalctl -f for your first nodes.

After a while your new nodes should pop up in kubectl.

The init script doesn't currently provide any way of passing in node labels for Kubelet so to make things clearer I've manually added a cloud label to each node.

While we're at it, lets add another cloud provider into the mix. This time I'm going to add a bare metal instance from Equinix Metal.

Finally, let's add an on-premise instance to our cluster (or rather a virtual machine running on my Mac).

We've now got a single Kubernetes cluster with worker nodes spread across (at least) 5 physical locations, 3 different cloud providers and a mix of different resource capabilities.

Making use of the cluster

The main thing I wanted to test with this multi-cloud approach was how ingress handled pods being scheduled elsewhere. Turns out there's no surprises here. A LoadBalancer is created and hosted by Scaleway that then points to the appropriate Kuberentes Service resource which in turn points to the appropriate pod no matter where it happens to be scheduled.

It'd be good to do some more testing at some point on latency and network traffic costs to work out if this is a cost-effective approach or not. If anyone does dig into this more please do let me know!

Some improvements needed

It's not all perfect, after all it is still an early beta.

I've already mentioned some of the limitations of the multicloud-init.sh script such as not being able to add node labels at creation time. These are fairly easy to work around though and likely to be supported in the future. The lack of progress visibility when adding a new external node is a bit of a pain when first trying out the service but not really an issue once you've got everything set up right.

One thing I did notice that wasn't ideal was if an external node is deleted in the external cloud the associated node resource in the cluster doesn't get removed and instead just changes to a "NotReady" status.

This doesn't seem like a big deal at first but it does leave some pods in a pending state while they wait for the node to become ready again. If you're taking advantage of autoscaling of your external nodes this is likley to crop up fairly quickly and could cause unexpected issues.

The last issue I hit was when trying to add an ARM based machine (either a local Raspberry Pi or a cloud based ARM instance). The multicloud-init.sh script doesn't currently support architectures other than x86_64 but a small tweak to the script and we can get an ARM node set up...

Adding ARM support

To get ARM-based instances working we need to make a few changes to the multicloud-init.sh script provided by Scaleway. Below you can see the diff of the changes I used to get an ARM instance running on Equinix Metal.

7a8,12
> os_arch="amd64"
> if [[ "$(arch)" != "x86_64" ]]; then
>   os_arch="arm64"
> fi
>
90c95,99
<   apt-get install -y containerd.io > /dev/null 2>&1
---
>   if [[ "${os_arch}" == "amd64" ]]; then
>     apt-get install -y containerd.io > /dev/null 2>&1
>   else
>     apt-get install -y containerd > /dev/null 2>&1
>   fi
221c230
<   curl -L "https://storage.googleapis.com/kubernetes-release/release/v${kubernetes_version}/bin/linux/amd64/kubectl" \
---
>   curl -L "https://storage.googleapis.com/kubernetes-release/release/v${kubernetes_version}/bin/linux/${os_arch}/kubectl" \
223c232
<   curl -L "https://storage.googleapis.com/kubernetes-release/release/v${kubernetes_version}/bin/linux/amd64/kubelet" \
---
>   curl -L "https://storage.googleapis.com/kubernetes-release/release/v${kubernetes_version}/bin/linux/${os_arch}/kubelet" \

This currently only works with 64-bit ARM instances (so not all Raspberry Pis) but it shouldn't take too much to expand it to correctly support more architectures. Hopefully Scaleway will provide support for ARM soon as we're seeing a lot of advances in ARM machines lately.

There is still one small issue with ARM instances after using this modified script - the node-problem-detector DaemonSet used by Scaleway currently only targets x86 so will fail to run on our new ARM instance.

Cross-cloud latency

Mark Boost asked on Twitter about the general performance and latency involved with multi-cloud clusters so I did a quick and dirty test to see how things generally behave.

My test involved 4 different node pools, each with a single instance:

• Scaleway - par1 = 1x DEV1_M
• Scaleway - ams1 = 1x DEV1_M
• Civo - lon1 = 1x Small
• Civo - nyc1 = 1x Small

(Note: The Kosmos control plane is hosted in Scaleway par1 region)

✨ k get nodes --output custom-columns=Name:.metadata.name,Cloud:".metadata.labels.topology\.kubernetes\.io/region"
Name                                             Cloud
scw-multicloud-civo-lon-00a1d7529f8340109990e1   civo-lon1
scw-multicloud-civo-lon-8deec4c46be0445fb89c2f   civo-nyc1
scw-multicloud-scaleway-ams1-6d6b9ac4eeed4f50b   nl-ams
scw-multicloud-scaleway-par1-fd031cee6aa64843b   fr-par

For my test case I'm leveraging Linkerd along with their emojivoto demo application. The Linkerd pods are running on one of the two Scaleway instances and for each test I will target a different node for the emojivoto deployments. Each test is only run for a very short period of time (a few minutes) so results are very much ballpark figures and not to be given too much weight.

My main reason for this approach is that I'm lazy and Linkerd gives a fairly nice set of dashboard for latency with very little effort.

Ok, now for the results...

First up we have all deployments scheduled onto the Scaleway-ams1 node:

You can see this is fairly bumpy to start with as the pods get started but all fairly low as we'd expect.

Next up we have all pods scheduled onto the Scaleway-par1 node:

This is the same region as our control plane. All the latencies are very low with only a little bit of fluctuation.

We then have the first of our external nodes - Civo-lon1:

This seems to have much more fluctuation in the latency but overall still showing as very low with a lot of requests still inline with those nodes hosted on Scaleway.

The last node to try is Civo-nyc1:

This one looks to be trending slightly slower but still within the same sort of range as the other nodes. It's worth pointing out that this node is physically located the furthest away with all the other nodes located within Europe.

Finally, as we have 4 nodes and 4 different deployments, I wanted to test how things would look with the application spread over all the available nodes. This next result shows the latencies with each of the deployments scheduled to a different node:

Right away you can see that this have much more latency with the Y-axis being at least double that of any of the previous results. That being said, it is still all within 100ms and for this very small test at least within an acceptable range.

Final Thoughts

Combining it with something like cluster-api or Gardener could make the management of multi-cloud resources much more manageable.

Overall I'm pretty impressed with Kosmos so far, I think it has a lot of potential and I expect some of the other cloud providers will offer something similar eventually. It seems like a simpler alternative to Kubernetes federation when multi-tenancy isn't a requirement.

getopts makes it easier to parse arguments passed to a shell script by defining which letters your script supports. It supports both boolean and string style arguments but only supports single letter flags. (e.g. -h and not --help)

Example usage:

#!/bin/bash

NAME="World"
FORCE=false

showHelp() {
    echo "Usage: example.sh [args]"
    exit 0
}

while getopts 'hfn:' FLAG
do
  case $FLAG in
    h) showHelp ;;
    f) FORCE=true ;;
    n) NAME=$OPTARG ;;
    *) echo "Unsupported argument flag passed" ;;
  esac
done

echo "Hello, $NAME"

Notice the : following the n? That indicates that a value should follow the argument flag (n in this example) and will be made available as the OPTARG variable.

---
# This handles the loading of my automations
#
# https://www.home-assistant.io/docs/automation/
#
automation: !include ../automations.yaml
automation split: !include_dir_list ../automations    # <--

I found myself staring at this for a while, followed by searching the Home Assistant documentation website to see if split was a special keyword I wasn't aware of.

And then it dawned on me! As all JSON is valid YAML, and JSON keys can be pretty much any string it makes sense that YAML supports it.

The above example converted to JSON using yq looks like this:

// yq config.yaml -o json
{
  "automation": "../automations.yaml",
  "automation split": "../automations"
}

Knowing this, I decided to try out a few more variations to see what works...

YAML:

---
123: Valid
---: also valid
5.5: yup! this too
#how about this?: nope, this is treated as a comment
//: yeah, totally valid
✨: yep!
[1]: Works
[1, 2]: Still works, treated as string
{another}: This one is interesting

JSON:

{
  "123": "Valid",
  "---": "also valid",
  "5.5": "yup! this too",
  "//": "yeah, totally valid",
  "✨": "yep!",
  "[1]": "Works",
  "[1, 2]": "Still works, treated as string",
  "{\"another\"=>nil}": "This one is interesting"
}

Depending on the library used, varying results can be generated. For example, yamlonline (update: website no longer online) returns the following for the same input:

{
	"1": "Works",
	"123": "Valid",
	"---": "also valid",
	"5.5": "yup! this too",
	"//": "yeah, totally valid",
	"✨": "yep!",
	"1,2": "Still works, treated as string",
	"[object Object]": "This one is interesting"
}

I discovered this today when none of my nodes seemed to be connecting to the control plane. Eventually discovered the hostname of the node was longer than 63 characters (mainly due to multiple subdomain levels) and so the kubernetes.io/hostname label being automtically added to the node was causing Kubernetes to reject it.

If you hit this like me, the hostname used for the label can be overridden using the --hostname-override flag on kubelet or by setting the value of the label yourself with the --node-labels flag.

https://s2.googleusercontent.com/s2/favicons?domain_url=https://marcusnoble.co.uk/

Example:

You can even provide any page, not just the root URL.

e.g. https://s2.googleusercontent.com/s2/favicons?domain_url=https://marcusnoble.co.uk/2020-11-10-t-i-l-how-to-get-the-favicon-of-any-site/