Troubleshooting k3s Agent Container Errors: A Real-World Fix
Recently, I ran into a pesky issue on one of my k3s nodes, a server I’ll call NEXUS, running in a homelab cluster. The k3s agent logs were filling up with errors about missing containers, and I couldn’t ignore them any longer. If you’ve seen messages like Failed to create existing container: ... task ... not found in your k3s logs, this post is for you. I’ll walk you through how I diagnosed and resolved the issue, with practical steps to get your cluster back on track.
The Problem: Missing Container Errors
The logs on NEXUS looked something like this (I’ve anonymized the details):
Apr 13 23:51:57 NEXUS k3s[1234]: E0413 23:51:57.033807 1234 manager.go:1116] Failed to create existing container: /kubepods.slice/.../cri-containerd-abcd1234.scope: task abcd1234 not found
Apr 13 23:52:00 NEXUS k3s[1234]: E0413 23:52:00.140450 1234 manager.go:1116] Failed to create existing container: /kubepods.slice/.../cri-containerd-efgh5678.scope: task efgh5678 not found
These errors suggested that k3s was trying to manage containers that no longer existed in the container runtime (containerd, in my case). It was as if k3s and containerd were out of sync, with k3s holding onto references to “ghost” containers.
Why This Happens
This issue can pop up for a few reasons:
Stale State: k3s thinks a container exists, but containerd has already cleaned it up.
Runtime Issues: containerd might have crashed or failed to track tasks properly.
Node Disruptions: A reboot, disk pressure, or network hiccup could desynchronize the cluster.
In my case, NEXUS had been running smoothly until a recent power cycle during some network maintenance on my 192.168.10.0/24 subnet. I suspected the abrupt restart left k3s and containerd in a mismatched state.
Step-by-Step Diagnosis and Fix
Here’s how I tackled the problem, with commands you can adapt to your setup.
1. Check the Environment
I started by confirming the k3s version:
k3s --version
Output: k3s version v1.27.8+k3s1. Not the latest, but recent enough. NEXUS was an agent node in a cluster with two servers (let’s call them ALPHA and BETA, reachable at 10.0.0.10:6443). I also verified containerd was running:
systemctl status containerd
It was active, but I checked its logs for clues:
journalctl -u containerd
Nothing obvious, so I moved on.
2. Inspect k3s Health
The k3s service was up:
systemctl status k3s
But the errors kept piling up. I checked the cluster’s view of NEXUS:
kubectl get nodes
NEXUS was Ready, so the issue wasn’t critical enough to break workloads, but it was annoying.
3. Peek at Containers
I used containerd’s CLI (ctr) to see what containers were actually running:
ctr -n k8s.io tasks ls
ctr -n k8s.io containers ls
None of the task IDs from the logs (like abcd1234) showed up. This confirmed my suspicion: k3s was trying to manage containers that containerd had already forgotten.
4. Clean Up the Mess
To fix the desync, I decided to clear out the stale state. Here’s what I did:
Stop k3s:
systemctl stop k3s
Clear containerd’s k8s namespace:
I listed all containers in the k8s.io namespace:
ctr -n k8s.io containers ls
Since I was okay with potentially restarting workloads, I deleted everything (use caution here):
ctr -n k8s.io containers ls -q | xargs ctr -n k8s.io containers delete
Restart services:
systemctl restart containerd
systemctl start k3s
5. Verify the Fix
After restarting, I checked the logs:
journalctl -u k3s
The errors were gone! I also confirmed pods were running:
kubectl get pods --all-namespaces
Everything looked healthy, and NEXUS was back to normal.
Preventing Future Issues
To avoid this headache again, I took a few steps:
Upgraded k3s:
curl -sfL https://get.k3s.io | sh -
The latest version might have fixes for edge cases like this.
Monitored Disk Space:
df -h
NEXUS had plenty of space, but I set up alerts to catch issues early.
Checked Clock Sync:
timedatectl
My NTP server (10.0.0.5) was keeping things in sync, but it’s worth double-checking.
When to Escalate
I find myself doing this pretty frequently in my more experimental lab environments. In a lot of cases, it does help. k3s keeps behind a lot of state to help recovery, but in some cases this is extra baggage that can prevent a clean rebuild. Make sure you always backup. If the above doesn’t work, try these:
Rejoin the Node:
Stop k3s, uninstall it (k3s-agent-uninstall.sh), and rejoin the cluster using the token from ALPHA or BETA.
Dig Deeper:
Collect logs (journalctl -u k3s > k3s.log) and search the k3s GitHub issues for similar reports.
Backup First:
If you’re clearing state, back up /var/lib/rancher/k3s to avoid losing critical data.
Final Thoughts
This issue was a reminder that lightweight Kubernetes distros like k3s, while awesome for small clusters, can still throw curveballs. By methodically checking containerd, k3s, and the cluster state, I got NEXUS back online without major disruption. If you hit similar errors, don’t panic—just follow the steps above, and you’ll likely sort it out.
Have you run into this issue before? Drop a comment