- The pf9-kubelet service is continuously restarting.
- The following errors are present in the /var/log/pf9/kube/kube.log on the node.
watcher.go:146] Failed to watch directory "/sys/fs/cgroup/memory/system.slice/run-<UUID>.scope": inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-<UUID>.scope: no space left on device
kubelet.go:1365] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/run-<UUID>.scope: no space left on device
- Platform9 Managed Kubernetes - v4.0 and Higher
The kubelet service fails to start due to inotify resource issues. The error itself means that system is getting low on inotify watches, which enable programs to monitor file or directory changes.
- Identify the current setting.
# cat /proc/sys/fs/inotify/max_user_watches
- The output resembles the following code
- Increase value.
# echo 1048576 > /proc/sys/fs/inotify/max_user_watches
- To make the changes persistent across reboot.
# echo "fs.inotify.max_user_watches = 1048576" >> /etc/sysctl.conf
- This is a known Bug #10421 in Kubernetes.
- Use the attached script inotify_watcher_count.sh to find out which application is using the inotify resource and the exact count of inotify watches on a system.
Note: It will need the root access to parse the details from /proc filesystem