Kubelet Mirror Pod Behaviours   Kubelet Mirror Pod 行为

Introduction  介绍

Kubernetes, and specifically the kubelet, let you load pod specifications from a directory on disk. Improper use of this functionality can lead to some strange things happening. This post will explore some of those edge cases.
Kubernetes，特别是 kubelet，允许你从磁盘上的目录加载 Pod 的规范。不当使用此功能可能会导致一些奇怪的事情发生。本文将探讨一些边缘情况。

In a typical Kubernetes cluster deployment, the Kubelet on each node reaches out to the API Server to find out what pods should be running on its node. This is core to maintaining the correct cluster status in the event of a node outage. However, some deployments of Kubernetes run the API Server and other control plane components as pods themselves. To allow this bootstrapping when the Kubelet has no API Server to communicate with, the Kubelet can be configured with a location from which to read static pod configurations.
在典型的 Kubernetes 集群部署中，每个节点上的 Kubelet 都会联系 API 服务器，以确定哪些 Pod 应该在其节点上运行。这对于在节点中断时维持正确的集群状态至关重要。然而，某些 Kubernetes 部署会将 API 服务器和其他控制平面组件本身作为 Pod 运行。为了在 Kubelet 没有 API 服务器进行通信时也能进行这种引导，可以为 Kubelet 配置一个读取静态 Pod 配置的位置。

This location can be configured as either a web location or a local directory, with the latter being the most common. In a kubeadm setup, the default directory is /etc/kubernetes/manifests. Inspecting this directory on a clean KinD cluster shows a number of manifests, generated to allow essential control plane components to start.
此位置可以配置为 Web 位置或本地目录，其中后者最为常见。在 kubeadm 设置中，默认目录为 /etc/kubernetes/manifests 。在干净的 KinD 集群上检查此目录会显示一些清单，这些清单是为了启动必要的控制平面组件而生成的。

1
2
3
4
5
6
7
8

root@kind-control-plane:/etc/kubernetes/manifests# ls -la
total 28
drwxr-xr-x 1 root root 4096 Oct 11 14:05 .
drwxr-xr-x 1 root root 4096 Oct 11 14:05 ..
-rw------- 1 root root 2406 Oct 11 14:05 etcd.yaml
-rw------- 1 root root 3896 Oct 11 14:05 kube-apiserver.yaml
-rw------- 1 root root 3428 Oct 11 14:05 kube-controller-manager.yaml
-rw------- 1 root root 1463 Oct 11 14:05 kube-scheduler.yaml

These static pods correspond to objects we can observe through the Kubernetes API Server. These are mirror pods, which are a specific instance of a pod type created to track statically created pods in the API Server. Each pod is named with the value specified in the yaml file, suffixed with the name of the node the pod was created on. Note 4 pods in the output below which end with kind-control-plane and correspond to the manifests above.
这些静态 Pod 对应于我们可以通过 Kubernetes API 服务器观察到的对象。它们被称为镜像 Pod ，是特定 Pod 类型的实例，用于跟踪 API 服务器中静态创建的 Pod。每个 Pod 都使用 yaml 文件中指定的值命名，并以创建 Pod 的节点名称作为后缀。请注意，下方输出中有 4 个 Pod，它们以 kind-control-plane 结尾，与上面的清单相对应。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

❯ kubectl get pods -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-6f6b679f8f-9r7j9                     1/1     Running   0          27s
coredns-6f6b679f8f-qf4vz                     1/1     Running   0          27s
etcd-kind-control-plane                      1/1     Running   0          33s
kindnet-v5gcj                                1/1     Running   0          27s
kube-apiserver-kind-control-plane            1/1     Running   0          33s
kube-controller-manager-kind-control-plane   1/1     Running   0          32s
kube-proxy-ltn69                             1/1     Running   0          27s
kube-scheduler-kind-control-plane            1/1     Running   0          32s

The Kubernetes documentation on static pods specifies a good amount of the behaviour and the constraints which are placed on static pods. Some of these have significant security benefits, like not allowing static pods to use configmaps, secrets, or service accounts. It is possible to mount host volumes in to the container, potentially allowing a user to escalate permissions on the node if they have write access to the directory¹.
Kubernetes 文档中关于静态 Pod 的说明详细说明了静态 Pod 的诸多行为和约束。其中一些约束具有显著的安全优势，例如不允许静态 Pod 使用 configmap、secret 或服务账户。此外，还可以将主机卷挂载到容器中，如果用户对目录 ¹ 具有写权限，则可能允许其提升节点的权限。

This seems like some fairly complex functionality, and as with anything complicated, there are edge cases which lead to odd behaviours. We’ll dive into one of these below.
这看起来是一个相当复杂的功能，而且和任何复杂的事情一样，也存在一些会导致奇怪行为的极端情况。我们将在下面深入探讨其中一种情况。

Admission Control  准入控制

In a modern cluster, the kubelet is always permitted to create a pod through Node Authorization. Created pods aren’t restricted to specific namespaces and, until recently, I’d never tried to create a pod in a namespace restricted by Pod Security Admission.
在现代集群中，kubelet 始终被允许通过节点授权创建 Pod。创建的 Pod 并不受特定命名空间的限制，而且直到最近，我还从未尝试在受 Pod 安全准入 (Pod Security Admission ) 限制的命名空间中创建 Pod。

If you’re reading this, take a second to think about what will happen. I posed the question “Do the kubelet’s static pods get affected by admission control” to some colleagues, and the response was heavily weighted towards “No”. This makes sense, and was my initial suspicion too. The whole point of a static manifest is that it allows the kubelet to start workloads without a control plane to communicate with, so why should admission control get involved when that is a component which runs in the control plane itself?
如果你正在读这篇文章，请花点时间想想会发生什么。我向一些同事提出了“kubelet 的静态 pod 会受到准入控制的影响吗？”这个问题，他们的回答大多是“不会”。这很有道理，也是我最初的怀疑。静态清单的意义在于它允许 kubelet 无需与控制平面通信即可启动工作负载，那么既然准入控制本身就是一个在控制平面中运行的组件，为什么还要介入呢？

So let’s try it out by creating two static pods. One is a standard pod with an empty security context, and one sets the privileged flag.
让我们尝试创建两个静态 Pod。一个是标准 Pod，安全上下文为空，另一个设置了 privileged 标志。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

❯ cat pod-standard.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: standard-pod
  name: standard-pod
spec:
  containers:
  - image: nginx
    name: standard-pod
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

❯ cat pod-privileged.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: privileged-pod
  name: privileged-pod
spec:
  containers:
  - image: nginx
    name: privileged-pod
    resources: {}
    securityContext:
      privileged: true
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

We can then check if the pods are running:

1
2
3

❯ kubectl get pods -n baseline -w
NAME                              READY   STATUS    RESTARTS      AGE
standard-pod-kind-control-plane   1/1     Running   0             14s

Only one pod is running, so that answers that question, right? The privileged pod wasn’t created. Let’s check the kubelet’s logs to be sure.

1
2
3
4
5
6
7

root@kind-control-plane:/# journalctl -u kubelet | grep standard-pod
Oct 12 13:38:16 kind-control-plane kubelet[693]: I1012 13:38:16.210457     693 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="baseline/standard-pod-kind-control-plane" podStartSLOduration=18.210431419 podStartE2EDuration="18.210431419s" podCreationTimestamp="2024-10-12 13:37:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-10-12 13:38:16.210392461 +0000 UTC m=+55.347144485" watchObservedRunningTime="2024-10-12 13:38:16.210431419 +0000 UTC m=+55.347183443"

root@kind-control-plane:/# journalctl -u kubelet | grep privileged-pod
Oct 12 13:38:00 kind-control-plane kubelet[693]: E1012 13:38:00.645679     693 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"privileged-pod-kind-control-plane\" is forbidden: violates PodSecurity \"baseline:latest\": privileged (container \"privileged-pod\" must not set securityContext.privileged=true)" pod="baseline/privileged-pod-kind-control-plane"
Oct 12 13:38:17 kind-control-plane kubelet[693]: E1012 13:38:17.209797     693 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"privileged-pod-kind-control-plane\" is forbidden: violates PodSecurity \"baseline:latest\": privileged (container \"privileged-pod\" must not set securityContext.privileged=true)" pod="baseline/privileged-pod-kind-control-plane"
Oct 12 13:38:18 kind-control-plane kubelet[693]: E1012 13:38:18.217397     693 kubelet.go:1915] "Failed creating a mirror pod for" err="pods \"privileged-pod-kind-control-plane\" is forbidden: violates PodSecurity \"baseline:latest\": privileged (container \"privileged-pod\" must not set securityContext.privileged=true)" pod="baseline/privileged-pod-kind-control-plane"

This certainly looks like the container wasn’t created, as there was no pod startup, but something still doesn’t seem right. Let’s confirm there’s definitely no container running on the node:

1
2
3
4
5

root@kind-control-plane:/# crictl ps 
CONTAINER     IMAGE         CREATED            STATE   NAME           ATTEMPT POD ID        POD
cfb34cd8921c4 048e090385966 About a minute ago Running privileged-pod 0       0057552e19e0f privileged-pod-kind-control-plane
fc9e700421fe6 048e090385966 About a minute ago Running standard-pod   0       c145bd43367e8 standard-pod-kind-control-plane
<-- snip -->

Well, that’s interesting. The pod wasn’t able to register on the API Server, but the container itself is running. We can confirm this again by checking the pod ID rather than the container ID:

1
2
3
4

root@kind-control-plane:/etc/kubernetes/manifests# crictl pods 
POD ID        CREATED            STATE NAME                               NAMESPACE ATTEMPT RUNTIME
0057552e19e0f About a minute ago Ready privileged-pod-kind-control-plane  baseline  0       (default)
c145bd43367e8 About a minute ago Ready standard-pod-kind-control-plane    baseline  0       (default)

Yep, definitely running. We can confirm this another way by using the kubelet API (accessed, in this case, using the wonderful Kubeletctl):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

root@kind-control-plane:/# ./kubeletctl_linux_arm64 --server 127.0.0.1 pods 
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
┌─────────────────────────────────────────────────────────────────────┐
│                          Pods from Kubelet                          │
├────┬───────────────────────────────────┬───────────┬────────────────┤
│    │ POD                               │ NAMESPACE │ CONTAINERS     │
├────┼───────────────────────────────────┼───────────┼────────────────┤
<-- snip -->
│  6 │ privileged-pod-kind-control-plane │ baseline  │ privileged-pod │
├────┼───────────────────────────────────┼───────────┼────────────────┤
<-- snip -->
│ 11 │ standard-pod-kind-control-plane   │ baseline  │ standard-pod   │
<-- snip -->

Non-existent Namespaces

Okay, we’ve confirmed that running a pod that can’t get past admission control will run as a static node on the pod, but not as a mirror pod in the Kubernetes API Server. How about other attempts to create a local container that won’t be recognised by the API Server?

When we try to create a pod in a namespace that doesn’t exist, two errors are returned². One is from kubelet.go, and one is from event.go, both detailing that the API Server rejected the request for the pod to be created. However, as with the admission control example, we can see that the pod was successfully started.

1
2
3

root@kind-control-plane:/# crictl ps 
CONTAINER     IMAGE         CREATED         STATE   NAME        ATTEMPT POD ID         POD
78c94040ffd50 048e090385966 2 minutes ago   Running fake-ns-pod 0       89d585212ff8b fake-ns-pod-kind-control-plane

As with the previous example, we can execute commands inside the running container through Kubeletctl or by talking directly to the kubelet with curl, but not through kubectl exec.

Uses

It’s probably fair to say that creating pods which can’t be viewed by cluster administrators through kubectl is not a useful behaviour in day-to-day running of a cluster. I can’t think of any legitimate uses of this behaviour, but I do think it would be useful to someone trying to hide a running workload on a Kubernetes cluster they’ve compromised. Not that I’d talk about that or anything.