etcd: Error: context deadline exceeded

[Ming0706]

ENV

cluster

k8s-master01: 10.10.10.11, k8s-master02: 10.10.10.12, k8s-master03: 10.10.10.13

etcdctl version

etcdctl version: 3.5.13
API version: 3.5

service-file

/usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

config-file

/etc/etcd/etcd.config.yml

name: 'k8s-master01'
data-dir: /data/etcd
wal-dir: /data/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.10.10.11:2380'
listen-client-urls: 'https://10.10.10.11:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.10.10.11:2380'
advertise-client-urls: 'https://10.10.10.11:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://10.10.10.11:2380,k8s-master02=https://10.10.10.12:2380,k8s-master03=https://10.10.10.13:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

problem

Help! The same command executed on cluster machine k8s-master01 produces different results than on the other two cluster machines. k8s-master01 reports an error: 'Error: context deadline exceeded', but the other two cluster machines display normally.

10.10.10.11:

# command1:
export ETCDCTL_API=3
etcdctl --endpoints="https://10.10.10.13:2379,https://10.10.10.12:2379,https://10.10.10.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem member list
# output1:
{"level":"warn","ts":"2024-08-22T11:22:04.788108+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00056e000/10.10.10.13:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded

# command2:
etcdctl --endpoints="10.10.10.13:2379,10.10.10.12:2379,10.10.10.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status
# output2:
{"level":"warn","ts":"2024-08-22T11:24:18.395893+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004d8000/10.10.10.13:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint 10.10.10.13:2379 (context deadline exceeded)
{"level":"warn","ts":"2024-08-22T11:24:23.396922+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004d8000/10.10.10.13:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint 10.10.10.12:2379 (context deadline exceeded)
{"level":"warn","ts":"2024-08-22T11:24:28.398746+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004d8000/10.10.10.13:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint 10.10.10.11:2379 (context deadline exceeded)

# command3:
etcdctl member list
# output3:
4776526eb1c8ce1a, started, k8s-master02, https://10.10.10.12:2380, https://10.10.10.12:2379, false
4d0649dd161f73d8, started, k8s-master01, https://10.10.10.11:2380, https://10.10.10.11:2379, false
c6eaa306941ff399, started, k8s-master03, https://10.10.10.13:2380, https://10.10.10.13:2379, false

# command4:
etcdctl endpoint status
# output4:
127.0.0.1:2379, 4d0649dd161f73d8, 3.5.13, 20 kB, false, false, 5, 46, 46,

10.10.10.12 / 10.10.10.13:

# command1:
etcdctl --endpoints="10.10.10.13:2379,10.10.10.12:2379,10.10.10.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status
# output1:
10.10.10.13:2379, c6eaa306941ff399, 3.5.13, 20 kB, true, false, 5, 46, 46,
10.10.10.12:2379, 4776526eb1c8ce1a, 3.5.13, 20 kB, false, false, 5, 46, 46,
10.10.10.11:2379, 4d0649dd161f73d8, 3.5.13, 20 kB, false, false, 5, 46, 46,

# command2:
etcdctl --endpoints="10.10.10.13:2379,10.10.10.12:2379,10.10.10.11:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem member list
# output2:
4776526eb1c8ce1a, started, k8s-master02, https://10.10.10.12:2380, https://10.10.10.12:2379, false
4d0649dd161f73d8, started, k8s-master01, https://10.10.10.11:2380, https://10.10.10.11:2379, false
c6eaa306941ff399, started, k8s-master03, https://10.10.10.13:2380, https://10.10.10.13:2379, false