Lots of warnings : rejected connections in etcd

[RanjithNKtrr]

I am seeing lots of warning messages like below in etcd pod.

{"level":"warn","ts":"2023-12-01T07:12:44.205237Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:38852","server-name":"","error":"EOF"}

Version:
etcdctl version: 3.5.9
API version: 3.5

Kubernetes Version:
1.27.0

Etcd is configured with TLS and I tried calling status endpoint using etcdctl and getting the response. But seeing lots of above warning without any clue. Any help will be appreciated.

[jmhbnz]

Hi @RanjithNKtrr - Thanks for your question, so were you able to get a status response using etcdctl?

The log message you are seeing relates to TLS handshake failures, are you supplying parameters for certificates when you use etcdctl?

For an example of supplying certificates when using etcdctl refer to: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#securing-communication

[RanjithNKtrr]

Hi @jmhbnz , Thanks for your response. I can able to get the status response using etcdctl , Also I am passing parameters for certificates. I have given sample request I am sending

etcdctl --endpoints=https://127.0.0.1:2379 --cert=/etc/kubernetes/pki/etcd-client.crt --key=/etc/kubernetes/pki/etcd-client.key --cacert=/etc/kubernetes/pki/ca.crt endpoint status

I can able to get the response everything looks good. But, seeing lots of warning messages in etcd pod.

[jmhbnz]

Thanks @RanjithNKtrr - What is the process / responsible network traffic for the "remote-addr" combination of 127.0.0.1:38852 shown in the logs?

You will need to monitor and investigate the network traffic on the server where your etcd instance is installed to understand what is generating the traffic that is in turn generating the etcd logs about failed tls connections.

[RanjithNKtrr]

Thanks for the input @jmhbnz . I will monitor the traffic and update with further analysis.

[RanjithNKtrr]

Hi @jmhbnz , I found one observation like, when I removed the liveness probe from etcd.yaml , I am seeing the frequency of TLS error got reduced from like 10 to 1 . Looks the call from kubelet server to etcd is causing issue? Not sure

Below provided the liveness probe configuration

livenessProbe:
failureThreshold: 8
exec:
command:
- sh
- -c
- ETCDCTL_API=3 etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/etcd-client.crt --key=/etc/kubernetes/pki/etcd/etcd-client.key endpoint status
initialDelaySeconds: 15
timeoutSeconds: 15

I tried by directly calling above command in the pod , i am getting the response without any issue.

[IEatCakes]

The same.

ERROR:

{"level":"warn","ts":"2024-01-25T11:03:28.208539Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.150.67.5:46586","server-name":"","error":"EOF"}

get status successfully
ETCDCTL_API=3 etcdctl endpoint status --write-out=table --endpoints=https://10.150.66.22:2379,https://10.150.65.46:2379,https://10.150.67.5:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key

etcd Version: 3.5.9, k8s version v1.28.3

[RanjithNKtrr]

@jmhbnz I tried tcpdump by listening the etcd port but couldn't able to trace where the request is coming. Can you provide some tool that will help to trace it?

@IEatCakes Yup same issue i can also get the status successfully but seeing this error

[RanjithNKtrr]

One more observation: The remote address combination is changing to different port number for each request so couldn't able to figure it out

[RanjithNKtrr]

Hi @jmhbnz I got the issue, The issue is same like one already created in etcd board

#12760

So whenever we are calling etcdctl multiple times , we are getting this issue. So whenever we are providing liveness probe which is using etcdctl for checking the endpoint status we are getting more reject warning. looks some issue with etcdctl. Can you provide some update regarding this