Replication fails randomly on different namespaces during initial startup

[maxime1907]

Describe the bug
Replication fails randomly on different namespaces during initial startup with this error:

{"error":"Replicated default/mytest.com-tls to 125 out of 154 namespaces: 29 errors occurred:\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f8b4b29e: Failed to update secret f8b4b29e/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f8b4b29e/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e fce332ac: Failed to update secret fce332ac/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret fce332ac/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f1166051: Failed to update secret f1166051/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f1166051/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e fb8fad78: Failed to update secret fb8fad78/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret fb8fad78/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e faf8befe: Failed to update secret faf8befe/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret faf8befe/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e kube-public: Failed to update secret kube-public/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret kube-public/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e e17810ae: Failed to update secret e17810ae/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret e17810ae/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f0beef81: Failed to update secret f0beef81/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f0beef81/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e feb1de50: Failed to update secret feb1de50/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret feb1de50/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e ece170ab: Failed to update secret ece170ab/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret ece170ab/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f7c7c7ad: Failed to update secret f7c7c7ad/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f7c7c7ad/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e e58d51b1: Failed to update secret e58d51b1/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret e58d51b1/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e ec978b8c: Failed to update secret ec978b8c/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret ec978b8c/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e fa7d64ca: Failed to update secret fa7d64ca/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret fa7d64ca/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f6dac167: Failed to update secret f6dac167/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f6dac167/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e local-path-storage: Failed to update secret local-path-storage/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret local-path-storage/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e e1e4fe27: Failed to update secret e1e4fe27/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret e1e4fe27/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e edadccb2: Failed to update secret edadccb2/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret edadccb2/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f195ae69: Failed to update secret f195ae69/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f195ae69/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e ebc17c51: Failed to update secret ebc17c51/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret ebc17c51/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e kube-node-lease: Failed to update secret kube-node-lease/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret kube-node-lease/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e e38ef1d4: Failed to update secret e38ef1d4/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret e38ef1d4/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f18578c2: Failed to update secret f18578c2/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f18578c2/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f299acb9: Failed to update secret f299acb9/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f299acb9/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f5cf6e72: Failed to update secret f5cf6e72/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f5cf6e72/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e f780b178: Failed to update secret f780b178/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret f780b178/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e fc9bd61d: Failed to update secret fc9bd61d/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret fc9bd61d/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e ea973e7e: Failed to update secret ea973e7e/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret ea973e7e/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\t* Failed to replicate Secret default/mytest.com-tls -\u003e kube-system: Failed to update secret kube-system/mytest.com-tls: secrets \"mytest.com-tls\" already exists: Failed to update secret kube-system/mytest.com-tls: secrets \"mytest.com-tls\" already exists\n\n","kind":"Secret","level":"error","msg":"could not replicate object to other namespaces","resource":"default/mytest.com-tls","time":"2023-06-30T09:11:28Z"}
I0630 09:11:28.148091       1 trace.go:219] Trace[1409226033]: "DeltaFIFO Pop Process" ID:default/mytest.com-tls,Depth:33,Reason:slow event handlers blocking the queue (30-Jun-2023 09:11:23.554) (total time: 4593ms):
Trace[1409226033]: [4.593298362s] [4.593298362s] END

To Reproduce

1. Create a kind cluster:

kind create cluster --name kind

2. Create 150 namespaces (Add more if you dont have the error)

for i in `seq 1 150`;
do
  NAMESPACE_NAME=$(uuidgen -r | cut -d "-" -f 1)
  echo $NAMESPACE_NAME
  kubectl create namespace $NAMESPACE_NAME
done

3. Create a secret inside the default namespace:

echo """apiVersion: v1
kind: Secret
metadata:
  name: mytest.com-tls
  annotations:
    replicator.v1.mittwald.de/replicate-to: .*
data:
  tls.crt: >-
    dGVzdAo=
  tls.key: >-
    dGVzdAo=
type: kubernetes.io/tls""" > /tmp/replicatesecret.yaml
kubectl apply -f /tmp/replicatesecret.yaml

4. Install kubernetes replicator with helm inside the default namespace:

helm repo add mittwald https://helm.mittwald.de --force-update
helm upgrade --install kubernetes-replicator mittwald/kubernetes-replicator --version 2.9.1 --set "args={--log-level=warn,--log-format=json}"

5. Wait like 15 seconds and restart the deployment

sleep 15
kubectl scale deployment kubernetes-replicator --replicas=0
sleep 2
kubectl scale deployment kubernetes-replicator --replicas=1

Expected behavior
Replicate normally across all different namespaces without any error

Environment:

• Kubernetes version: 1.21.5, 1.26.5, 1.27.3
• kubernetes-replicator version: v2.9.1