Secret data keeps disappearing and reappearing per 10 seconds in argo

[chuyuou]

Current Behavior

Argo keeps to sync the app with the secret per 10 seconds. The status of the app keeps looping similarly as below:

 Normal  OperationStarted    12s    argocd-application-controller  Initiated automated sync to 'fdf03exxxxxx'
  Normal  ResourceUpdated     12s    argocd-application-controller  Updated sync status: Synced -> OutOfSync
  Normal  ResourceUpdated     9s     argocd-application-controller  Updated sync status: OutOfSync -> Synced
  Normal  OperationCompleted  8s     argocd-application-controller  Sync operation to fdf03exxxxxx succeeded

When we check the value of the secret, the expected values keeps appearing and disappearing roughly per 10-15 seconds as well:

Expected Behavior

Secret should be stable

Steps To Reproduce

Environment

• Helm Version:
• Helm Secrets Version: 4.6.2
• SOPS version:3.9.0
• ArgoCD Version:2.12.3
• OS:
• Shell:

Anything else?

No response

[jkroepke]

If possible, check audit log of kubernetes for who is invoke the delete of the secret.

[chuyuou]

If possible, check audit log of kubernetes for who is invoke the delete of the secret.

It's argocd-application-controller keeps on patching the secret. @jkroepke

Would it be because I mounted the secret in secrets://? My argo application manifest is similar to below:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  project: default
  source:
    repoURL: https://github.com/domain-group/argocd.git
    targetRevision: feat/xxx
    path: prometheus/
    helm:
      valueFiles:
        - secrets://../../../prometheus-secret.yaml
      releaseName: nonprod-prometheus
      version: v3
      parameters:
        - name: alertmanager.alertmanagerSpec.config.route.receiver
          value: nonprod_slack
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

[jkroepke]

How you install helm-secrets on ArgoCD?

[chuyuou]

How you install helm-secrets on ArgoCD?

I installed based on https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration, using custom docker image

# Stage 1: Build GitOps tools
FROM debian:bullseye-slim AS gitops-tools-builder

ENV HELM_SECRETS_VERSION=4.6.2
ENV SOPS_VERSION=3.9.0
ENV KUBECTL_VERSION=1.29.2
ENV VALS_VERSION=0.37.8


RUN apt-get update && apt-get install -y wget tar && rm -rf /var/lib/apt/lists/*
RUN ARCH=$(uname -m | sed -e 's/x86_64/amd64/' -e 's/aarch64/arm64/') \
    && mkdir -p /gitops-tools/helm-plugins \
    && wget -qO- --no-check-certificate https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /gitops-tools/helm-plugins -xzf- \
    && wget -qO /gitops-tools/sops --no-check-certificate https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.${ARCH} \
    && wget -qO /gitops-tools/kubectl --no-check-certificate https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${ARCH}/kubectl \
    && wget -qO- --no-check-certificate https://github.com/variantdev/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_${ARCH}.tar.gz | tar -xzf- -C /gitops-tools/ \
    && chmod +x /gitops-tools/*

# Stage 2: Build ArgoCD-based image
FROM {some ECR repo}:argocd-2.12.3

COPY --from=gitops-tools-builder /gitops-tools /gitops-tools
ENV PATH="/gitops-tools:$PATH"

[jkroepke]

The Dockerfile seems incomplete. At least all env variables are not defined, only. PATH

[chuyuou]

The Dockerfile seems incomplete. At least all env variables are not defined, only. PATH

Oh apologies, the env for argocd-repo-server is as:

        env:
          - name: HELM_PLUGINS
            value: /gitops-tools/helm-plugins/
          - name: HELM_SECRETS_SOPS_PATH
            value: /gitops-tools/sops
          - name: HELM_SECRETS_VALS_PATH
            value: /gitops-tools/vals
          - name: HELM_SECRETS_KUBECTL_PATH
            value: /gitops-tools/kubectl
          - name: HELM_SECRETS_CURL_PATH
            value: /gitops-tools/curl
          - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
            value: "true"
          - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
            value: "true"
          - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
            value: "true"
          - name: SOPS_AGE_KEY_FILE # For age,need to specify this to enable local
            value: /helm-secrets-private-keys/key.txt

[jkroepke]

Does this command works on argocd-repo server?

helm template oci://ghcr.io/jkroepke/charts/values:1.0.5 --set-file='hello=secrets+literal://vals!ref+echo://world'

if not, please re-run this with --debug:

helm template --debug oci://ghcr.io/jkroepke/charts/values:1.0.5 --set-file='hello=secrets+literal://vals!ref+echo://world'

[chuyuou]

Does this command works on argocd-repo server?

helm template oci://ghcr.io/jkroepke/charts/values:1.0.5 --set-file='hello=secrets+literal://vals!ref+echo://world'

if not, please re-run this with --debug:

helm template --debug oci://ghcr.io/jkroepke/charts/values:1.0.5 --set-file='hello=secrets+literal://vals!ref+echo://world'

I got this:

Thanks very much for looking into!

[jkroepke]

uhm, it looks like a old helm version. I tested the command locally, and at least they dont produce a helm error. not sure what the helm version is.

[chuyuou]

uhm, it looks like a old helm version. I tested the command locally, and at least they dont produce a helm error. not sure what the helm version is.

It's 3.15.2

$ helm version
version.BuildInfo{Version:"v3.15.2", GitCommit:"1a500d5625419a524fdae4b33de351cc4f58ec35", GitTreeState:"clean", GoVersion:"go1.22.4"}

[jkroepke]

Could you please run this one?

helm template --repo https://jkroepke.github.io/helm-charts/ values  --set-file='hello=secrets+literal://vals!ref+echo://world'

[chuyuou]

Could you please run this one?

helm template --repo https://jkroepke.github.io/helm-charts/ values  --set-file='hello=secrets+literal://vals!ref+echo://world'

I got this:

[chuyuou]

Could you advise some good versions for the tools? Currently I am using this combination:

Helm Version:3.15.2
Helm Secrets Version: 4.6.2
SOPS version:3.9.0
ArgoCD Version:2.12.3

I will test whether it could be fixed.

Thanks! @jkroepke