FluxCD bootstrap from Github failed

[duj4]

I was trying to bootstrap Flux from Github with command below:
flux bootstrap github --owner=$GITHUB_USER --repository=flux-fleet-infra --branch=main --path=clusters/homelab --personal --token-auth

However it stuck at:

► connecting to github.com
► cloning branch "main" from Git repository "https://github.com/duj4/flux-fleet-infra.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled

From kubectl describe gitrepository flux-system -n flux-system:

Name:         flux-system
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>
API Version:  source.toolkit.fluxcd.io/v1
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2025-02-08T05:29:17Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  4875357
  UID:               f5a76838-1497-40f2-bedc-0571486fbf64
Spec:
  Interval:  1m0s
  Ref:
    Branch:  main
  Secret Ref:
    Name:   flux-system
  Timeout:  60s
  URL:      https://github.com/duj4/flux-fleet-infra.git
Status:
  Conditions:
    Last Transition Time:  2025-02-08T05:29:43Z
    Message:               building artifact
    Observed Generation:   1
    Reason:                Progressing
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2025-02-08T05:29:43Z
    Message:               building artifact
    Observed Generation:   1
    Reason:                Progressing
    Status:                Unknown
    Type:                  Ready
    Last Transition Time:  2025-02-08T05:29:42Z
    Message:               failed to checkout and determine revision: unable to clone 'https://github.com/duj4/flux-fleet-infra.git': Get "https://github.com/duj4/flux-fleet-infra.git/info/refs?service=git-upload-pack": dial tcp 20.205.243.166:443: connect: connection refused
    Observed Generation:   1
    Reason:                GitOperationFailed
    Status:                True
    Type:                  FetchFailed
  Observed Generation:     -1
Events:
  Type     Reason              Age   From               Message
  ----     ------              ----  ----               -------
  Warning  GitOperationFailed  18s   source-controller  failed to checkout and determine revision: unable to clone 'https://github.com/duj4/flux-fleet-infra.git': Get "https://github.com/duj4/flux-fleet-infra.git/info/refs?service=git-upload-pack": dial tcp 20.205.243.166:443: connect: connection refused

Some checks I have done:

[dujas@k8s-master01 ~]$ nslookup github.com
Server:         192.168.126.2
Address:        192.168.126.2#53

Non-authoritative answer:
Name:   github.com
Address: 20.205.243.166
[dujas@k8s-master01 ~]$ curl -I https://github.com
HTTP/1.1 200 Connection established

HTTP/2 200
server: GitHub.com
date: Sat, 08 Feb 2025 06:12:29 GMT
[dujas@k8s-master01 ~]$ ssh -Tv -p 443 [email protected]
OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to github.com [20.205.243.166] port 443.
debug1: connect to address 20.205.243.166 port 443: Connection refused
ssh: connect to host github.com port 443: Connection refused

After adding ~/.ssh/config with config as below:

Host github.com
    Hostname ssh.github.com
    Port 443

Github connection test passed but the IP is different:

[dujas@k8s-master01 ~]$ ssh -Tv -p 443 [email protected]
...
debug1: Connecting to ssh.github.com [20.205.243.160] port 443.
debug1: Connection established.
...
Hi duj4! You've successfully authenticated, but GitHub does not provide shell access.

I changed the DNS mapping of github.com to another IP address in /etc/hosts, it is still trying to connect 20.205.243.166.

May I know what I could try next?

[stefanprodan]

To debug this I suggest running a pod with curl inside the flux-system namespace, my guess is that issue is with the CoreDNS forwarder.

[duj4]

To debug this I suggest running a pod with curl inside the flux-system namespace, my guess is that issue is with the CoreDNS forwarder.

Thanks @stefanprodan for the reply, it was working back to 1 month ago.

Besides, I am using proxy and no matter if it is configured in git config --global, still no luck.

Though I tried from another VM without K8S, still the same issue.

[duj4]

@stefanprodan I think I have fixed the issue by referring to the link: https://fluxcd.io/flux/installation/bootstrap/github/#bootstrap-without-a-github-pat, which is bootstrapping without PAT but private key, but there is a tiny difference on the url.

If I am using git@github,com, it will keep using the previous failed IP 20.205.243.166 and fail again:

[dujas@k8s-master01 flux-fleet-infra]$ flux bootstrap git --url=ssh://[email protected]/duj4/flux-fleet-infra.git --branch=main --private-key-file=/home/dujas/.ssh/id_ed25519 --path=clusters/homelab
✗ failed to create authentication options for ssh://[email protected]/duj4/flux-fleet-infra.git: SSH key scan for host github.com:22 failed, error: dial tcp 20.205.243.166:22: connect: connection refused

After switching it to [email protected], it passed:

[dujas@k8s-master01 flux-fleet-infra]$ flux bootstrap git --url=ssh://[email protected]/duj4/flux-fleet-infra.git --branch=main --private-key-file=/home/dujas/.ssh/id_ed25519 --path=clusters/homelab
► cloning branch "main" from Git repository "ssh://[email protected]/duj4/flux-fleet-infra.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
► installing components in "flux-system" namespace
✔ installed components
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: $public_key
Please give the key access to your repository: y
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for GitRepository "flux-system/flux-system" to be reconciled
✔ GitRepository reconciled successfully
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✔ Kustomization reconciled successfully
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy

I am not sure if this should be the correct one but per the link https://docs.github.com/en/authentication/troubleshooting-ssh/using-ssh-over-the-https-port, the test is based on [email protected]. Though I set the config in ~/.ssh/config as below, the bootstrap with [email protected] still failed with the same error as above:

Host github.com
    Hostname ssh.github.com
    Port 443
    User duj4