The cluster request failed for an unknown reason: unable to verify the first certificate

[vitofico]

Type: Bug

Context

• Managed Dependencies: on
• Cloud SDK Version: 431.0.0
• Skaffold Version: v2.3.0
• Minikube Version: 1.30.1

Add a google cluster and try to connect to it through the extension. It throws the error: The cluster request failed for an unknown reason: unable to verify the first certificate

Extension version: 1.21.5
VS Code version: Code 1.78.2 (b3e4e68a0bc097f0ae7907b217c1119af9e03435, 2023-05-10T15:00:40.428Z)
OS version: Darwin arm64 22.4.0
Modes:
Sandboxed: Yes

System Info

Item	Value
CPUs	Apple M2 (8 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	2, 3, 3
Memory (System)	16.00GB (4.34GB free)
Process Argv	--crash-reporter-id 559a707e-7182-4a65-8e77-d968dbf7d7ba
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
282f8724:30602487
pyind779:30671433
f6dab269:30613381
pythonsymbol12:30671437
2i9eh265:30646982
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
pythonfmttext:30731395
fixshowwlkth:30730052
showsbindicator:30730056
pythongtdpathcf:30739705

[davidponder]

Hello!

Do you see the same error when attempting to run kubectl get n?

[vitofico]

no, kubectl is working just fine. I can operate the cluster normally, the problem is with the cloud code extension. Also, the error does not appear with a local kind cluster.

[SKrupa]

Hi, I have a couple more questions: Were these clusters added to the kubeconfig manually or via the extension? If they were added manually could you try adding via the "+" button in the Kuberenetes explorer?

Also for the certificate-authority-data field on your cluster, do you see a hash value like LS0... or something else? Thanks!

[vitofico]

Hello! Thank you for the support!
Yes, I tested both methods. Additionally I did another test by clearing the kubeconfig file and adding them again, all without success.
Regarding the certification authority, yes they start with LS0:

clusters:
  - cluster:
      certificate-authority-data: >-
        LS0tLS1CRUdJTiBD...

[motus9-rl]

+1

[SKrupa]

I'm still investigating the issue, but I've not been able to reproduce the issue such that kubectl can still connect but the extension cannot.

Do you maybe have some necessary certificates configured outside of the vscode environment? If you have managed dependencies turned on ("cloudcode.autoDependencies": "on" in the vscode settings), the extension will be using a different instance of kubectl than what you have pre-installed; which might be contributing to this issue. You could try toggling that to "off" to see if it helps.

Lastly, are there any special settings on the clusters you are trying to connect to? Or would a freshly made cluster with default settings still fail with this error?

Thanks!

[vitofico]

Hello!

In order:
1 - I tried to toggle on and off the cloudcode.autoDependencies but this not helped with the issue
2 - No special settings, I tried with both freshly made and existing gke servers. I previously mentioned that the extension works correctly with a local kind cluster.

Thank you!

[SKrupa]

Thanks for testing that out! And it sounds like these aren't private clusters either (https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept); Cloud Code doesn't support private clusters at this time.

Are you using the default kubeconfig (HOME/.kube/config) or some other kube file? Can you confirm by comparing the output of kubectl config view to what you see if you click the "Open File" hover-over button on the kubeconfig in the KUBERNETES tab of the extension? Also do you have the $KUBECONFIG environment variable set?

Could you also run
kubectl config view --raw -o go-template='{{index ((index (index .clusters 0) "cluster")) "certificate-authority-data"|base64decode}}'
and compare the certificate there to what is shown in cloud console? (There is a 'Show cluster certificate' button in Kuberenetes Engine->Clusters->{your cluster}->Details->Cluster Basics->External Endpoint)

I'm still trying to get a repro on my side, but I really appreciate you trying these things out to help debug the issue in the meantime.

[vitofico]

Thank you very much for your help!

1 - The kubeconfig is the default one. The $KUBECONFIG env variable is set.
2 - I can not compare the certificates, the access through Cloud Code Extension is blocked by the error of this issue. This is the result of clicking on the cluster from the extension:

btw, I can correctly access the cluster from the VSCode Kubernetes Extension.

[SKrupa]

You can access the kubeconfig from this button here, so you should be able to load it without access to the cluster. (Its the box with the arrow pointing to the top right.)

Is the $KUBECONFIG variable set to HOME/.kube/config, or something else? If its something else, we might be running into a problem when its merging them.

[SKrupa]

Oh and for the comparing certificates, the kubectl config view --raw -o go-template='{{index ((index (index .clusters 0) "cluster")) "certificate-authority-data"|base64decode}}' command you can run outside of the extension, and then you can compare that to certificate in cloud.google.com.

Could you also check if NODE_EXTRA_CA_CERTS environment variable is set? That might also be conflicting.

[vitofico]

• Sorry for not being accurate in my previous answer. Yes the $KUBECONFIG is set to HOME/.kube/config.
• The certificates shown by kubectl config view --raw -o go-template='{{index ((index (index .clusters 0) "cluster")) "certificate-authority-data"|base64decode}}' and through 'Show cluster certificate' button in Kuberenetes Engine->Clusters->{your cluster}->Details->Cluster Basics->External Endpoint are the same
• The certificate shown by accessing kubeconfig file through the cloud extension is different from the previous ones
• $NODE_EXTRA_CA_CERTS is not set

[conghai]

+1

[j-windsor]

Another workaround could be to set insecure-skip-tls-verify on your cluster in your kubeconfig, although that could make your control plane connection insecure.

clusters:
  - cluster:
      insecure-skip-tls-verify: true

Are you trying to access you in a corporate network or other place where you might be subject to a TLS decrypting firewall? Do other TLS connections on your machine rely on a custom certificate installed in Keychain Access? If kubectl works, this is very likely the case. kubectl is written in golang which grabs all certificates from Keychain Access. NodeJS (which VS Code extensions are written in) instead keeps a static list of root certificates.

If so, a workaround would be to export that certificate as a PEM file and set NODE_EXTRA_CA_CERTS. This is unfortunately the only way to extend the static list of certificates bundled with NodeJS. See nodejs/node#39657 for the latest issue discussing this in detail.

[bgeesaman]

That workaround was helpful. Thank you @j-windsor

$ cat ~/.kube/config
...snip...
- cluster:
    certificate-authority-data: LS0tL...   <--- copy the entire base64 encoded certificate string from here
    server: https://gke.cluster.ip.here
  name: gke_CLUSTERNAMEHERE
...snip...

Copied the entire value of certificate-authority-data and ran echo 'LS0tL.....several hundred characters here.......' | base64 -d > gkecerts.pem. Then ran export NODE_EXTRA_CA_CERTS=~/gkecerts.pem. Restarted VSCode. Success.

[vitofico]

Hi! thanks for providing a detailed solution.
Unfortunately it is not working for me. In my case, every time I restart VSCode, NODE_EXTRA_CA_CERTS variable is unset.

[j-windsor]

Make sure you are setting the environment variable in a way that VS Code can access it. I would recommend adding it to your default shell .rc file so it is available everywhere you could open VS Code.

I would also recommend using a certificate from a trusted CA, likely from your enterprise in this case.

I've never used it before, but this also looks promising. An extension that loads all certificates on your mac into VS Code like kubectl does.

[j-windsor]

Closing this issue, as there is unfortunately nothing we can do about this on our side.

[j-windsor]

Please see #812 (comment) and see if that works for you. I believe the real solution to this problem is to disable VS Code's overriding of HTTP request options that include the CA cert from the KubeConfig.

[vitofico]

This last turned out to be the solution! Thank you @j-windsor