diff --git a/keps/prod-readiness/sig-storage/5040.yaml b/keps/prod-readiness/sig-storage/5040.yaml new file mode 100644 index 00000000000..60744add49e --- /dev/null +++ b/keps/prod-readiness/sig-storage/5040.yaml @@ -0,0 +1,6 @@ +# The KEP must have an approver from the +# "prod-readiness-approvers" group +# of http://git.k8s.io/enhancements/OWNERS_ALIASES +kep-number: 5040 +alpha: + approver: "@jpbetz" diff --git a/keps/sig-storage/5040-remove-gitrepo-driver/README.md b/keps/sig-storage/5040-remove-gitrepo-driver/README.md new file mode 100644 index 00000000000..da5213d06bf --- /dev/null +++ b/keps/sig-storage/5040-remove-gitrepo-driver/README.md @@ -0,0 +1,1047 @@ + +# KEP-5040: Remove gitRepo volume driver. + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Validating Admission Policy](#validating-admission-policy) + - [Timeline](#timeline) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Disabled](#disabled) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + - [Alternative 1: Add admission plugin that blocks gitRepo volumes](#alternative-1-add-admission-plugin-that-blocks-gitrepo-volumes) + - [Alternative 2: Use ValidatingAdmissionPolicy](#alternative-2-use-validatingadmissionpolicy) + - [Alternative 3: Make Git hooks disableable](#alternative-3-make-git-hooks-disableable) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +We propose removing support for the in-tree gitRepo volume driver. We aren't +proposing removing gitVOlumes from the Kubernetes API, meaning pods with gitRepo +volumes will be admitted by kube-apiserver but kubelets with the feature-gate +GitRepoVolumeDriver set to false will not run them and return an appropriate +error to the user. + +We acknowledge that this is highly unusual, but there are mitigating circumstances: +* gitRepo volume types can be exploited to gain remote code execution as root on the nodes as shown in [CVE-2024-10220](https://nvd.nist.gov/vuln/detail/cve-2024-10220) +* gitRepo has perfectly workable alternatives: + * Using [git-sync](https://github.com/kubernetes/git-sync) + * Using an initContainer. See https://kubernetes.io/docs/concepts/storage/volumes/#gitrepo for more details +* gitRepo has been deprecated for a long time and is unmaintained +* gitRepo has low usage (based on limited data) +* there exists precedent for removing volume drivers which were unmaintained like [glusterfs](https://kubernetes.io/docs/concepts/storage/volumes/#glusterfs) + +Given this, we think kubernetes should EOL this feature with urgency. + +## Motivation + + + +Remove a vector for remote code execution through an ancient and unmaintained +volume plugin. + +SIG Storage was actually very close to doing this and had even added a notice of +removal in the release notes for v1.32. But due to concerns raised by SIG Architecture +they rolled back the notice. SIG Storage provided [this rationale](https://github.com/kubernetes/kubernetes/issues/125983#issuecomment-2522201283) for considering removal. + +### Goals + + + +- Remove in-tree gitRepo volume support. + +### Non-Goals + + +- Remove gitRepo volumes from the Kubernetes API. + +## Proposal + +Kubernetes drops support for gitRepo volume types by removing the in-tree driver +that actually mounts the volume in the pod. In this case we will still keep the +gitRepo volume in the API. Pods with gitRepo volumes will be admitted by +kube-apiserver however, kubelet will fail to run them with an appropriate +error message. + + +This removal is similar to removal of support for +[glusterfs](https://kubernetes.io/docs/concepts/storage/volumes/#glusterfs) +which was deprecated in v1.25 and removed in v1.26. + +### Risks and Mitigations + + + +The biggest risk is in breaking users. Despite being deprecated for years, there +may be some users who are still using it and their workloads would be broken +when their clusters are upgraded. + +We plan to mitigate the risk by: +1. Allowing users to opt-in to re-enabling the driver for 3 versions to give + them enough time to fix workloads +2. Announcing the change via release notes, updates to the gitRepo documentation + and publishing kubernetes release blog for this feature so that customers are + aware and can take action before they upgrade + +## Design Details + + + +We will add a new feature-gate to kubelet called `GitRepoVolumeDriver`. This +feature-gate will be defaulted to `false`, which will disable the +[gitRepo driver][gitRepo driver](https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/git_repo/git_repo.go). +If a workload with gitRepo volume is encountered, kubelet will fail to run such a +pod with an appropriate error message guiding users to either use an alternative +or setting the feature-gate `GitRepoVolumeDriver` to `true`. + +### Validating Admission Policy + +Users can prevent workloads with gitRepo volumes from being admitted +using the following Validating Admission Policy (VAP). + +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: "no-gitrepo-volumes" +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["pods"] + - apiGroups: ["apps"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["deployments", "replicasets", "statefulsets", "daemonsets"] + - apiGroups: ["batch"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE"] + resources: ["jobs", "cronjobs"] + validations: + - expression: |- + object.kind != 'Pod' || + !object.spec.volumes.exists(v, has(v.gitRepo)) + message: "gitRepo volumes are not allowed." + - expression: |- + !['Deployment','ReplicaSet','DaemonSet','StatefulSet','Job'].exists(kind, object.kind == kind) || + !object.spec.template.spec.?volumes.orValue([]).exists(v, has(v.gitRepo)) + message: "gitRepo volumes are not allowed." + - expression: |- + object.kind != 'CronJob' || + !object.spec.jobTemplate.spec.template.spec.?volumes.orValue([]).exists(v, !has(v.csi)) + message: "gitRepo volumes are not allowed." +``` + +Users can using the following ValidatingAdmissionPolicyBinding to apply the +ValidatingAdmissionPolicy above to all namespaces: + +```yaml +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: "no-gitrepo-volumes" +spec: + policyName: "no-gitrepo-volumes" + validationActions: + - Deny +``` + +### Timeline +v1.33 (04/2025): +- We announce the removal in kubernetes release notes, documentation and via +kubernetes release blog post. +- We add the `GitRepoVolumeDriver` feature-gate to kubelet and +default it to `false`. + +v1.34 (08/2025): +- No change. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + +v1.35 (11/2025): +- No change. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + +v1.36 (04/2026): +- Lock the feature-gate, i.e. users cannot change its value. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + +v1.37 (08/2026): +- No change. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + +v1.38 (11/2026): +- No change. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + +v1.39 (04/2027): +- We will remove the feature-gate. +- We will remove the gitRepo driver code. +- We will announce removal in kubernetes release notes, documentation and via +kubernetes release blog post. + + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + + + +We will add the following unit tests to the [git_repo](https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/git_repo/git_repo_test.go) +pacakage: + +- When the feature-gate GitRepoVolumeDriver is false kubelet returns an error + for pods that have gitRepo volumes in them +- When the feature-gate GitRepoVolumeDriver is false kubelet does not return an + error for pods that have gitRepo volumes in them +- When the feature-gate GitRepoVolumeDriver is true kubelet does not returns an + error for pods that have gitRepo volumes in them + +##### Integration tests + + + + + +- : + +##### e2e tests + + + +We will add the following e2e tests to the [e2e_node](https://github.com/kubernetes/kubernetes/tree/master/test/e2e_node) +pacakage: + +- When the feature-gate GitRepoVolumeDriver is false kubelet returns an error + for pods that have gitRepo volumes in them +- When the feature-gate GitRepoVolumeDriver is false kubelet does not return an + error for pods that have gitRepo volumes in them +- When the feature-gate GitRepoVolumeDriver is true kubelet does not returns an + error for pods that have gitRepo volumes in them + +### Graduation Criteria + + + +#### Disabled + +Please see the proposed timelines for this removal in the +[Design Details](#design-details) section. To move from the disabled to the +removed stage the following criteria needs to be met: + +- All code and tests to disable gitRepo volume driver by default is implemented +- Two versions have passed since introducing the default disablement of gitRepo volume driver (to address version skew) +- All feedback on usage/changed behavior, provided on GitHub issues has been addressed + +### Upgrade / Downgrade Strategy + + +- If you are upgrading a cluster with no pods that have gitRepo volume then no +action is required +- If you are upgrading a cluster with pods that have gitRepo volume then there +are a few options: + - Enable the feature-gate GitRepoVolumeDriver + - Migrate the workloads to use [git-sync](https://github.com/kubernetes/git-sync) + - Migrate the workloads to use emptyDir + initContainer to sync the git repo + +### Version Skew Strategy + + + +We aren't chaning the Kubernetes API in regards to pods with gitRepo volumes +being admitted by kube-apiserver. These pods will still be admitted but kubelets +with GitRepoVolumeDriver=false will not run them and return an appropriate error +to the user. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - GitRepoVolumeDriver + - Components depending on the feature gate: + - kubelet +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? + +###### Does enabling the feature change any default behavior? + + + +Yes, gitRepo volume driver will be disabled. So pods with gitRepo volumes will +fail to run. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes. + +###### What happens if we reenable the feature if it was previously rolled back? + +gitRepo volume driver will not work anymore. + +###### Are there any tests for feature enablement/disablement? + + + +Yes, please see testing sections above. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +Operators can check if there are pods with gitRepo volumes in their cluster by +running the following command: + +```bash +kubectl get pods -A -o json \ +| jq -r '.items[] | select(.spec.volumes[]? | has("hostPath")) | .metadata.namespace + " " + .metadata.name' +``` + +Operators can check if the feature-gate `GitRepoVolumeDriver` is enabled on a +particular node by running the following command: + +```bash +kubectl get --raw "/api/v1/nodes//proxy/metrics" \ +| grep kubernetes_feature_enabled \ +| grep GitRepoVolumeDriver +``` + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +No. + +###### Will enabling / using this feature result in introducing new API types? + + + +No. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +No. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +No. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +No. + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +No. + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + +### Alternative 1: Add admission plugin that blocks gitRepo volumes +We add an in-tree admission plugin that, when enabled, will prevent Pods with +gitRepo volumes from being admitted. + +Pros: +- Backwards compatible. +- Allows cloud providers to make a call if they want to support gitRepo volumes or not. + +Cons: +- Not secure by default, requires cluster-admin action. +- The only escape valve is cluster-scoped. +- We'll be adding an in-tree admission plugin for a volume type that is not + maintained. + +### Alternative 2: Use ValidatingAdmissionPolicy +We publish a VAP which prevents Pods and other workloads from using gitRepo, and +strongly recommend cluster-admins (or cluster-providers) install it. + +Pros: +- Backwards compatible. +- Allows cluster-admins to make a call if they want to support gitRepo volumes or not. +- Can be scoped to namespaces rather than whole clusters. +- Works on older clusters. + +Cons: +- Not secure by default +- Requires cluster-admin action. + +### Alternative 3: Make Git hooks disableable +We add a new field to gitRepo volume called `enableHooks` to +configure if Git hooks should be run as part of gitRepo volume mounting. +When this value is set to true kubelet will execute Git hooks when performing +`git clone` and `git checkout`. If enableHooks is unset or set to false, kubelet +will not run any Git hooks when performing `git clone` and `git checkout`. + +Pros: +- gitRepo volume type will still be supported and we will eliminate the risks. + +Cons: +- We are adding new functionality to a deprecated and unmaintained volume type + that has better alternatives which are not affected by the vulnerability. + + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-storage/5040-remove-gitrepo-driver/kep.yaml b/keps/sig-storage/5040-remove-gitrepo-driver/kep.yaml new file mode 100644 index 00000000000..b5dff3687fc --- /dev/null +++ b/keps/sig-storage/5040-remove-gitrepo-driver/kep.yaml @@ -0,0 +1,41 @@ +title: Remove gitRepo volumes driver. +kep-number: 5040 +authors: +- "@vinayakankugoyal" +owning-sig: sig-storage +participating-sigs: +status: implementable +creation-date: 2024-12-14 +reviewers: +- "@thockin" +- "@liggitt" +- "@saad-ali" +approvers: +- "@thockin" + +see-also: + +replaces: + +# The target maturity stage in the current dev cycle for this KEP. +stage: disabled + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.33" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.33" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: GitRepoVolumeDriver + components: + - kubelet +disable-supported: true + +# The following PRR answers are required at beta release +metrics: