Appetite for abstraction to other build environments

[Nava2]

We use non-Github Actions CI pipelines for our build systems. We'd love to be able to contribute and leverage the existing tooling within the Gradle GitHub actions within other CI tools.

I wonder if there is appetite to refactor components of the GitHub actions to be published as npm packages and abstract the GitHub actions aspects away such that the common code can be reused within other environments.

I'd be willing to take on some of this refactoring as we would be able to reuse caching particularly in our environments!

[bigdaz]

@Nava2 Thanks for your interest in the project. Indeed it could be useful to be able to apply similar logic to other CI platforms. However, there are features of GitHub Actions that we depend on that mean this isn't so simple.

I can see value in abstracting away from the direct use of GitHub APIs for logging and other features, but before commencing this project I think it would be helpful to consider some of the core features that we rely on in GitHub Actions.

A primary use case for setup-gradle is to save and restore content in Gradle User Home. The current implementation leverages some properties of the GitHub Actions cache that are likely not present in other systems.

So as a first step to making the setup-gradle action applicable in other environments, I think it would be good to consider how we could adopt the caching mechanism to leverage other storage backends, like Amazon S3. This would be very useful for existing users of GitHub Actions, and would likely ease the complete migration of setup-gradle to other CI platforms.

Are you interested in helping push things forward in that direction?

[Nava2]

@Nava2 Thanks for your interest in the project. Indeed it could be useful to be able to apply similar logic to other CI platforms. However, there are features of GitHub Actions that we depend on that mean this isn't so simple.

I can see value in abstracting away from the direct use of GitHub APIs for logging and other features, but before commencing this project I think it would be helpful to consider some of the core features that we rely on in GitHub Actions.

Is there specific ones of concern? Reimplementing the cache save/restore is definitely more effort but doesn't seem particularly sophisticated in my very ignorant and open minded view 😅

A primary use case for setup-gradle is to save and restore content in Gradle User Home. The current implementation leverages some properties of the GitHub Actions cache that are likely not present in other systems.

Which features in particular are you referencing? I think with enough elbow grease anything could technically be implemented.

So as a first step to making the setup-gradle action applicable in other environments, I think it would be good to consider how we could adopt the caching mechanism to leverage other storage backends, like Amazon S3. This would be very useful for existing users of GitHub Actions, and would likely ease the complete migration of setup-gradle to other CI platforms.

We're exactly aligned. With a proper abstraction layer in place, this seems like a reasonable thing to try and implement. Without a layer, it gets tough.

Are you interested in helping push things forward in that direction?

Absolutely, I've been chipping away at abstracting the environment in a fork for awhile now. It's not too difficult to do easily without impacting the GitHub actions version.

[Nava2]

https://github.com/Nava2/gradle-actions/pull/18/files here is an example of some abstraction

[bigdaz]

Which features in particular are you referencing? I think with enough elbow grease anything could technically be implemented.

some properties of the GitHub Actions cache

Implementing this won't be a simple matter of adapting to a different API, but will require re-design and re-implementation of some of the features.

I'd like to have a plan for implementing caching on a regular key-value store before investing much effort into the simpler process of abstraction away basic APIs. And if we have a plan then we could implement something on top of S3 (or another store): this would be hugely beneficial to current users of setup-gradle, since the GHA implementation can be quite limited.

[Nava2]

Which features in particular are you referencing? I think with enough elbow grease anything could technically be implemented.

some properties of the GitHub Actions cache

Implementing this won't be a simple matter of adapting to a different API, but will require re-design and re-implementation of some of the features.

I'd like to have a plan for implementing caching on a regular key-value store before investing much effort into the simpler process of abstraction away basic APIs. And if we have a plan then we could implement something on top of S3 (or another store): this would be hugely beneficial to current users of setup-gradle, since the GHA implementation can be quite limited.

Ah, apologies that original link wasn't directly linking on mobile. I see where you're coming from.

It seems like for each case listed there's some possibilities:

• Immutable entries: once a cache entry is written for a key, it cannot be overwritten or changed.

Pardon my ignorance, but this seems like a good thing for consistency purposes 😅

• Branch scope: cache entries written for a Git branch are not visible from actions running against different branches or tags. Entries written for the default branch are visible to all. https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

This seems a bit more tricky to implement and a little surprising. Though, could also decide to abstract and avoid this feature.

• Restore keys: if no exact match is found, a set of partial keys can be provided that will match by cache key prefix. https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

Seems like some providers have this support:

• Amazon S3
• R2 claims to support the S3 API
• OCI
• GCP

Seems reasonable -- more than I even expected 🤔

All in, I'm aligned that a proper design would be beneficial to ensuring we're iterating with a purpose.

[Starlight220]

Docker's build cache supports a variety of these backends, perhaps something there can be used as a reference.