Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cu): read message results from the file system, and optionally store them and hydrate them from a remote source #1108

Open
VinceJuliano opened this issue Jan 14, 2025 · 0 comments · May be fixed by #1140
Assignees

Comments

@VinceJuliano
Copy link
Collaborator

VinceJuliano commented Jan 14, 2025

Background

Currently, the CU stores its json results in sqlite or postgres. This is not optimal for large results especially when getting a list. Also when booting up a new CU and using it for the same processes, unless it is using postgres it will not have access to the previously cached results and will have to evaluate them.

Solution

  1. Create 2 environment variables EVALUATION_RESULTS_DIR and EVALUATION_RESULTS_BUCKET they will default to not being set and if they are set the feature is enabled.
  2. Create a hydrator worker modelled off of the evaluation worker. and compose it into ao-evaluation.js.
  3. ao-evaluation.js will ask hydrator for eval results, hydrator will then use EVALUATION_RESULTS_DIR and EVALUATION_RESULTS_BUCKET to try and retrieve cached results. ao-evaluation will use these in output.
  4. Current persistance in ao-evaluation will remain the same the hydrator will just hydrate and flush from the persistance in ao-evaluation.js
  5. EVALUATION_RESULTS_BUCKET will be a remote storage system like s3.
  6. hydrator will flush a single process or all processes (compose the single process function into the many one) upon receiving a signal.
  7. The CU will signal the hydrator upon receiving its own signal to flush a single or many processes.
@jfrain99 jfrain99 linked a pull request Feb 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants