Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write files to new container #233

Merged
merged 5 commits into from
Aug 2, 2023
Merged

Write files to new container #233

merged 5 commits into from
Aug 2, 2023

Conversation

bamader
Copy link
Collaborator

@bamader bamader commented Aug 1, 2023

This PR modifies the analytics notebook in a couple ways:

  1. it handles an edge case in the event a record would require blocking on all empty values (we just skip that pass of an algorithm for that record)
  2. it creates a new container in the storage account for the analytics notebook outputs (via terraform)
  3. it writes the statistical evaluation of the notebook to that new container
  4. it recomputes the log-odds weights of the algorithm based on the EMC-identified training data and writes the updated values to a file in the new container (note that these updated values are not used in the current evaluation run; it calculates them afterward and saves them; also, changes to these values will require a change to the cutoff scores of each pass of the DIBBs algorithm--currently that function is implemented only graphically, requiring a user to plot the cutoffs and choose the inflection point themselves, but we should be able to easily write a ticket in the future that can computationally figure out where the elbow is)

@bamader bamader temporarily deployed to main August 1, 2023 21:02 — with GitHub Actions Inactive
@bamader bamader temporarily deployed to main August 1, 2023 21:05 — with GitHub Actions Inactive
@bamader bamader temporarily deployed to main August 1, 2023 21:16 — with GitHub Actions Inactive
Copy link
Collaborator

@DanPaseltiner DanPaseltiner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@bamader bamader merged commit 6ace10b into main Aug 2, 2023
9 checks passed
@bamader bamader deleted the nb-file-write branch August 2, 2023 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants