Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simulation-based testing #14

Draft
wants to merge 13 commits into
base: main
Choose a base branch
from
Draft

Simulation-based testing #14

wants to merge 13 commits into from

Conversation

afmagee42
Copy link
Collaborator

⚠️ Work in progress ⚠️

This PR adds simulation-based tests of performance as package data to provide users context on how well inference works in some parameter regimes of interest (and one or two of perhaps less interest, with larger $R$ and $k$). Inference summaries are stored in the form of point (posterior median, maximum likelihood) and (95% credible and confidence) interval estimates.

An in-development vignette uses this data to provides a sense of coverage and bias in practice when using the model. (It doesn't cover cases where the model is misspecified, but that universe is far larger and out of scope.)

To do all this, this PR adds a make-based workflow to data-raw which can be run to simulate across a grid of $R$, $k$, and dataset size, as well as analyze with both Bayesian inference and maximum likelihood (both under default settings). The use of make and bash scripts which exist only to call R may not be strictly necessary. However, previous attempts at a purely R-based workflow resulted in a variety of failures, while this appears to be working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant