DM-46588: Configure connection limits and instrument configs for fan-out #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kfindeisen
force-pushed
the
tickets/DM-46588
branch
from
222de21 to
b80372f
Compare
The previous code would leave a request marked as "in process" even if the service, for example, timed out waiting for a response. This introduced a permanent offset into the metric that could only be cleared by rebooting the fan-out service. The new code is exception-safe and should not have this problem.
Grouping the Gauges together, and indexing them by instrument name, lets us remove a large amount of boilerplate code in both initializing the Gauges and passing them around.
This commit begins the process of removing specific instrument references from the source code. However, some references still remain.
The configuration is now handled in Phalanx, and can be changed without rebuilding the container.
The old format was called a "detector" config despite having room for more fields. Calling it an instrument config makes it more natural to add other instrument-specific information to the file.
kfindeisen
force-pushed
the
tickets/DM-46588
branch
2 times, most recently
from
38984e7 to
8d93f83
Compare
The original config was organized instrument->detectors->detector, which required the entire config to be overridden if only one per-instrument property was environment-dependent. The new config is organized detectors->instrument->detector, which allows non-detector configs to be changed without touching or duplicating the detectors.
kfindeisen
force-pushed
the
tickets/DM-46588
branch
3 times, most recently
from
b210ced to
c83bbd1
Compare
This consolidates the instrument-specific configs in one place.
Grouping the detectors and URLs together, and indexing them by instrument name, lets us remove most instrument-specific branches. The exceptions are the HSC Cosmos mini-datasets, which still need special treatment.
These datasets require specific detector combinations, but these combinations are uniquely identifiable by visit ID.
Coordinates are pairs of floats, not ints, and not all values are strings.
Collecting the target Knative URL and the fanned-out messages into a single object makes it easy to factor out the shared code.
The HSC Cosmos datasets still need special casing, but the relevant function and switch branch can be simply deleted once HSC support is phased out.
The old add_detectors was an instance method implemented like a static method. This commit changes it to a proper instance method, since that is the existing usage pattern.
kfindeisen
force-pushed
the
tickets/DM-46588
branch
from
c83bbd1 to
2c647b3
Compare
This change makes the loop much easier to read and edit. However, the functions need to take an inordinate number of "global" variables, so they could use some further factoring.
The original behavior was to crash and restart the entire fan-out service if it got a message it didn't know how to handle. If the message loop instead catches the corresponding exception, then there is no need to specifically exempt the LSST cameras. With this commit, it's now safe to configure the SUPPORTED_INSTRUMENTS env variable.
This makes it possible to change logging levels from the configuration. This commit also replaces the double `logging.basicConfig` calls of the original code; `basicConfig` only has an effect the first time it's called.
kfindeisen
force-pushed
the
tickets/DM-46588
branch
from
2c647b3 to
0694f60
Compare
kfindeisen
changed the title
kfindeisen
changed the title
The code previously used the default of 100, which is much less than the number of concurrent images we will need to handle with LSSTCam.
kfindeisen
force-pushed
the
tickets/DM-46588
branch
from
0694f60 to
d6b361a
Compare
erinleighh
approved these changes
Oct 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes the fan-out service's supported instruments and detectors configurable, and makes the maximum number of outgoing connections tunable. It also performs some refactoring to minimize code duplication and special-case variables.