💥 Definition of objectives and metrics

[gmaze]

Following #39 (comment)

The Working Group aims to investigate few possible formats in order to guide the ADMT in assessing what is the best format to serve Argo data from the cloud.

To do so, we will perform tests providing metrics to evaluate each formats.

Note: Defining test conditions (e.g. who's running and from where and from which machine, linux/windows, cluster/laptop, python/matlab, etc...) should come in another discussion thread.

Here is a few definitions we need to settle on:

Definition of objectives

The objective of the tests are to evaluate data formats in how they improve:

• Performances and efficiency
• User experience
• Functionalities

Definition of metrics

Metrics can be quantitative as well as qualitative.
They will be gathered into a simple tabular form to be presented to the next ADMT.

Metrics for Performance and efficiency

Given a clear definition of what should be accessed, the "Access Patterns" (e.g.: data for one or more floats, data for a given space/time region) we shall gather metrics in the 2 use cases:

• serial data access (legacy way),
• parallel data access, (cloud infra. core added value)

Possible metrics (1):

• Time to perform the operation from a computer, fetching the data across the Internet. This simulates non-cloud usage.
• Time to perform the operation from a cloud infra. This simulates cloud-based usage.
• Number of http requests to cloud infra. made in performing the operation, which has both a cost and performance impact.
• Number of bytes transferred from cloud infra. in performing the operation, which has both a cost and performance impact.

Other metrics:

• change in file size compared to legacy netcdf,
• support for granular data access (e.g.: is it possible and how complex it is to download a single variable or meta-data, i.e. subsets of the raw data can be fetched and processed much faster than downloading files)
• ?

Metrics for User experience

• low level support by main programming languages (python, matlab, R, julia)
• availability of high level libraries supporting the file format as a remote data storage backend
• community size (define the community using the format and its ability to support Argo)
• level of disruption (describe how "close" is the new format to the Netcdf legacy format)
• ?

Functionalities

List of new functionalities not supported by the Netcdf legacy format, e.g.:

• support chunking and/or pagination
• highly granular direct data access (e.g. possibility to download a single variable like TEMP without downloading the complete cycle data ?)
• lazy access (gather meta-data to build a data-model without transferring data values before really needed)
• ?

--
(1) Metrics inspired by work reported at #44