store DateTime as nanoseconds in doc store #2486

PSeitz · 2024-09-04T14:25:47Z

The doc store DateTime was truncated to microseconds previously. This
removes this truncation, while still keeping backwards compatibility.

This is done by passing the doc store version to the DocumentBinarySerializer and having different deserialization logic depending on the version for dates.

bump version format to 7.
add compat test to check the date time truncation.

closes #2380

fmassot · 2024-09-25T14:49:32Z

@fulmicoton can you review it?

src/store/footer.rs

src/store/reader.rs

common/src/serialize.rs

fulmicoton · 2024-09-26T07:18:00Z

src/schema/document/se.rs

-                    self.write_type_code(type_codes::HIERARCHICAL_FACET_CODE)?;
-
-                    Cow::Borrowed(val).serialize(self.writer)
+                    self.serialize_with_type_code(type_codes::DATE_CODE, &val)


I think we should just have some short date specific code here.

yes, I moved the version handling here

src/schema/document/de.rs

fulmicoton

Working on the DocumentBinarySerializer is a great idea. Introducing a ConfigurableBinarySerializer is overkill at this point however. Can we just push the logic on the date specific method in de.rs, se.rs.

The doc store DateTime was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. This is done by adding the trait `ConfigurableBinarySerializable`, which works like `BinarySerializable`, but with a config that allows de/serialize as different date time precision currently. bump version format to 7. add compat test to check the date time truncation.

* store DateTime as nanoseconds in doc store The doc store DateTime was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. This is done by adding the trait `ConfigurableBinarySerializable`, which works like `BinarySerializable`, but with a config that allows de/serialize as different date time precision currently. bump version format to 7. add compat test to check the date time truncation. * remove configurable binary serialize, add enum for doc store version * test doc store version ord

PSeitz force-pushed the doc_store_nano_prec branch from 65c366e to 18e548e Compare September 4, 2024 14:28

PSeitz requested a review from fulmicoton September 4, 2024 14:42