#991 Support Relative Date Times #1006

currantw · 2025-01-03T00:35:54Z

Signed-off-by: currantw [email protected]

Description

Adds support for relative date times via the new user-defined method relative_datetime.

Related Issues

Resolves #991.

Check List

Updated documentation (docs/ppl-lang/README.md)
Implemented unit tests
Implemented tests for combination with other commands
New added source code should include a copyright header
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/TimeUtils.java

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

acarbonetto · 2025-01-03T17:14:53Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+        testValid("@w6", "2000-01-01T00:00");
+        testValid("@w7", "2000-01-02T00:00");
+
+        testInvalid("@INVALID", "The relative date time unit 'INVALID' is not supported.");


you will need to test a lot more invalid cases (such as "@W8") to satisfy implementation.

I would also like to see more complicated use cases such as
source=table | WHERE earliest=-5d@w1 AND latest=@w6 | ...
I remembered we will not be supporting this functionality in the first iteration - can you please add this test and mark it as ignore until we add this capability ...

@acarbonetto I have added this test case, as well as a few more invalid ones. If you have any more in mind, please let me know.

@YANG-DB. I have added ignored tests for earliest and latest in FlintSparkPPLBuiltInDateTimeFunctionITSuite. That being said, more complex integration tests would, I think, require mocking/overriding the current time in order to be stable. Let me know if you think this is something I should pursue.

I think you can leverage the timestampdiff to test more complex cases in IT:
timestampdiff(HOUR, relative_timestamp("+1d"), relative_timestamp("+1h"))
timestampdiff(HOUR, relative_timestamp("-1h@w3"), relative_timestamp("@d"))
And something else, they should be stable.

I've added some more complex IT tests along those lines - although I don't think timestampdiff(HOUR, relative_timestamp("-1h@w3"), relative_timestamp("@d")) would actually work, since the difference would depend on the current time and day of the week, right?

YANG-DB · 2025-01-04T19:57:53Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+        testValid("@w6", "2000-01-01T00:00");
+        testValid("@w7", "2000-01-02T00:00");
+
+        testInvalid("@INVALID", "The relative date time unit 'INVALID' is not supported.");


I would also like to see more complicated use cases such as
source=table | WHERE earliest=-5d@w1 AND latest=@w6 | ...
I remembered we will not be supporting this functionality in the first iteration - can you please add this test and mark it as ignore until we add this capability ...

YANG-DB

please also add such time based relative dates queries in the example commands doc

Add a new relative time query filter example to the ppl where clause documentation

currantw · 2025-01-09T22:08:55Z

please also add such time based relative dates queries in the example commands doc

Add a new relative time query filter example to the ppl where clause documentation

Done! Let me know if this is what you had in mind.

currantw · 2025-01-09T22:20:28Z

...ation/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLBuiltInDateTimeFunctionITSuite.scala

+  // TODO #957: Support earliest
+  ignore("test EARLIEST") {
+    var frame = sql(s"""
+                       | source = $testTable
+                       | | eval earliest_hour_before = earliest(now(), "-1h")
+                       | | eval earliest_now = earliest(now(), "now")
+                       | | eval earliest_hour_after = earliest(now(), "+1h")
+                       | | fields earliest_hour_before, earliest_now, earliest_hour_after
+                       | | head 1
+                       | """.stripMargin)
+    assertSameRows(Seq(Row(true), Row(true), Row(false)), frame)
+  }
+
+  // TODO #957: Support latest
+  ignore("test LATEST") {
+    var frame = sql(s"""
+                       | source = $testTable
+                       | | eval latest_hour_before = latest(now(), "-1h")
+                       | | eval latest_now = latest(now(), "now")
+                       | | eval latest_hour_after = latest(now(), "+1h")
+                       | | fields latest_hour_before, latest_now, latest_hour_after
+                       | | head 1
+                       | """.stripMargin)
+    assertSameRows(Seq(Row(false), Row(true), Row(true)), frame)
+  }


@acarbonetto Any idea on better tests for earliest and latest that don't require mocking the current time for testing? Ultimately, earliest and latest are really only wrappers around relative_timestamp function (earliest(field_name, "-1h@d") is equivalent to field_name >= relative_timestamp("-1h@d"), so it seems fine to me as long as we have pretty robust unit tests for relative timestamp. Thoughts?

tests are cheap - so it's okay to have duplicate tests.

Since IT tests are mostly focusing on testing the API and integration, I don't thinks that's overly valuable to test the backend logic. Leave that to Unit Tests where mocking is easily done.

If you need to mock the IT test backend, you're probably not doing testing correctly.

If you need to mock the IT test backend, you're probably not doing testing correctly.

IT tests have been updated, and don't rely on mocking. Let me know what you think!

build.sbt

docs/ppl-lang/functions/ppl-datetime.md

...spark-integration/src/main/java/org/opensearch/sql/ppl/utils/BuiltinFunctionTransformer.java

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/TimeUtils.java

LantaoJin · 2025-01-10T10:45:16Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+        testValid("@w6", "2000-01-01T00:00");
+        testValid("@w7", "2000-01-02T00:00");
+
+        testInvalid("@INVALID", "The relative date time unit 'INVALID' is not supported.");


I think you can leverage the timestampdiff to test more complex cases in IT:
timestampdiff(HOUR, relative_timestamp("+1d"), relative_timestamp("+1h"))
timestampdiff(HOUR, relative_timestamp("-1h@w3"), relative_timestamp("@d"))
And something else, they should be stable.

acarbonetto · 2025-01-13T22:34:34Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+
+    @Test
+    public void testRelativeOffsetValue() {
+        testValid("+h", "2000-01-03T02:01:01.100");


Thanks. Duplicate removed

acarbonetto · 2025-01-13T22:35:29Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+        testValid("+2wk", "2000-01-17T01:01:01.100");
+        testValid("-1h@W3", "1999-12-29T00:00:00");
+        testValid("@d", "2000-01-03T00:00");
+        testValid("now", "2000-01-03T01:01:01.100");


but @now is invalid, or +now.

Thanks. Added these to testRelativeOffsetUnit and testRelativeSnap, respectively.

acarbonetto · 2025-01-13T22:39:11Z

ppl-spark-integration/src/test/java/org/opensearch/sql/expression/function/TimeUtilsTest.java

+        testValid("+qtr", "2000-04-03T01:01:01.100");
+        testValid("+qtrs", "2000-04-03T01:01:01.100");
+        testValid("+quarter", "2000-04-03T01:01:01.100");
+        testValid("+quarters", "2000-04-03T01:01:01.100");


are there limits to numbers of any other types? Such as: can I do +5q?

Nope. Any integer should work. I've added another test case (-3d) to testRelativeOffsetValue to help make this more clear.

Signed-off-by: currantw <[email protected]>

…unit tests. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

…sponding unit tests in `SerializableTimeUdfTest`. Add `mockito-inline` to dependencies for `ppl-spark-integration` to allow mocking of current datetime. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

…st and latest. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

…eturned, but output from `$CurrentTimestamp` is an `Instant` Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

currantw commented Jan 3, 2025

View reviewed changes

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/TimeUtils.java Outdated Show resolved Hide resolved

acarbonetto reviewed Jan 3, 2025

View reviewed changes

YANG-DB requested changes Jan 4, 2025

View reviewed changes

YANG-DB reviewed Jan 5, 2025

View reviewed changes

currantw commented Jan 9, 2025

View reviewed changes

currantw marked this pull request as ready for review January 9, 2025 23:38

currantw requested review from dai-chen, mengweieric, penghuo, seankao-az, anirudha, kaituo, noCharger, LantaoJin and ykmr1224 as code owners January 9, 2025 23:38

acarbonetto reviewed Jan 9, 2025

View reviewed changes

LantaoJin reviewed Jan 10, 2025

View reviewed changes

acarbonetto approved these changes Jan 13, 2025

View reviewed changes

currantw added 11 commits January 13, 2025 16:21

Unrelated typos

bc6548b

Signed-off-by: currantw <[email protected]>

Initial implementation of relative date time logic and unit tests

1eb7f45

Signed-off-by: currantw <[email protected]>

Minor cleanup

fd526ad

Signed-off-by: currantw <[email protected]>

Update to make relative time case-insensitive, and add corresponding …

94e081c

…unit tests. Signed-off-by: currantw <[email protected]>

Add a few more unit test cases.

9d2406b

Signed-off-by: currantw <[email protected]>

Split pattern into smaller strings.

b4089a5

Signed-off-by: currantw <[email protected]>

Add relativeDateTimeFunction to SerializableUdf, along with corre…

3c2a5a8

…sponding unit tests in `SerializableTimeUdfTest`. Add `mockito-inline` to dependencies for `ppl-spark-integration` to allow mocking of current datetime. Signed-off-by: currantw <[email protected]>

Fix dangling Javadoc

73fd657

Signed-off-by: currantw <[email protected]>

Initial implementation of relative_timestamp UDF.

789c516

Signed-off-by: currantw <[email protected]>

Add integration tests, refactor to use Timestamp

ca73b58

Signed-off-by: currantw <[email protected]>

Add documentation

176ec46

Signed-off-by: currantw <[email protected]>

currantw added 7 commits January 13, 2025 16:21

Review comments: add more documentation, add ignored tests for earlie…

1c8e79d

…st and latest. Signed-off-by: currantw <[email protected]>

Minor clean up

16c64a9

Signed-off-by: currantw <[email protected]>

Update to use Instant and ZoneId. For some reason, timestamp can be r…

521feca

…eturned, but output from `$CurrentTimestamp` is an `Instant` Signed-off-by: currantw <[email protected]>

Remove unused import

98a1172

Signed-off-by: currantw <[email protected]>

Address review comments

f5992c9

Signed-off-by: currantw <[email protected]>

Address review comments

a33f093

Signed-off-by: currantw <[email protected]>

Update unit tests as per review comments

4db9de7

Signed-off-by: currantw <[email protected]>

currantw force-pushed the #991_relative_datetime branch from cad87e3 to 4db9de7 Compare January 14, 2025 00:22

LantaoJin approved these changes Jan 14, 2025

View reviewed changes

currantw added 2 commits January 15, 2025 09:39

Remove duplicate documentation.

f5cd625

Signed-off-by: currantw <[email protected]>

Fix failing IT

7daa8ec

Signed-off-by: currantw <[email protected]>

YANG-DB approved these changes Jan 15, 2025

View reviewed changes

YANG-DB merged commit 4a40676 into opensearch-project:main Jan 15, 2025
4 checks passed

acarbonetto deleted the #991_relative_datetime branch January 15, 2025 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#991 Support Relative Date Times #1006

#991 Support Relative Date Times #1006

currantw commented Jan 3, 2025 •

edited

Loading

acarbonetto Jan 3, 2025

YANG-DB Jan 4, 2025 •

edited

Loading

currantw Jan 9, 2025

LantaoJin Jan 10, 2025 •

edited

Loading

currantw Jan 13, 2025

YANG-DB Jan 4, 2025 •

edited

Loading

YANG-DB left a comment

currantw commented Jan 9, 2025

currantw Jan 9, 2025

acarbonetto Jan 9, 2025

currantw Jan 13, 2025

LantaoJin Jan 10, 2025 •

edited

Loading

acarbonetto Jan 13, 2025

currantw Jan 13, 2025

acarbonetto Jan 13, 2025

currantw Jan 13, 2025

acarbonetto Jan 13, 2025

currantw Jan 13, 2025

#991 Support Relative Date Times #1006

#991 Support Relative Date Times #1006

Conversation

currantw commented Jan 3, 2025 • edited Loading

Description

Related Issues

Check List

Choose a reason for hiding this comment

YANG-DB Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LantaoJin Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

YANG-DB left a comment

Choose a reason for hiding this comment

currantw commented Jan 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LantaoJin Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

currantw commented Jan 3, 2025 •

edited

Loading

YANG-DB Jan 4, 2025 •

edited

Loading

LantaoJin Jan 10, 2025 •

edited

Loading

YANG-DB Jan 4, 2025 •

edited

Loading

LantaoJin Jan 10, 2025 •

edited

Loading