#3016 New `expand` PPL command #3305

currantw · 2025-02-06T01:05:52Z

Signed-off-by: currantw [email protected]

Description

Adds a new expand PPL command to OpenSearch SQL that create multiple rows corresponding to the elements on an multi-valued field. If the field contains a single value (not an array, including null or missing), the row is not modified.

This does not support expanding object fields. This is handled by the flatten command - see #3030.

Related Issues

Resolves #3030

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
~~API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

currantw · 2025-02-06T01:09:19Z

core/src/main/java/org/opensearch/sql/data/model/ExprTupleValue.java

+  public boolean equal(ExprValue other) {
+
+    if (!(other instanceof ExprTupleValue)) {
      return false;
-    } else {
-      ExprTupleValue other = (ExprTupleValue) o;
-      Iterator<Entry<String, ExprValue>> thisIterator = this.valueMap.entrySet().iterator();
-      Iterator<Entry<String, ExprValue>> otherIterator = other.valueMap.entrySet().iterator();
-      while (thisIterator.hasNext() && otherIterator.hasNext()) {
-        Entry<String, ExprValue> thisEntry = thisIterator.next();
-        Entry<String, ExprValue> otherEntry = otherIterator.next();
-        if (!(thisEntry.getKey().equals(otherEntry.getKey())
-            && thisEntry.getValue().equals(otherEntry.getValue()))) {
-          return false;
-        }
-      }
-      return !(thisIterator.hasNext() || otherIterator.hasNext());
    }
+
+    /**
+     * {@link Map#equals} returns true if the two maps' entry sets are equal, and works properly
+     * across all implementation of the {@link Map} interface. See {@link
+     * https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#equals-java.lang.Object-} for
+     * more details.
+     */
+    return valueMap.equals(other.tupleValue());
  }


Change pulled from flatten PR. See discussion there.
#3267

currantw · 2025-02-06T01:18:08Z

docs/user/ppl/functions/condition.rst

@@ -103,7 +103,7 @@ Description

 Usage: nullif(field1, field2) return null if two parameters are same, otherwise return field1.

-Argument type: all the supported data type, (NOTE : if two parameters has different type, if two parameters has different type, you will fail semantic check)


Just a typo that I came across. Unrelated.

currantw · 2025-02-06T01:18:59Z

integ-test/src/test/java/org/opensearch/sql/ppl/ExpandCommandIT.java

+        result,
+        rows("Seattle Seahawks", 2014),
+        rows("Seattle Kraken", null),
+        rows("Vancouver Canucks", null),


Apologies to any Canucks fans... 😢

currantw · 2025-02-06T01:26:49Z

core/src/main/java/org/opensearch/sql/utils/PathUtils.java

+import org.opensearch.sql.exception.SemanticCheckException;
+
+@UtilityClass
+public class PathUtils {


Added this new utilities module for dealing with paths. Includes unit tests. Once flatten has been merged (see here, I will also update that implementation to use these methods.

currantw · 2025-02-06T01:48:27Z

docs/user/ppl/cmd/expand.rst

+The ``expand`` command expands a field that contains an array of values to produce a seperate row for each value in the
+array. If the field does not contain an array, the row is not modified.
+
+Syntax
+============
+
+``expand <field>``
+
+* ``field``: reference to the field to flatten.
+
+Example 1: Expand a field
+=========================
+
+PPL query::
+
+    os> source=expand | expand team | fields city, team.name
+    fetched rows / total rows = 7/7
+    +--------------+-------------------+
+    | city         | team.name         |
+    |--------------+-------------------|
+    | Seattle      | Seattle Seahawks  |
+    | Seattle      | Seattle Kraken    |
+    | Vancouver    | Vancouver Canucks |
+    | Vancouver    | BC Lions          |
+    | San Antonio  | San Antonio Spurs |
+    | Null Team    | null              |
+    | Missing Team | null              |
+    +--------------+-------------------+


Bit of a difference from both Splunk and OpenSearch Spark.

This support multi-values and single values. If expand is applied to a field that contains both scalars (i.e. single values, including null and missing) and multi-values (i.e. arrays), then the arrays are expanded into separate rows, while the scalar rows remain unchanged. An empty array appears to be converted to null "under-the-hood" by OpenSearch Spark (see OpenSearchExprValueFactoryTest.constructNullArrayValue), so an empty array is handled the same as a null value.

currantw · 2025-02-06T01:52:27Z

core/src/test/java/org/opensearch/sql/planner/physical/ExpandOperatorTest.java

+    /** With {@link org.opensearch.sql.data.model.ExprMissingValue} */
+    inputRow = ExprValueUtils.tupleValue(Map.of());
+    mockInput(inputRow);
+
+    actualRows = execute(expand(inputPlan, DSL.ref("scalar_missing", ARRAY)));
+    expectedRows = List.of(inputRow);
+
+    assertEquals(expectedRows, actualRows);
+
+    /** Without {@link org.opensearch.sql.data.model.ExprMissingValue} */
+    inputRow = ExprValueUtils.tupleValue(Map.of("scalar_missing", missingExprValue));
+    mockInput(inputRow);
+
+    actualRows = execute(expand(inputPlan, DSL.ref("scalar_missing", ARRAY)));
+    expectedRows = List.of(inputRow);
+
+    assertEquals(expectedRows, actualRows);


When debugging, I found that "missing" values seem to simply by missing from the ExprValue mapping when they get read into OpenSearch SQL, rather than being explicitly mapped to ExprMissingValue. I test both cases here.

Signed-off-by: currantw <[email protected]>

…ation. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

currantw requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin and noCharger as code owners February 6, 2025 01:05

currantw commented Feb 6, 2025

View reviewed changes

currantw marked this pull request as draft February 6, 2025 01:54

currantw force-pushed the #3016_expand_command branch 2 times, most recently from 8f6838b to 7e0a31b Compare February 12, 2025 23:50

currantw added 7 commits February 12, 2025 18:34

Add flatten command to ANTLR lexer and parser.

f462e2a

Signed-off-by: currantw <[email protected]>

Skeleton implementation, tests, and documents with lots of TODOs.

69f0b1a

Signed-off-by: currantw <[email protected]>

Initial implementation

c1ac737

Signed-off-by: currantw <[email protected]>

Fix typo

366e162

Signed-off-by: currantw <[email protected]>

Initial implementation

0cbd8d4

Signed-off-by: currantw <[email protected]>

Update/fix tests.

26e9443

Signed-off-by: currantw <[email protected]>

Update integration tests to align with doc tests.

237b69e

Signed-off-by: currantw <[email protected]>

currantw added 28 commits February 12, 2025 18:34

Review comments - simplify FlattenOperator.flattenExprValueAtPath.

c1666ee

Signed-off-by: currantw <[email protected]>

Change braces in documentation.

6e176a3

Signed-off-by: currantw <[email protected]>

Initial implementation of skeleton classes and methods.

ab5a2fe

Signed-off-by: currantw <[email protected]>

Implement some of the expand logic

e7e5a5a

Signed-off-by: currantw <[email protected]>

Add PathUtils and unit tests.

a8f6855

Signed-off-by: currantw <[email protected]>

Update ExpandOperator and unit tests.

cf357ea

Signed-off-by: currantw <[email protected]>

Implement integration tests, update Expand logic, rename data set.

c260f6b

Signed-off-by: currantw <[email protected]>

Implement integration tests, update Expand logic, rename data set.

0d3c33c

Signed-off-by: currantw <[email protected]>

Add expand.rst documentation and further updates to tests/implement…

209326d

…ation. Signed-off-by: currantw <[email protected]>

Unrelated typo fix

dd9a024

Signed-off-by: currantw <[email protected]>

Cleanup, modify to return null for an empty array.

e31df37

Signed-off-by: currantw <[email protected]>

Fix test_docs.py typo, order alphabetically.

274719d

Signed-off-by: currantw <[email protected]>

Minor cleanup, mostly alphabetizing constants.

459ca24

Signed-off-by: currantw <[email protected]>

Add new doc and integration tests

eef907f

Signed-off-by: currantw <[email protected]>

Fix missing test coverage

77afac7

Signed-off-by: currantw <[email protected]>

Move PathUtils to ExprValueUtils and update tests.

49078e9

Signed-off-by: currantw <[email protected]>

Use ExprValueUtils to simplify FlattenOperator

ff0d5ae

Signed-off-by: currantw <[email protected]>

Simplify and make consistent flatten and expand operators.

790104c

Signed-off-by: currantw <[email protected]>

Update ExprValueUtils and unit tests.

ec47f8f

Signed-off-by: currantw <[email protected]>

Make constants private

101b653

Signed-off-by: currantw <[email protected]>

Spotless

ec0f44f

Signed-off-by: currantw <[email protected]>

Add hashCode unit tests

da7738a

Signed-off-by: currantw <[email protected]>

Trivial documentation cleanup.

4284f6f

Signed-off-by: currantw <[email protected]>

Fix doctest

1e51881

Signed-off-by: currantw <[email protected]>

General cleanup, combine flatten and expand datasets.

c4f732e

Signed-off-by: currantw <[email protected]>

Spotless

582fdaa

Signed-off-by: currantw <[email protected]>

Fix failing doctests

e8494f2

Signed-off-by: currantw <[email protected]>

Fix ExplainIT test

028e074

Signed-off-by: currantw <[email protected]>

currantw force-pushed the #3016_expand_command branch from 7e0a31b to 028e074 Compare February 13, 2025 03:10

Handle empty collections.

01569ae

Signed-off-by: currantw <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3016 New `expand` PPL command #3305

#3016 New `expand` PPL command #3305

currantw commented Feb 6, 2025

currantw Feb 6, 2025

currantw Feb 6, 2025

currantw Feb 6, 2025

currantw Feb 6, 2025

currantw Feb 6, 2025 •

edited

Loading

currantw Feb 6, 2025

		@@ -103,7 +103,7 @@ Description

		Usage: nullif(field1, field2) return null if two parameters are same, otherwise return field1.

		Argument type: all the supported data type, (NOTE : if two parameters has different type, if two parameters has different type, you will fail semantic check)

#3016 New expand PPL command #3305

Are you sure you want to change the base?

#3016 New expand PPL command #3305

Conversation

currantw commented Feb 6, 2025

Description

Related Issues

Check List

currantw Feb 6, 2025

Choose a reason for hiding this comment

currantw Feb 6, 2025

Choose a reason for hiding this comment

currantw Feb 6, 2025

Choose a reason for hiding this comment

currantw Feb 6, 2025

Choose a reason for hiding this comment

currantw Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

currantw Feb 6, 2025

Choose a reason for hiding this comment

#3016 New `expand` PPL command #3305

#3016 New `expand` PPL command #3305

currantw Feb 6, 2025 •

edited

Loading