PPL: Add `json_set` and `json_extend` command to spark-ppl #1038

acarbonetto · 2025-02-07T04:34:16Z

Description

Adds json_set and json_extend functions to the spark PPL UDF.

Related Issues

Resolves #996

Check List

Updated documentation (docs/ppl-lang/README.md)
Implemented unit tests
Implemented tests for combination with other commands
New added source code should include a copyright header
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andrew Carbonetto <[email protected]>

LantaoJin

Could you add some ITs for testing integrated with multiple json functions

LantaoJin · 2025-02-07T08:21:33Z

...rk-integration/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java

@@ -223,10 +223,6 @@ public enum BuiltinFunctionName {
  JSON_EXTRACT(FunctionName.of("json_extract")),
  JSON_KEYS(FunctionName.of("json_keys")),
  JSON_VALID(FunctionName.of("json_valid")),
-//  JSON_DELETE(FunctionName.of("json_delete")),


Em, how did json_delete work if we didn't included here?

Its not a built-in function, but a user-defined function

"Built-in" here means that this function is built-in in PPL-on-Spark, not that it is built-in in Spark. Therefore, as long as the implementation is provided, those that do not require a user to provide their own implementation are considered "built-in" (we currently do not provide a public interface for user-defined functions, so the functions we add are all built-in functions). It seems due to this code

opensearch-spark/ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/BuiltinFunctionTransformer.java

Lines 203 to 208 in c6d8793

if (BuiltinFunctionName.of(function.getFuncName()).isEmpty()) {

ScalaUDF udf = SerializableUdf.visit(function.getFuncName(), args);

if(udf == null) {

throw new UnsupportedOperationException(function.getFuncName() + " is not a builtin function of PPL");

}

return udf;

, there is no need to declare them in this class. I think we need unified them in BuiltinFunctionName.( Not a blocker for this PR).

LantaoJin · 2025-02-07T08:23:58Z

cc @qianheng-aws

qianheng-aws

Please add IT for this PR and ensure the examples you add in the document are all correct.

...st/scala/org/opensearch/flint/spark/ppl/PPLLogicalPlanJsonFunctionsTranslatorTestSuite.scala

qianheng-aws · 2025-02-07T11:39:34Z

docs/ppl-lang/functions/ppl-json.md

@@ -278,12 +301,57 @@ Example:
    |{"teacher":["Alice","Tom","Walt"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}                                 |
    +-----------------------------------------------------------------------------------------------------------------------------------+

-
-    os> source=people | eval append = json_append(`{"school":{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}}`,array('school.teacher', 'Tom', 'Walt')) | head 1 | fields append
+    os> source=people | eval append = json_append(`{"school":{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}}`,array('school.teacher', array('Tom', 'Walt'))) | head 1 | fields append


Have you ever checked this PPL? Does it work?

As far as I know, function array in Spark won't accept elements of different types.

Please take a look at the latest example/IT test.
We can use a json-encoded string of arrays to do this. But passing the array does have abide by the rules of our Spark array object.

qianheng-aws · 2025-02-07T11:45:33Z

docs/ppl-lang/functions/ppl-json.md

+
+Example:
+
+    os> source=people | eval extend = json_extend(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, 'student', '{"name":"Tommy","rank":5}') | head 1 | fields extend


The same problem as found in json_set. These 2 functions should only have 2 parameters as defined in the code while given 3 here.

@qianheng-aws the second parameter is actually a list of key-value pairs if I'm not mistaken - @acarbonetto is this correct?

docs/ppl-lang/functions/ppl-json.md

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java

qianheng-aws · 2025-02-07T12:20:26Z

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java

+     * @param depth - current traversal depth
+     * @param valueToUpdate - value to update
+     */
+    static void updateNestedValue(Object currentObj, String[] pathParts, int depth, Object valueToUpdate) {


Can we reuse the code from appendNestedValue? These 2 functions seem to have mostly duplicated code with differences only in their final operations -- set or append.

I was thinking the same thing. I will try and consolidate today - maybe with another functional argument.

Same thought, I reckon we will end up having something similar in term of class structure:

JsonUtils { updateNestedValue( ) { // Invoke METHOD_TRAVEERSE method with Lambda for the handling (Update). } appendNestedValue( ) { // Invoke METHOD_TRAVEERSE method with Lambda for the handling (Append). } private static METHOD_TRAVEERSE (FunctionalInterface ) { } }

Please take a look at the latest iteration @qianheng-aws
I've added traversal and update functions, as well passing in a lambda that does the actual implemented.
Let me know if you find any more candidates for optimization.

Signed-off-by: Andrew Carbonetto <[email protected]>

docs/ppl-lang/functions/ppl-json.md

YANG-DB · 2025-02-10T19:36:55Z

@acarbonetto can you plz check why CI failed ?

acarbonetto · 2025-02-10T20:25:28Z

@acarbonetto can you plz check why CI failed ?

Sure. IT failed, but it doesn't look like it's from a failed test.

docs/ppl-lang/functions/ppl-json.md

andy-k-improving · 2025-02-10T20:45:14Z

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java

+     * @param depth - current traversal depth
+     * @param valueToUpdate - value to update
+     */
+    static void updateNestedValue(Object currentObj, String[] pathParts, int depth, Object valueToUpdate) {


Same thought, I reckon we will end up having something similar in term of class structure:

JsonUtils { updateNestedValue( ) { // Invoke METHOD_TRAVEERSE method with Lambda for the handling (Update). } appendNestedValue( ) { // Invoke METHOD_TRAVEERSE method with Lambda for the handling (Append). } private static METHOD_TRAVEERSE (FunctionalInterface ) { } }

Signed-off-by: Andrew Carbonetto <[email protected]>

qianheng-aws

CI build failure due to scala style format issue. Please run sbt scalafmtAll before submitting your code.

.../src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLJsonFunctionITSuite.scala

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java

Signed-off-by: Andrew Carbonetto <[email protected]>

docs/ppl-lang/functions/ppl-json.md

acarbonetto · 2025-02-11T01:03:50Z

docs/ppl-lang/functions/ppl-json.md

@@ -245,9 +270,12 @@ Example:

 **Description**

-`json_append(json_string, [path_key, list of values to add ])`  appends values to end of an array within the json elements. Return the updated json object after appending .
+`json_append(json_string, array(key1, value1, key2, value2, ...))` appends values to end of an array at key within the json elements. Returns the updated json object after appending.


Update json_append syntax to align with SP2 syntax, which takes pairs of key/values. Tests also updated.

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java

Signed-off-by: Andrew Carbonetto <[email protected]>

qianheng-aws

LGTM, Thanks for the contribution!

LantaoJin · 2025-02-14T02:46:40Z

.../src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLJsonFunctionITSuite.scala

+  test("test json_extend() function: add single value key not found") {
+    val frame = sql(s"""
+                       | source = $testTable
+                       | | eval result = json_extend('$validJson7',array('headmaster', 'Tom')) | head 1 | fields result


Why this 'Tom' works?
I see three different formats in this IT:

'\"Foobar\"'

'"Tom"'

'Tom'
This is a bit confusing.

1 is completely equal to 2 in scala.StringContext;

and 2 (""Tom"") will be transformed to 3 ("Tom") after parsing by jackson.

1 is completely equal to 2 in scala.StringContext;

I see.

and 2 (""Tom"") will be transformed to 3 ("Tom") after parsing by jackson.

Do you mean they are all equality?

Yeah, at least equality in our code implementation.

LantaoJin · 2025-02-14T02:59:55Z

...rk-integration/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java

@@ -223,10 +223,6 @@ public enum BuiltinFunctionName {
  JSON_EXTRACT(FunctionName.of("json_extract")),
  JSON_KEYS(FunctionName.of("json_keys")),
  JSON_VALID(FunctionName.of("json_valid")),
-//  JSON_DELETE(FunctionName.of("json_delete")),


"Built-in" here means that this function is built-in in PPL-on-Spark, not that it is built-in in Spark. Therefore, as long as the implementation is provided, those that do not require a user to provide their own implementation are considered "built-in" (we currently do not provide a public interface for user-defined functions, so the functions we add are all built-in functions). It seems due to this code

opensearch-spark/ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/BuiltinFunctionTransformer.java

Lines 203 to 208 in c6d8793

if (BuiltinFunctionName.of(function.getFuncName()).isEmpty()) {

ScalaUDF udf = SerializableUdf.visit(function.getFuncName(), args);

if(udf == null) {

throw new UnsupportedOperationException(function.getFuncName() + " is not a builtin function of PPL");

}

return udf;

, there is no need to declare them in this class. I think we need unified them in BuiltinFunctionName.( Not a blocker for this PR).

acarbonetto added 3 commits February 6, 2025 11:01

PPL: Add the json_extend command

e21c42d

Signed-off-by: Andrew Carbonetto <[email protected]>

PPL: add json_set function

1de4410

Signed-off-by: Andrew Carbonetto <[email protected]>

scalafmt

401fa93

Signed-off-by: Andrew Carbonetto <[email protected]>

acarbonetto requested review from dai-chen, mengweieric, penghuo, seankao-az, anirudha, kaituo, YANG-DB, noCharger, LantaoJin and ykmr1224 as code owners February 7, 2025 04:34

YANG-DB enabled auto-merge (squash) February 7, 2025 04:35

YANG-DB disabled auto-merge February 7, 2025 04:35

acarbonetto changed the title ~~PPL: Add json_extend command to spark-ppl~~ PPL: Add json_set and json_extend command to spark-ppl Feb 7, 2025

Remove comment

8f1539f

Signed-off-by: Andrew Carbonetto <[email protected]>

LantaoJin reviewed Feb 7, 2025

View reviewed changes

qianheng-aws reviewed Feb 7, 2025

View reviewed changes

Update JSON_APPEND, JSON_EXTEND, JSON_SET

0937560

Signed-off-by: Andrew Carbonetto <[email protected]>

YANG-DB reviewed Feb 10, 2025

View reviewed changes

docs/ppl-lang/functions/ppl-json.md Outdated Show resolved Hide resolved

YANG-DB approved these changes Feb 10, 2025

View reviewed changes

andy-k-improving reviewed Feb 10, 2025

View reviewed changes

acarbonetto added 3 commits February 10, 2025 15:30

Consolidate nested function calls

f20b0ee

Signed-off-by: Andrew Carbonetto <[email protected]>

Add IT tests for json_set, json_expand

09b4304

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix examples

6c69545

Signed-off-by: Andrew Carbonetto <[email protected]>

qianheng-aws suggested changes Feb 11, 2025

View reviewed changes

.../src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLJsonFunctionITSuite.scala Outdated Show resolved Hide resolved

.../src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLJsonFunctionITSuite.scala Show resolved Hide resolved

qianheng-aws reviewed Feb 11, 2025

View reviewed changes

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java Outdated Show resolved Hide resolved

ppl-spark-integration/src/main/java/org/opensearch/sql/expression/function/JsonUtils.java Outdated Show resolved Hide resolved

acarbonetto added 2 commits February 11, 2025 10:06

Fix IT tests

a3d8f39

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix unit tests

1e02881

Signed-off-by: Andrew Carbonetto <[email protected]>

acarbonetto commented Feb 11, 2025

View reviewed changes

Fix syntax errors in IT tests

6307509

Signed-off-by: Andrew Carbonetto <[email protected]>

YANG-DB self-requested a review February 13, 2025 19:17

acarbonetto added 4 commits February 13, 2025 12:39

Add IT tests for flatten calls

ab531cd

Signed-off-by: Andrew Carbonetto <[email protected]>

Refactor nested traverse

d17e337

Signed-off-by: Andrew Carbonetto <[email protected]>

Update ppl-json.md examples

2675763

Signed-off-by: Andrew Carbonetto <[email protected]>

Re-add missing import

d123e52

Signed-off-by: Andrew Carbonetto <[email protected]>

qianheng-aws approved these changes Feb 14, 2025

View reviewed changes

LantaoJin reviewed Feb 14, 2025

View reviewed changes

LantaoJin approved these changes Feb 14, 2025

View reviewed changes

YANG-DB approved these changes Feb 14, 2025

View reviewed changes

YANG-DB merged commit 8d0e591 into opensearch-project:main Feb 14, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PPL: Add `json_set` and `json_extend` command to spark-ppl #1038

PPL: Add `json_set` and `json_extend` command to spark-ppl #1038

acarbonetto commented Feb 7, 2025 •

edited

Loading

LantaoJin left a comment

LantaoJin Feb 7, 2025

acarbonetto Feb 7, 2025

LantaoJin Feb 14, 2025

LantaoJin commented Feb 7, 2025

qianheng-aws left a comment

qianheng-aws Feb 7, 2025

acarbonetto Feb 13, 2025

qianheng-aws Feb 7, 2025

YANG-DB Feb 10, 2025

acarbonetto Feb 10, 2025

qianheng-aws Feb 7, 2025

acarbonetto Feb 7, 2025

andy-k-improving Feb 10, 2025

acarbonetto Feb 13, 2025

YANG-DB commented Feb 10, 2025

acarbonetto commented Feb 10, 2025

andy-k-improving Feb 10, 2025

qianheng-aws left a comment

acarbonetto Feb 11, 2025

qianheng-aws left a comment

LantaoJin Feb 14, 2025 •

edited

Loading

qianheng-aws Feb 14, 2025

LantaoJin Feb 14, 2025

qianheng-aws Feb 14, 2025

LantaoJin Feb 14, 2025

	if (BuiltinFunctionName.of(function.getFuncName()).isEmpty()) {
	ScalaUDF udf = SerializableUdf.visit(function.getFuncName(), args);
	if(udf == null) {
	throw new UnsupportedOperationException(function.getFuncName() + " is not a builtin function of PPL");
	}
	return udf;


		Example:

		os> source=people \| eval extend = json_extend(`{"teacher":["Alice"],"student":[{"name":"Bob","rank":1},{"name":"Charlie","rank":2}]}`, 'student', '{"name":"Tommy","rank":5}') \| head 1 \| fields extend

PPL: Add json_set and json_extend command to spark-ppl #1038

PPL: Add json_set and json_extend command to spark-ppl #1038

Conversation

acarbonetto commented Feb 7, 2025 • edited Loading

Description

Related Issues

Check List

LantaoJin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LantaoJin commented Feb 7, 2025

qianheng-aws left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YANG-DB commented Feb 10, 2025

acarbonetto commented Feb 10, 2025

Choose a reason for hiding this comment

qianheng-aws left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qianheng-aws left a comment

Choose a reason for hiding this comment

LantaoJin Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PPL: Add `json_set` and `json_extend` command to spark-ppl #1038

PPL: Add `json_set` and `json_extend` command to spark-ppl #1038

acarbonetto commented Feb 7, 2025 •

edited

Loading

LantaoJin Feb 14, 2025 •

edited

Loading