PPL: Add `json_extract` function #3262

kenrickyap · 2025-01-23T16:33:02Z

Description

Implements json_extract function that would be useful to extract specific objects and scalars from an existing json_object.

Related Issues

Resolves #3211
Follow up issue for #3027

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Kenrick Yap <[email protected]>

Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

….java Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

…nsearch-project-sql into feature/json-valid Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Andrew Carbonetto <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: kenrickyap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]>

acarbonetto

a few comments to address

core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

docs/user/ppl/functions/json.rst

Signed-off-by: Kenrick Yap <[email protected]>

Signed-off-by: Kenrick Yap <[email protected]> # Conflicts: # core/src/main/java/org/opensearch/sql/expression/DSL.java # core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java # core/src/main/java/org/opensearch/sql/expression/json/JsonFunctions.java # core/src/main/java/org/opensearch/sql/utils/JsonUtils.java # core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java # docs/user/ppl/functions/json.rst # integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionsIT.java # integ-test/src/test/resources/json_test.json # ppl/src/main/antlr/OpenSearchPPLLexer.g4 # ppl/src/main/antlr/OpenSearchPPLParser.g4

Signed-off-by: Kenrick Yap <[email protected]>

currantw · 2025-02-13T20:35:22Z

docs/user/ppl/functions/json.rst

+Usage: `json_extract(doc, path[, path])` Extracts a JSON value from a json document based on the path specified.
+
+Argument type: STRING, STRING
+
+Return type: STRING/BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY
+
+- Up to 3 paths can be provided, and results of each `path` with be returned in an ARRAY.
+- Returns an ARRAY if `path` points to multiple results (e.g. $.a[*]) or if the `path` points to an array.
+- Return null if `path` is not valid, or if JSON `doc` is MISSING or NULL.
+- Throws SemanticCheckException if `doc` or `path` is malformed.
+- Throws ExpressionEvaluationException if `path` is missing.


Looks like this documentation might need some updates for multiple paths?

Usage: json_extract(doc, path[, path[, path[[) (or another way to indicate that up to 3 paths can be provided)

Usage: "based on the path(s) specified"

Argument type: "STRING, STRING[, STRING[, STRING]] (or another way to indicate that up to 3 paths can be provided)

What about the case where only one of the paths is invalid? Should this read "Returns null if any path is not valid"? Similarly, should the last two bullets be updated to specify that they apply to "any path"?

currantw · 2025-02-13T20:37:59Z

integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionsIT.java

+  @Test
+  public void test_json_extract() throws IOException {
+    JSONObject result;
+    result =
+        executeQuery(
+            String.format(
+                "source=%s | where json_valid(json_string) | eval"
+                    + " extracted=json_extract(json_string, '$.b') | fields test_name, extracted",
+                TEST_INDEX_JSON_TEST));
+    verifySchema(
+        result, schema("test_name", null, "string"), schema("extracted", null, "undefined"));
+    verifyDataRows(
+        result,
+        rows("json nested object", new JSONObject(Map.of("c", "3"))),
+        rows("json object", "2"),
+        rows("json array", null),
+        rows("json nested array", null),
+        rows("json scalar string", null),
+        rows("json scalar int", null),
+        rows("json scalar float", null),
+        rows("json scalar double", null),
+        rows("json scalar boolean true", null),
+        rows("json scalar boolean false", null),
+        rows("json empty string", null),
+        rows("json nested list", new JSONArray(List.of(Map.of("c", "2"), Map.of("c", "3")))));
+  }


As discussed during yesterday's review session, we certainly don't need to test all edge cases in IT, but I think we probably want to have more than one test. At least, we should probably have one with two path strings and one with three path strings?

...and maybe test the 4 path case where we get a parsing error

we want to test a couple of other error scenarios mentioned: such as null returns and Semantic errors for malformed json.

added test IT test cases for use of multiple paths and for the cases when semantic errors are thrown. Though I am curious why we need to test for errors in this case as I cannot find any other IT test testing for errors.

In addition by the time the IT test receive errors they are received as ResponseErrors so had to check if error contains reference to semantic error.

currantw · 2025-02-13T20:38:51Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+    List<ExprValue> resultList = new ArrayList<>();
+
+    for (ExprValue path : paths) {
+      System.out.println("Processing path: " + path);


Looks like an errant debug statement?

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

currantw · 2025-02-13T20:43:50Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+   * @return ExprValue of value at given path of json string.
+   */
+  public static ExprValue extractJson(ExprValue json, ExprValue... paths) {
+    List<ExprValue> resultList = new ArrayList<>();


Since know that the array list is always going to be the same since as paths, can we just initialize it with a specified initialCapacity to save some memory space?

currantw · 2025-02-13T20:49:43Z

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

+  }
+
+  private static ExprValue extractJsonPath(String json, String path) {
+    if (json.isEmpty() || json.equals("null")) {


Do we need a separate case for either of these? Seems like this would just throw a PathNotFoundException and returns LITERAL_NULL below if not (at least for "null", I'm not sure about the empty string). Probably makes sense to reduce branching here if possible. 🤷

Moreover, do we actually want to return LITERAL_NULL for the empty string? Is that an invalid JSON string? If so, does it make sense to raise the corresponding exception instead?

both "null" and "" are considered valid json strings by json_valid, since we expect that users will use json_valid to first filter for valid json then use json extract to avoid errors, we need these specific cases to be consistent with json_valid

currantw · 2025-02-13T20:52:35Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+    assertEquals(expected, actual);
+  }
+
+  private static void execute_extract_json(ExprValue expected, String json, String path) {


Can this be generalized to allow multiple paths, so you can use it for json_extract_search_list_of_paths as well?

Suggested change

private static void execute_extract_json(ExprValue expected, String json, String path) {

private static void execute_extract_json(ExprValue expected, String json, String... paths) {

currantw · 2025-02-13T20:53:27Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+  }
+
+  @Test
+  void json_extract_throws_ExpressionEvaluationException() {


You could split execute_extract_json into two methods: execute_extract_json (which would just actually run jsonExtract) and test_extract_json (or assert_extract_json), which would call execute_extract_json and then actually do the comparison.

That way, test methods like this could call execute_extract_json, and no have to worry about calling DSL.literal 🤷

currantw · 2025-02-13T20:58:32Z

core/src/test/java/org/opensearch/sql/expression/json/JsonFunctionsTest.java

+  }
+
+  @Test
+  void json_extract_search_arrays() {


Do we have tests for the following:

Multiple paths where one or more paths match an array

Multiple paths where one or more paths match more than one value

currantw

Reviewed to ec6ff5e.

Co-authored-by: Taylor Curran <[email protected]> Signed-off-by: kenrickyap <[email protected]>

acarbonetto · 2025-02-13T22:59:52Z

docs/user/ppl/functions/json.rst

+
+Argument type: STRING, STRING
+
+Return type: STRING/BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY


Use JSON_STRING instead

Shouldn't we always return an ARRAY?

in the case that only one string is provided, we return the associated type which will not be a json string. e.g. if path "$.a" is provided from json_string "{"a": 1}: then a integer is returned

acarbonetto · 2025-02-13T23:02:05Z

docs/user/ppl/functions/json.rst

+
+Return type: STRING/BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY
+
+- Up to 3 paths can be provided, and results of each `path` with be returned in an ARRAY.


phrasing is a little confusing.

Suggested change

- Up to 3 paths can be provided, and results of each `path` with be returned in an ARRAY.

- Up to 3 paths can be provided, and results of all possible `path`s will be returned in an ARRAY.

acarbonetto · 2025-02-13T23:03:30Z

docs/user/ppl/functions/json.rst

+
+- Up to 3 paths can be provided, and results of each `path` with be returned in an ARRAY.
+- Returns an ARRAY if `path` points to multiple results (e.g. $.a[*]) or if the `path` points to an array.
+- Return null if `path` is not valid, or if JSON `doc` is MISSING or NULL.


null or empty array?

in the case of multiple paths an array of nulls would be returned

acarbonetto · 2025-02-13T23:04:06Z

docs/user/ppl/functions/json.rst

+- Returns an ARRAY if `path` points to multiple results (e.g. $.a[*]) or if the `path` points to an array.
+- Return null if `path` is not valid, or if JSON `doc` is MISSING or NULL.
+- Throws SemanticCheckException if `doc` or `path` is malformed.
+- Throws ExpressionEvaluationException if `path` is missing.


this is not handled in code in this PR, this is the default error thrown if provided function does not matched defined schema for functions. Thrown by DefaultFunctionResolver.

acarbonetto · 2025-02-13T23:05:04Z

integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionsIT.java

+  @Test
+  public void test_json_extract() throws IOException {
+    JSONObject result;
+    result =
+        executeQuery(
+            String.format(
+                "source=%s | where json_valid(json_string) | eval"
+                    + " extracted=json_extract(json_string, '$.b') | fields test_name, extracted",
+                TEST_INDEX_JSON_TEST));
+    verifySchema(
+        result, schema("test_name", null, "string"), schema("extracted", null, "undefined"));
+    verifyDataRows(
+        result,
+        rows("json nested object", new JSONObject(Map.of("c", "3"))),
+        rows("json object", "2"),
+        rows("json array", null),
+        rows("json nested array", null),
+        rows("json scalar string", null),
+        rows("json scalar int", null),
+        rows("json scalar float", null),
+        rows("json scalar double", null),
+        rows("json scalar boolean true", null),
+        rows("json scalar boolean false", null),
+        rows("json empty string", null),
+        rows("json nested list", new JSONArray(List.of(Map.of("c", "2"), Map.of("c", "3")))));
+  }


...and maybe test the 4 path case where we get a parsing error

acarbonetto · 2025-02-13T23:06:18Z

integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionsIT.java

+  @Test
+  public void test_json_extract() throws IOException {
+    JSONObject result;
+    result =
+        executeQuery(
+            String.format(
+                "source=%s | where json_valid(json_string) | eval"
+                    + " extracted=json_extract(json_string, '$.b') | fields test_name, extracted",
+                TEST_INDEX_JSON_TEST));
+    verifySchema(
+        result, schema("test_name", null, "string"), schema("extracted", null, "undefined"));
+    verifyDataRows(
+        result,
+        rows("json nested object", new JSONObject(Map.of("c", "3"))),
+        rows("json object", "2"),
+        rows("json array", null),
+        rows("json nested array", null),
+        rows("json scalar string", null),
+        rows("json scalar int", null),
+        rows("json scalar float", null),
+        rows("json scalar double", null),
+        rows("json scalar boolean true", null),
+        rows("json scalar boolean false", null),
+        rows("json empty string", null),
+        rows("json nested list", new JSONArray(List.of(Map.of("c", "2"), Map.of("c", "3")))));
+  }


we want to test a couple of other error scenarios mentioned: such as null returns and Semantic errors for malformed json.

Signed-off-by: Kenrick Yap <[email protected]>

Yury-Fridlyand · 2025-02-14T21:10:30Z

docs/user/ppl/functions/json.rst

+    | json empty string  |                                      | null                    |
+    +--------------------+--------------------------------------+-------------------------+
+
+    os> source=json_test | where test_name="json nested list" | eval json_extract=json_extract(json_string, '$.b[1].c') | fields test_name, json_string, json_extract


It will be kind of you if you post examples and description to the PR

Yury-Fridlyand · 2025-02-14T21:11:14Z

docs/user/ppl/functions/json.rst

+Description
+>>>>>>>>>>>
+
+Usage: `json_extract(doc, path[, path[, path])` Extracts a JSON value from a json document based on the path(s) specified.


So, it accepts 1-3 paths. Why not array of paths then?

Yury-Fridlyand · 2025-02-14T21:14:46Z

docs/user/ppl/functions/json.rst

+
+Example::
+
+    os> source=json_test | where json_valid(json_string) | eval json_extract=json_extract(json_string, '$.b') | fields test_name, json_string, json_extract


Does it accept "legacy" path (path which starts with ".")? if yes, this should be tested and documented. If no, this should be documented.

14yapkc1 and others added 30 commits January 3, 2025 10:54

added implementation

70152eb

Signed-off-by: Kenrick Yap <[email protected]>

added doctest, integ-tests, and unit tests

76c3995

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

ce2c551

Signed-off-by: Kenrick Yap <[email protected]>

fixed unit tests

ad1bde3

Signed-off-by: Kenrick Yap <[email protected]>

addressed pr comments

ccf47a2

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

acc76a0

Signed-off-by: Kenrick Yap <[email protected]>

removed unused dependencies

519c6f2

Signed-off-by: Kenrick Yap <[email protected]>

linting

2e319fe

Signed-off-by: Kenrick Yap <[email protected]>

addressed pr comment and rolling back disabled test case

ee0820d

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'main' into feature/json-valid

d44fc5a

Signed-off-by: Kenrick Yap <[email protected]>

removed disabled import

3407d4a

Signed-off-by: Kenrick Yap <[email protected]>

Update docs/user/ppl/functions/json.rst

7ef6cc9

Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

Update integ-test/src/test/java/org/opensearch/sql/ppl/JsonFunctionIT…

e5e90ac

….java Co-authored-by: Andrew Carbonetto <[email protected]> Signed-off-by: kenrickyap <[email protected]>

nit

2187a5a

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'feature/json-valid' of https://github.com/Bit-Quill/ope…

5e1e488

…nsearch-project-sql into feature/json-valid Signed-off-by: Kenrick Yap <[email protected]>

fixed integ test

3512b33

Signed-off-by: Kenrick Yap <[email protected]>

change text type to keyword

9fea606

Signed-off-by: Kenrick Yap <[email protected]>

addressed PR comments

fbc54bc

Signed-off-by: Kenrick Yap <[email protected]>

fix doc-test

31ad2a4

Signed-off-by: Kenrick Yap <[email protected]>

added null test

2b2a8f3

Signed-off-by: Kenrick Yap <[email protected]>

Merge branch 'main' into feature/json-valid

dc96563

SQL: adding error case unit tests for json_valid

1913bfe

Signed-off-by: Andrew Carbonetto <[email protected]>

json_valid: null and missing should return false

67d979d

Signed-off-by: Andrew Carbonetto <[email protected]>

PPL: Add json and cast to json functions

aa6b723

Signed-off-by: Andrew Carbonetto <[email protected]>

PPL: Update json cast for review

4c99235

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix testes

9ccde7f

Signed-off-by: Andrew Carbonetto <[email protected]>

spotless

4306bf3

Signed-off-by: Andrew Carbonetto <[email protected]>

Fix tests

613137b

Signed-off-by: Andrew Carbonetto <[email protected]>

SPOTLESS

ab28872

Signed-off-by: Andrew Carbonetto <[email protected]>

Clean up for merge

3ec16e0

Signed-off-by: Andrew Carbonetto <[email protected]>

YANG-DB previously approved these changes Jan 30, 2025

View reviewed changes

kenrickyap added 2 commits January 31, 2025 09:54

address comments

306ac97

Signed-off-by: Kenrick Yap <[email protected]>

fix build error

75e9cc3

Signed-off-by: Kenrick Yap <[email protected]>

kenrickyap force-pushed the feature/json-extract branch from dfe5d3a to 75e9cc3 Compare January 31, 2025 14:57

Merge branch 'main' into feature/json-extract

77827bb

Signed-off-by: kenrickyap <[email protected]>

kenrickyap dismissed YANG-DB’s stale review via 77827bb January 31, 2025 15:00

add header

80f44e2

Signed-off-by: Kenrick Yap <[email protected]>

YANG-DB previously approved these changes Feb 3, 2025

View reviewed changes

acarbonetto reviewed Feb 5, 2025

View reviewed changes

andy-k-improving reviewed Feb 7, 2025

View reviewed changes

kenrickyap added 2 commits February 11, 2025 20:59

addressing PR comments

0d1cc28

Signed-off-by: Kenrick Yap <[email protected]>

added multi path use case

b6ae5ba

Signed-off-by: Kenrick Yap <[email protected]>

kenrickyap dismissed YANG-DB’s stale review via b6ae5ba February 12, 2025 18:02

kenrickyap added 3 commits February 12, 2025 10:24

linting

aa8b81e

Signed-off-by: Kenrick Yap <[email protected]>

fixing doc tests

ec6ff5e

Signed-off-by: Kenrick Yap <[email protected]>

currantw reviewed Feb 13, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/utils/JsonUtils.java Outdated Show resolved Hide resolved

currantw reviewed Feb 13, 2025

View reviewed changes

Update core/src/main/java/org/opensearch/sql/utils/JsonUtils.java

95e996b

Co-authored-by: Taylor Curran <[email protected]> Signed-off-by: kenrickyap <[email protected]>

acarbonetto reviewed Feb 13, 2025

View reviewed changes

addressed PR comments

591fb71

Signed-off-by: Kenrick Yap <[email protected]>

Yury-Fridlyand reviewed Feb 14, 2025

View reviewed changes

	private static void execute_extract_json(ExprValue expected, String json, String path) {
	private static void execute_extract_json(ExprValue expected, String json, String... paths) {


		Argument type: STRING, STRING

		Return type: STRING/BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY


		Return type: STRING/BOOLEAN/DOUBLE/INTEGER/NULL/STRUCT/ARRAY

		- Up to 3 paths can be provided, and results of each `path` with be returned in an ARRAY.


		Example::

		os> source=json_test \| where json_valid(json_string) \| eval json_extract=json_extract(json_string, '$.b') \| fields test_name, json_string, json_extract

PPL: Add json_extract function #3262

Are you sure you want to change the base?

PPL: Add json_extract function #3262

Conversation

kenrickyap commented Jan 23, 2025 • edited Loading

Description

Related Issues

Check List

acarbonetto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

currantw Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

currantw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PPL: Add `json_extract` function #3262

PPL: Add `json_extract` function #3262

kenrickyap commented Jan 23, 2025 •

edited

Loading

currantw Feb 13, 2025 •

edited

Loading