Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PPL-Lang] Separate overlap in functionality between flatten and expand PPL commands. #1024

Open
acarbonetto opened this issue Jan 30, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@acarbonetto
Copy link
Contributor

Is your feature request related to a problem?
Related PPL command request: #669

The flatten command takes either a STRUCT or ARRAY, and if an ARRAY is given, the command will essentially expand the array into multiple ROWs between flattening the STRUCT adding multiple COLUMNs.

Splunk documentation includes an example specifically for this case, and suggests that the user pipe flatten into expand, e.g. flatten <field> | expand <field>. Reference: https://docs.splunk.com/Documentation/SCS/current/SearchReference/expandcommandexamples#Expand_and_flatten_the_nested_array_fields

Additionally, the expand command's documentation indicates that it can also handle MAP<any> which isn't the intended purpose.
There is no testing for the MAP<any> type: https://github.com/YANG-DB/opensearch-spark/blob/cdadb1810a18359778c0c5bf795d7e1c9c4e74f7/docs/ppl-lang/ppl-expand-command.md?plain=1#L4-L6

That should instead read something like:

Using expand command to explode a field of type Array<Any> into multiple rows. 

What solution would you like?

Update usage of the flatten to only take a STRUCT. If it receives a multi-value/array field, only use the first value.

Add Proposal Document

  1. Update flatten documentation to recommend the use of expand to explode arrays.
  2. Update expand documentation to recommend the use of flatten to flatten struct data before expanding it.

Example Proposal Document

Do you have any additional context?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants