[PPL-Lang] Separate overlap in functionality between flatten
and expand
PPL commands.
#1024
Labels
enhancement
New feature or request
Is your feature request related to a problem?
Related PPL command request: #669
The
flatten
command takes either a STRUCT or ARRAY, and if an ARRAY is given, the command will essentiallyexpand
the array into multiple ROWs between flattening the STRUCT adding multiple COLUMNs.Splunk documentation includes an example specifically for this case, and suggests that the user pipe flatten into expand, e.g.
flatten <field> | expand <field>
. Reference: https://docs.splunk.com/Documentation/SCS/current/SearchReference/expandcommandexamples#Expand_and_flatten_the_nested_array_fieldsAdditionally, the
expand
command's documentation indicates that it can also handleMAP<any>
which isn't the intended purpose.There is no testing for the
MAP<any>
type: https://github.com/YANG-DB/opensearch-spark/blob/cdadb1810a18359778c0c5bf795d7e1c9c4e74f7/docs/ppl-lang/ppl-expand-command.md?plain=1#L4-L6That should instead read something like:
What solution would you like?
Update usage of the
flatten
to only take a STRUCT. If it receives a multi-value/array field, only use the first value.Add Proposal Document
flatten
documentation to recommend the use ofexpand
to explode arrays.expand
documentation to recommend the use offlatten
to flatten struct data before expanding it.Example Proposal Document
Do you have any additional context?
flatten
Command to PPL sql#3029expand_field
command sql#3015The text was updated successfully, but these errors were encountered: