[FEATURE] Flatten command should not remove existing field

[currantw]

Is your feature request related to a problem?

The behaviour of the flatten command is inconsistent between OpenSearch Spark, OpenSearch SQL, and Splunk:

• In OpenSearch SQL and Splunk, the flatten command does not remove the existing field that is being flattened.
• In OpenSearch Spark, the existing field is removed by the flatten command.

The behaviour should, at least,m be consistent between OpenSearch Spark and OpenSearch SQL.

What solution would you like?

Update OpenSearch Spark's flatten command so that the existing field is not removed.

What alternatives have you considered?

We could alternatively update OpenSearch SQL so that its behaviour is consistent with OpenSearch Spark (i.e. so that the field being flattened is removed from the search results).

Do you have any additional context?

• Splunk:
  • Flatten command overview
  • Flatten command usage
  • Flatten command examples
• OpenSearch Spark:
  • Documentation of flatten command
  • Issue to add flatten command
• OpenSearch SQL:
  • Documentation of flatten command
  • Issue to add flatten command
• Meta issue for flatten command