Added end-to-end PPL tests for Spark

[normanj-bitquill]

Description

Adding end-to-end PPL tests for Spark. The tests queries are run on the Spark master container using Spark Connect. OpenSearch is not used.

Tests include PPL queries form the Python test suite and the TPC-H queries.

Related Issues

#1022

Check List

• Updated documentation (docs/ppl-lang/README.md)
• Implemented unit tests
• Implemented tests for combination with other commands
• New added source code should include a copyright header
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[YANG-DB]

@normanj-bitquill this is great !!!

[penghuo]

Thanks for adding these. I have a few questions:
1 Could you explain how the queries and results are generated?
2 If a new command is added, how should it be incorporated into the existing test sets?
3 Are these tests already integrated with GitHub Actions?

[normanj-bitquill]

Thanks for adding these. I have a few questions: 1 Could you explain how the queries and results are generated? 2 If a new command is added, how should it be incorporated into the existing test sets? 3 Are these tests already integrated with GitHub Actions?

@penghuo

1. The queries were taken from here and here

   The results were generated by running the queries on Spark and saving the results in CSV format. This creates a baseline that enables changes to the results to be detected.

2. To add new tests, create the PPL files in e2e-test/src/test/resources/spark/queries/ppl

   Generate the results by running the queries on the Spark container when the integration test docker cluster is running and save the results in CSV format. The query and results have the same base filename. The query file ends with .ppl and the results file ends with .results.

3. This is not yet integrated with GitHub actions, but that work is planned. After this PR, there will be another to add Async Query API tests. Once all of those tests are in, I will work on using these tests in the GitHub actions.

[YANG-DB]

@normanj-bitquill do you think we can add another paragraph (in our E2E document) on how to add a new query to this baseline ?

[penghuo]

Thx!

[normanj-bitquill]

@YANG-DB I'll update the README.md

[normanj-bitquill]

@YANG-DB I have updated the README.md.