Add Pathways Recipe Support for Scale Testing

[SujeethJinesh]

Description

This PR contains the following changes:

1. Adds the following 2 recipes:

   a. Benchmarking Pathways and McJAX code
   b. Long running test on Pathways.

   This is done to improve code/recipe sharing between developers during testing and make benchmarking simpler by
   sharing a single, well-defined config. This will be used for testing and verification on v6e capacity.

2. By default allows all configs to run pathways (enable_single_controller, disable zarr3, enable_pathways_goodput etc.), with additional configs that can be added by the user with pathways_tuning_params.

3. Allow removing problematic XLA flags for pathways with pathways_xla_flag_options.

4. Cleans up a lot of old pathways specific configs.

Tests

Run multiple rounds on multiple v6e clusters

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.