Add local tests with Kind

[IrvingMg]

Fixes / Features

• Related to Create integration test using Kind #267. As a follow-up to Integrate kind for local testing #242, this PR introduces the --kind-cluster flag to most commands in XPK to enable local testing. A script has been included to demonstrate that most of the tests in build_tests.yaml, which use a real cluster, can also be tested locally.

Testing / Documentation

Output from the script running tests on Kind:

./tools/run-kind-tests.sh
======================= Start of 'Create a cluster' =======================
+ python xpk.py kind create --cluster xpk-kind-test
[XPK] Starting xpk
[XPK] Starting cluster create for cluster xpk-kind-test:
[XPK] Task: `Find if Cluster Exists` is implemented by `kind get clusters`, hiding output unless there is an error.
[XPK] Task: `Kind Cluster Create` is implemented by `kind create cluster --config /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmpls1bmook --name=xpk-kind-test`, streaming output live.
[XPK] Waiting for `Kind Cluster Create`, for 0 seconds
Creating cluster "xpk-kind-test" ...
 ✓ Ensuring node image (kindest/node:v1.32.0) 🖼
⠈⠱ Preparing nodes 📦 📦  [XPK] Waiting for `Kind Cluster Create`, for 1 seconds
⢎⠁ Preparing nodes 📦 📦  [XPK] Waiting for `Kind Cluster Create`, for 2 seconds
⢆⡱ Preparing nodes 📦 📦  [XPK] Waiting for `Kind Cluster Create`, for 3 seconds
⠈⠱ Preparing nodes 📦 📦  [XPK] Waiting for `Kind Cluster Create`, for 4 seconds
 ✓ Preparing nodes 📦 📦  
⠈⠁ Writing configuration 📜 [XPK] Waiting for `Kind Cluster Create`, for 5 seconds
 ✓ Writing configuration 📜 
⢄⡱ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 6 seconds
⠈⠑ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 7 seconds
⢎⡀ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 8 seconds
⢄⡱ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 9 seconds
⠈⠑ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 10 seconds
⢎⡀ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 11 seconds
⢄⡱ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 12 seconds
⠈⠑ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 13 seconds
⢎⡀ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 14 seconds
⢄⡱ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 15 seconds
⠈⠑ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 16 seconds
⢎⡀ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 17 seconds
⢄⡱ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 18 seconds
⠈⠑ Starting control-plane 🕹️ [XPK] Waiting for `Kind Cluster Create`, for 19 seconds
 ✓ Starting control-plane 🕹️ 
⠈⠁ Installing CNI 🔌 [XPK] Waiting for `Kind Cluster Create`, for 20 seconds
 ✓ Installing CNI 🔌 
⠈⡱ Installing StorageClass 💾 [XPK] Waiting for `Kind Cluster Create`, for 21 seconds
 ✓ Installing StorageClass 💾 
⢀⡱ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 22 seconds
⠊⠁ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 23 seconds
⢎⡰ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 24 seconds
⢀⡱ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 25 seconds
⠊⠁ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 26 seconds
⢎⡠ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 27 seconds
⢄⡱ Joining worker nodes 🚜 [XPK] Waiting for `Kind Cluster Create`, for 28 seconds
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-xpk-kind-test"
You can now use your cluster with:

kubectl cluster-info --context kind-xpk-kind-test

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
[XPK] Task: `Kind Cluster Create` terminated with code `0`
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Enabling the jobset API on our cluster, to be deprecated when Jobset is globally available
[XPK] Try 1: Install Jobset on xpk-kind-test
[XPK] Task: `Install Jobset on xpk-kind-test` is implemented by `kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.7.2/manifests.yaml`, streaming output live.
[XPK] Waiting for `Install Jobset on xpk-kind-test`, for 0 seconds
[XPK] Waiting for `Install Jobset on xpk-kind-test`, for 1 seconds
namespace/jobset-system serverside-applied
customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied
serviceaccount/jobset-controller-manager serverside-applied
role.rbac.authorization.k8s.io/jobset-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied
rolebinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied
[XPK] Waiting for `Install Jobset on xpk-kind-test`, for 2 seconds
clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied
secret/jobset-webhook-server-cert serverside-applied
service/jobset-controller-manager-metrics-service serverside-applied
service/jobset-webhook-service serverside-applied
deployment.apps/jobset-controller-manager serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied
[XPK] Task: `Install Jobset on xpk-kind-test` terminated with code `0`
[XPK] Enabling Kueue on the cluster
[XPK] Task: `Get kueue version on server` is implemented by `kubectl kueue version`, hiding output unless there is an error.
[XPK] Try 1: Set Kueue On Cluster
[XPK] Task: `Set Kueue On Cluster` is implemented by `kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.10.0/manifests.yaml`, streaming output live.
[XPK] Waiting for `Set Kueue On Cluster`, for 0 seconds
[XPK] Waiting for `Set Kueue On Cluster`, for 1 seconds
namespace/kueue-system serverside-applied
customresourcedefinition.apiextensions.k8s.io/admissionchecks.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/clusterqueues.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/cohorts.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/localqueues.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/multikueueclusters.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/multikueueconfigs.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/provisioningrequestconfigs.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/resourceflavors.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/topologies.kueue.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/workloadpriorityclasses.kueue.x-k8s.io serverside-applied
[XPK] Waiting for `Set Kueue On Cluster`, for 2 seconds
customresourcedefinition.apiextensions.k8s.io/workloads.kueue.x-k8s.io serverside-applied
serviceaccount/kueue-controller-manager serverside-applied
role.rbac.authorization.k8s.io/kueue-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-batch-admin-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-batch-user-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-clusterqueue-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-clusterqueue-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-job-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-job-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-jobset-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-jobset-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-localqueue-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-localqueue-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-mpijob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-mpijob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-mxjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-mxjob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-paddlejob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-paddlejob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-pending-workloads-cq-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-pending-workloads-lq-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-proxy-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-pytorchjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-pytorchjob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-raycluster-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-raycluster-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-rayjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-rayjob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-resourceflavor-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-resourceflavor-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-tfjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-tfjob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-workload-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-workload-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-xgboostjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/kueue-xgboostjob-viewer-role serverside-applied
rolebinding.rbac.authorization.k8s.io/kueue-visibility-server-auth-reader serverside-applied
rolebinding.rbac.authorization.k8s.io/kueue-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kueue-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kueue-proxy-rolebinding serverside-applied
configmap/kueue-manager-config serverside-applied
secret/kueue-webhook-server-cert serverside-applied
service/kueue-controller-manager-metrics-service serverside-applied
service/kueue-visibility-server serverside-applied
service/kueue-webhook-service serverside-applied
deployment.apps/kueue-controller-manager serverside-applied
apiservice.apiregistration.k8s.io/v1beta1.visibility.kueue.x-k8s.io serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/kueue-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/kueue-validating-webhook-configuration serverside-applied
[XPK] Task: `Set Kueue On Cluster` terminated with code `0`
[XPK] Verifying kjob installation
[XPK] Task: `Verify kjob installation ` is implemented by `kubectl-kjob help`, hiding output unless there is an error.
[XPK] kjob found
[XPK] Applying kjob CDRs
[XPK] Task: `Create kjob CRDs on cluster` is implemented by `kubectl kjob printcrds | kubectl apply --server-side -f -`, streaming output live.
[XPK] Waiting for `Create kjob CRDs on cluster`, for 0 seconds
customresourcedefinition.apiextensions.k8s.io/applicationprofiles.kjobctl.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/jobtemplates.kjobctl.x-k8s.io serverside-applied
[XPK] Waiting for `Create kjob CRDs on cluster`, for 1 seconds
customresourcedefinition.apiextensions.k8s.io/rayclustertemplates.kjobctl.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/rayjobtemplates.kjobctl.x-k8s.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/volumebundles.kjobctl.x-k8s.io serverside-applied
[XPK] Task: `Create kjob CRDs on cluster` terminated with code `0`
[XPK] Creating kjob CRDs succeeded
[XPK] Preparing kjob
[XPK] Task: `Creating JobTemplate` is implemented by `kubectl apply -f /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmpp4vfscfd`, streaming output live.
[XPK] Waiting for `Creating JobTemplate`, for 0 seconds
[XPK] Waiting for `Creating JobTemplate`, for 1 seconds
[XPK] Waiting for `Creating JobTemplate`, for 2 seconds
jobtemplate.kjobctl.x-k8s.io/xpk-def-batch created
[XPK] Task: `Creating JobTemplate` terminated with code `0`
[XPK] Task: `Creating PodTemplate` is implemented by `kubectl apply -f /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmp3kdc0fu7`, streaming output live.
[XPK] Waiting for `Creating PodTemplate`, for 0 seconds
podtemplate/xpk-def-pod created
[XPK] Task: `Creating PodTemplate` terminated with code `0`
[XPK] Task: `Creating AppProfile` is implemented by `kubectl apply -f /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmp66wry1vk`, streaming output live.
[XPK] Waiting for `Creating AppProfile`, for 0 seconds
applicationprofile.kjobctl.x-k8s.io/xpk-def-app-profile created
[XPK] Task: `Creating AppProfile` terminated with code `0`
[XPK] Wait for Kueue to be fully available
[XPK] Task: `Wait for Kueue to be available` is implemented by `kubectl wait deploy/kueue-controller-manager -nkueue-system --for=condition=available --timeout=5m`, streaming output live.
[XPK] Waiting for `Wait for Kueue to be available`, for 0 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 1 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 2 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 3 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 4 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 5 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 6 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 7 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 8 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 9 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 10 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 11 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 12 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 13 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 14 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 15 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 16 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 17 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 18 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 19 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 20 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 21 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 22 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 23 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 24 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 25 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 26 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 27 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 28 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 29 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 30 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 31 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 32 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 33 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 34 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 35 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 36 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 37 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 38 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 39 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 40 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 41 seconds
[XPK] Waiting for `Wait for Kueue to be available`, for 42 seconds
deployment.apps/kueue-controller-manager condition met
[XPK] Task: `Wait for Kueue to be available` terminated with code `0`
[XPK] Install Kueue Custom Resources
[XPK] Try 1: Applying Kueue Custom Resources
[XPK] Task: `Applying Kueue Custom Resources` is implemented by `kubectl apply -f /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmpkcov2q5p`, streaming output live.
[XPK] Waiting for `Applying Kueue Custom Resources`, for 0 seconds
resourceflavor.kueue.x-k8s.io/kind created
clusterqueue.kueue.x-k8s.io/cluster-queue created
localqueue.kueue.x-k8s.io/multislice-queue created
priorityclass.scheduling.k8s.io/very-low created
priorityclass.scheduling.k8s.io/low created
priorityclass.scheduling.k8s.io/medium created
priorityclass.scheduling.k8s.io/high created
priorityclass.scheduling.k8s.io/very-high created
[XPK] Task: `Applying Kueue Custom Resources` terminated with code `0`
[XPK] Kind commands done! Resources are created.
[XPK] Exiting XPK cleanly
+ log_group_end 'Create a cluster'
+ set +x
======================= End of 'Create a cluster' =======================

======================= Start of 'List out the nodepools on the cluster' =======================
+ python xpk.py cluster describe --kind-cluster --cluster xpk-kind-test
[XPK] Starting xpk
[XPK] Starting nodepool list for cluster: xpk-kind-test
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `Nodepool list` is implemented by `kubectl get node --no-headers=true -o custom-columns='NODEPOOL:.metadata.labels.cloud\.google\.com/gke-nodepool' | grep -v 'none' | sort | uniq`, hiding output unless there is an error.
[XPK] Task: `Count nodes per nodepool slice` is implemented by `kubectl get node --no-headers=true -o custom-columns=':metadata.labels.cloud\.google\.com/gke-nodepool' | grep -v 'none' | sort | uniq -c`, hiding output unless there is an error.
[XPK] Task: `Instance type of nodepools` is implemented by `kubectl get node --no-headers=true -o custom-columns='NODEPOOL:.metadata.labels.cloud\.google\.com/gke-nodepool, TYPE:.metadata.labels.node\.kubernetes\.io/instance-type' | grep -v 'none' | sort | uniq`, hiding output unless there is an error.
[XPK] Task: `Count expected healthy nodes per nodepool` is implemented by `kubectl get node --no-headers=true -o custom-columns=':metadata.labels.cloud\.google\.com/gke-nodepool' | grep -v 'none' | sort | uniq -c`, hiding output unless there is an error.
[XPK] Task: `Count actual healthy nodes per nodepool` is implemented by `kubectl get node --no-headers=true -o custom-columns='NODE_NAME:metadata.name, READY_STATUS:.status.conditions[?(@.type=="Ready")].status, NODEPOOL:metadata.labels.cloud\.google\.com/gke-nodepool'  | grep -w True | grep -v 'none' | awk {'print $3'} | sort | uniq -c`, hiding output unless there is an error.
[XPK] Task: `Count total nodes per nodepool` is implemented by `kubectl get node --no-headers=true -o custom-columns='NODE_NAME:metadata.name, READY_STATUS:.status.conditions[?(@.type=="Ready")].status, NODEPOOL:metadata.labels.cloud\.google\.com/gke-nodepool' | grep -v 'none' | awk {'print $3'} | sort | uniq -c`, hiding output unless there is an error.
[XPK] Nodepools info:
 NODEPOOL_NAME      SLICE    TYPE    EXPECTED_HEALTHY_NODES    ACTUAL_HEALTHY_NODES
kind-pool-0            1       1                         1                       1
[XPK] Task: `Count TPU Nodes` is implemented by `kubectl get node --no-headers=true --selector='cloud.google.com/gke-tpu-accelerator' | wc -l`, hiding output unless there is an error.
[XPK] Task: `Count TPU Pods` is implemented by `kubectl get pod -o=custom-columns='Status:.status.phase' | grep -i Running | wc -l`, hiding output unless there is an error.
[XPK] The cluster contains 0 TPUVMs of which 0 are in use.
[XPK] GKE commands done!

[XPK] Exiting XPK cleanly
+ log_group_end 'List out the nodepools on the cluster'
+ set +x
======================= End of 'List out the nodepools on the cluster' =======================

======================= Start of 'Run a base-docker-image workload' =======================
+ python xpk.py workload create --kind-cluster --cluster xpk-kind-test --workload xpk-wl-test --command 'echo "Hello world!"' --docker-image=ubuntu
[XPK] Starting xpk
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `Check if Workload Already Exists` is implemented by `kubectl get workloads -o=custom-columns='Jobset:.metadata.ownerReferences[0].name'`, hiding output unless there is an error.
[XPK] Starting workload create
[XPK] Task: `GKE Cluster Get ConfigMap` is implemented by `kubectl get configmap xpk-kind-test-resources-configmap -o=custom-columns="ConfigData:data" --no-headers=true`, hiding output unless there is an error.
[XPK] Task GKE Cluster Get ConfigMap failed with 1
[XPK] ********************************************************************************
[XPK] b'Error from server (NotFound): configmaps "xpk-kind-test-resources-configmap" not found\n'
[XPK] ********************************************************************************
[XPK] GKE Cluster Get ConfigMap request returned ERROR 1
[XPK] No ConfigMap exist for cluster with the name xpk-kind-test-resources-configmap.
[XPK] Starting workload create
[XPK] Task: `GKE Cluster Get ConfigMap` is implemented by `kubectl get configmap xpk-kind-test-metadata-configmap -o=custom-columns="ConfigData:data" --no-headers=true`, hiding output unless there is an error.
[XPK] Task GKE Cluster Get ConfigMap failed with 1
[XPK] ********************************************************************************
[XPK] b'Error from server (NotFound): configmaps "xpk-kind-test-metadata-configmap" not found\n'
[XPK] ********************************************************************************
[XPK] GKE Cluster Get ConfigMap request returned ERROR 1
[XPK] Warning: Unable to find ConfigMap: xpk-kind-test-metadata-configmap for the cluster. We recommend to upgrade your cluster by running `xpk cluster create`.
[XPK] Task: `GKE Cluster Get ConfigMap` is implemented by `kubectl get configmap xpk-kind-test-resources-configmap -o=custom-columns="ConfigData:data" --no-headers=true`, hiding output unless there is an error.
[XPK] Task GKE Cluster Get ConfigMap failed with 1
[XPK] ********************************************************************************
[XPK] b'Error from server (NotFound): configmaps "xpk-kind-test-resources-configmap" not found\n'
[XPK] ********************************************************************************
[XPK] GKE Cluster Get ConfigMap request returned ERROR 1
[XPK] Unable to find config map: xpk-kind-test-resources-configmap. Autoprovisioning is not enabled.
[XPK] Task: `Creating Workload` is implemented by `kubectl apply -f /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmp0zbbt5_4`, streaming output live.
[XPK] Waiting for `Creating Workload`, for 0 seconds
jobset.jobset.x-k8s.io/xpk-wl-test created
[XPK] Task: `Creating Workload` terminated with code `0`
[XPK] Exiting XPK cleanly
+ log_group_end 'Run a base-docker-image workload'
+ set +x
======================= End of 'Run a base-docker-image workload' =======================

======================= Start of 'Run xpk inspector with the workload created above' =======================
+ python xpk.py inspector --kind-cluster --cluster xpk-kind-test --workload xpk-wl-test
[XPK] Starting xpk
[XPK] Namespace(xpk_subcommands='inspector', func=<function inspector at 0x107b862a0>, xpk_inspector_subcommands=None, cluster='xpk-kind-test', project=None, zone=None, dry_run=False, kind_cluster=True, workload='xpk-wl-test', print_to_terminal=False, enable_ray_cluster=False)
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `Kubectl: All Nodes` is implemented by `kubectl get node -o custom-columns='NODE_NAME:metadata.name, READY_STATUS:.status.conditions[?(@.type=="Ready")].status, NODEPOOL:metadata.labels.cloud\.google\.com/gke-nodepool'`, hiding output unless there is an error.
[XPK] Task: `Kubectl: Number of Nodes per Node Pool` is implemented by `kubectl get node -o custom-columns=':metadata.labels.cloud\.google\.com/gke-nodepool' | sort | uniq -c`, hiding output unless there is an error.
[XPK] Task: `Kubectl: Healthy Node Count Per Node Pool` is implemented by `kubectl get node -o custom-columns='NODE_NAME:metadata.name, READY_STATUS:.status.conditions[?(@.type=="Ready")].status, NODEPOOL:metadata.labels.cloud\.google\.com/gke-nodepool' | grep -w True | awk {'print $3'} | sort | uniq -c`, hiding output unless there is an error.
[XPK] Task: `Kueue: ClusterQueue Details` is implemented by `kubectl describe ClusterQueue cluster-queue`, hiding output unless there is an error.
[XPK] Task: `Kueue: LocalQueue Details` is implemented by `kubectl describe LocalQueue multislice-queue`, hiding output unless there is an error.
[XPK] Task: `Kueue: ResourceFlavor Details` is implemented by `kubectl describe ResourceFlavor`, hiding output unless there is an error.
[XPK] Task: `Kueue: Kueue Deployment Details` is implemented by `kubectl describe Deployment kueue-controller-manager -n kueue-system`, hiding output unless there is an error.
[XPK] Task: `Jobset: Deployment Details` is implemented by `kubectl describe Deployment jobset-controller-manager -n jobset-system`, hiding output unless there is an error.
[XPK] Task: `Kueue Manager Logs` is implemented by `kubectl logs deployment/kueue-controller-manager -n kueue-system --tail=100 --prefix=True`, hiding output unless there is an error.
[XPK] Task: `Jobset Manager Logs` is implemented by `kubectl logs deployment/jobset-controller-manager -n jobset-system --tail=100 --prefix=True`, hiding output unless there is an error.
[XPK] Task: `List Jobs with filter-by-status=EVERYTHING with filter-by-job=None` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"  `, hiding output unless there is an error.
[XPK] Task: `List Jobs with filter-by-status=QUEUED with filter-by-job=None` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"  | awk -e 'NR == 1 || ($7 ~ "Admitted|Evicted|QuotaReserved" && ($5 ~ "<none>" || $5 == 0)) {print $0}' `, hiding output unless there is an error.
[XPK] Task: `List Jobs with filter-by-status=RUNNING with filter-by-job=None` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"  | awk -e 'NR == 1 || ($7 ~ "Admitted|Evicted" && $5 ~ /^[0-9]+$/ && $5 > 0) {print $0}' `, hiding output unless there is an error.
[XPK] xpk-wl-test
[XPK] Task: `List Jobs with filter-by-status=EVERYTHING with filter-by-job=xpk-wl-test` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"   | awk -e 'NR == 1 || $1 ~ "xpk-wl-test" {print $0}'`, hiding output unless there is an error.
[XPK] Task: `Jobset config for xpk-wl-test` is implemented by `kubectl describe jobsets xpk-wl-test`, hiding output unless there is an error.
[XPK] Task: `Workload config for xpk-wl-test` is implemented by `kubectl describe workloads jobset-xpk-wl-test`, hiding output unless there is an error.
[XPK] Find xpk inspector output file: /var/folders/w_/9tldjpt90233ytcg3hycwns40000gp/T/tmpodod_jrz
[XPK] Exiting XPK cleanly
+ log_group_end 'Run xpk inspector with the workload created above'
+ set +x
======================= End of 'Run xpk inspector with the workload created above' =======================

======================= Start of 'Wait for workload completion and confirm it succeeded' =======================
+ python xpk.py workload list --kind-cluster --cluster xpk-kind-test --wait-for-job-completion xpk-wl-test --timeout 300
[XPK] Starting xpk
[XPK] Namespace(xpk_subcommands='workload', func=<function workload_list at 0x107986160>, xpk_workload_subcommands='list', cluster='xpk-kind-test', filter_by_status='EVERYTHING', filter_by_job=None, wait_for_job_completion='xpk-wl-test', timeout=300, project=None, zone=None, dry_run=False, kind_cluster=True, enable_ray_cluster=False)
[XPK] Starting workload list
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `Check if Workload Already Exists` is implemented by `kubectl get workloads -o=custom-columns='Jobset:.metadata.ownerReferences[0].name'`, hiding output unless there is an error.
[XPK] Task: `Get full workload name` is implemented by `kubectl get workloads | grep jobset-xpk-wl-test`, hiding output unless there is an error.
[XPK] Task: `Wait for workload to finish with timeout of 300s` is implemented by `kubectl  wait --for jsonpath='.status.conditions[-1].type'=Finished workload jobset-xpk-wl-test-b8f1f --timeout=300s`
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 0 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 1 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 2 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 3 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 4 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 5 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 6 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 7 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 8 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 9 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 10 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 11 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 12 seconds
[XPK] Waiting for `Wait for workload to finish with timeout of 300s`, for 13 seconds
[XPK] Task: `Wait for workload to finish with timeout of 300s` terminated with code `0`
[XPK] Task: `Get jobset status` is implemented by `kubectl get jobset xpk-wl-test -o jsonpath='{.status.conditions[-1].type}'`, hiding output unless there is an error.
[XPK] Your workload finished with status: Completed
[XPK] Task: `List Jobs with filter-by-status=EVERYTHING with filter-by-job=xpk-wl-test` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"   | awk -e 'NR == 1 || $1 ~ "xpk-wl-test" {print $0}'`, hiding output unless there is an error.
[XPK] Workload List Output:
awk: unknown option -e ignored

Jobset Name   Created Time           Priority   TPU VMs Needed   TPU VMs Running/Ran   TPU VMs Done   Status     Status Message                  Status Time
xpk-wl-test   2025-02-05T19:32:10Z   medium     1                1                     <none>         Finished   jobset completed successfully   2025-02-05T19:32:28Z

[XPK] Exiting XPK cleanly
+ log_group_end 'Wait for workload completion and confirm it succeeded'
+ set +x
======================= End of 'Wait for workload completion and confirm it succeeded' =======================

======================= Start of 'List out the workloads on the cluster' =======================
+ python xpk.py workload list --kind-cluster --cluster xpk-kind-test
[XPK] Starting xpk
[XPK] Namespace(xpk_subcommands='workload', func=<function workload_list at 0x102c82160>, xpk_workload_subcommands='list', cluster='xpk-kind-test', filter_by_status='EVERYTHING', filter_by_job=None, wait_for_job_completion=None, timeout=None, project=None, zone=None, dry_run=False, kind_cluster=True, enable_ray_cluster=False)
[XPK] Starting workload list
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `List Jobs with filter-by-status=EVERYTHING with filter-by-job=None` is implemented by `kubectl get workloads -o=custom-columns="Jobset Name:.metadata.ownerReferences[0].name,Created Time:.metadata.creationTimestamp,Priority:.spec.priorityClassName,TPU VMs Needed:.spec.podSets[0].count,TPU VMs Running/Ran:.status.admission.podSetAssignments[-1].count,TPU VMs Done:.status.reclaimablePods[0].count,Status:.status.conditions[-1].type,Status Message:.status.conditions[-1].message,Status Time:.status.conditions[-1].lastTransitionTime"  `, hiding output unless there is an error.
[XPK] Workload List Output:
Jobset Name   Created Time           Priority   TPU VMs Needed   TPU VMs Running/Ran   TPU VMs Done   Status     Status Message                  Status Time
xpk-wl-test   2025-02-05T19:32:10Z   medium     1                1                     <none>         Finished   jobset completed successfully   2025-02-05T19:32:28Z

[XPK] Exiting XPK cleanly
+ log_group_end 'List out the workloads on the cluster'
+ set +x
======================= End of 'List out the workloads on the cluster' =======================

======================= Start of 'Run xpk info' =======================
+ python xpk.py info --kind-cluster --cluster xpk-kind-test
[XPK] Starting xpk
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Veryfing kueuectl installation
[XPK] Task: `Verify kueuectl installation on cluster` is implemented by `kubectl kueue version`, hiding output unless there is an error.
[XPK] kueuectl found
[XPK] Task: `list localqueue` is implemented by `kubectl kueue list localqueue -o json`, hiding output unless there is an error.
[XPK] Task: `list clusterqueue` is implemented by `kubectl kueue list clusterqueue -o json`, hiding output unless there is an error.
[XPK] Local Queues usage 
 QUEUE               ADMITTED_WORKLOADS    PENDING_WORKLOADS
multislice-queue                     0                    0
[XPK] Cluster Queues usage 
 QUEUE            ADMITTED_WORKLOADS    PENDING_WORKLOADS
cluster-queue                     0                    0
[XPK] XPK Done.
+ log_group_end 'Run xpk info'
+ set +x
======================= End of 'Run xpk info' =======================

======================= Start of 'Delete the workload on the cluster' =======================
+ python xpk.py workload delete --kind-cluster --cluster xpk-kind-test --workload xpk-wl-test
[XPK] Starting xpk
[XPK] Starting Workload delete
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `Delete Workload` is implemented by `kubectl delete jobset xpk-wl-test -n default`, streaming output live.
[XPK] Waiting for `Delete Workload`, for 0 seconds
jobset.jobset.x-k8s.io "xpk-wl-test" deleted
[XPK] Task: `Delete Workload` terminated with code `0`
[XPK] Exiting XPK cleanly
+ log_group_end 'Delete the workload on the cluster'
+ set +x
======================= End of 'Delete the workload on the cluster' =======================

======================= Start of 'Create test script to execute in batch' =======================
+ echo -e '#!/bin/bash \n#SBATCH --unknown-flag=value\n echo "Hello world from a test script!"'
+ log_group_end 'Create test script to execute in batch'
+ set +x
======================= End of 'Create test script to execute in batch' =======================

======================= Start of 'Run a batch job on the cluster' =======================
+ python xpk.py batch --kind-cluster --cluster xpk-kind-test batch.sh --ignore-unknown-flags --array 1-5 --nodes 2 --ntasks 3 --time 60
[XPK] Starting xpk
[XPK] Task: `switch to cluster xpk-kind-test` is implemented by `kubectl config use-context kind-xpk-kind-test --namespace=default`, streaming output live.
[XPK] Waiting for `switch to cluster xpk-kind-test`, for 0 seconds
Switched to context "kind-xpk-kind-test".
[XPK] Task: `switch to cluster xpk-kind-test` terminated with code `0`
[XPK] Task: `submit job` is implemented by `kubectl kjob create slurm --profile xpk-def-app-profile --localqueue multislice-queue --ignore-unknown-flags -- batch.sh --partition multislice-queue --array 1-5 --nodes 2 --ntasks 3 --time 60`, hiding output unless there is an error.
[XPK] XPK Done.
+ log_group_end 'Run a batch job on the cluster'
+ set +x
======================= End of 'Run a batch job on the cluster' =======================

======================= Start of 'List out the jobs on the cluster' =======================
+ python xpk.py job ls --kind-cluster --cluster xpk-kind-test
+ grep xpk-def-app-profile-slurm-
xpk-def-app-profile-slurm-kchzj   xpk-def-app-profile   multislice-queue   0/2           1s         1s
+ log_group_end 'List out the jobs on the cluster'
+ set +x
======================= End of 'List out the jobs on the cluster' =======================

======================= Start of 'Get created job name' =======================
+ python xpk.py job ls --kind-cluster --cluster xpk-kind-test
+ grep xpk-def-app-profile-slurm-
+ head -1
+ awk '{print $1}'
+ JOB_NAME=xpk-def-app-profile-slurm-kchzj
+ log_group_end 'Get created job name'
+ set +x
======================= End of 'Get created job name' =======================

======================= Start of 'Check created job' =======================
+ kubectl get job xpk-def-app-profile-slurm-kchzj -o 'jsonpath={.metadata.labels}'
+ grep '"kueue.x-k8s.io/max-exec-time-seconds":"3600"'
{"kjobctl.x-k8s.io/mode":"Slurm","kjobctl.x-k8s.io/profile":"xpk-def-app-profile","kueue.x-k8s.io/max-exec-time-seconds":"3600","kueue.x-k8s.io/queue-name":"multislice-queue"}
+ kubectl get job xpk-def-app-profile-slurm-kchzj -o 'jsonpath={.spec}'
+ job_spec='{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}'
+ echo '{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-+ grep '"completions":2'
env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}'
{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}
+ grep '"parallelism":2'
+ echo '{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}'
{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}
+ echo '{"backoffLimit":6,"completionMode":"Indexed","completions":2,"manualSelector":false,"parallelism":2,"podReplacementPolicy":"TerminatingOrFailed","selector":{"matchLabels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8"}},"suspend":false,"template":{"metadata":{"creationTimestamp":null,"labels":{"batch.kubernetes.io/controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","batch.kubernetes.io/job-name":"xpk-def-app-profile-slurm-kchzj","controller-uid":"a4678278-ebf7-4f1f-9b89-b2471a016cb8","job-name":"xpk-def-app-profile-slurm-kchzj"}},"spec":{"containers":[{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"0"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-0","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"1"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-1","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]},{"command":["bash","/slurm/scripts/entrypoint.sh"],"env":[{"name":"USER_ID","value":"Irving_Mondragon"},{"name":"TASK_NAME","value":"default_xpk-def-app-profile"},{"name":"TASK_ID","value":"Irving_Mondragon_2025-02-05T20:32:35+01:00_default_xpk-def-app-profile"},{"name":"PROFILE","value":"default_xpk-def-app-profile"},{"name":"TIMESTAMP","value":"2025-02-05T20:32:35+01:00"},{"name":"JOB_CONTAINER_INDEX","value":"2"}],"image":"ubuntu:22.04","imagePullPolicy":"IfNotPresent","name":"xpk-batch-container-2","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"dnsPolicy":"ClusterFirst","initContainers":[{"command":["sh","/slurm/scripts/init-entrypoint.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"image":"registry.k8s.io/busybox:1.27.2","imagePullPolicy":"IfNotPresent","name":"slurm-init-env","resources":{},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/slurm/scripts","name":"slurm-scripts"},{"mountPath":"/slurm/env","name":"slurm-env"}]}],"restartPolicy":"OnFailure","schedulerName":"default-scheduler","securityContext":{},"subdomain":"xpk-def-app-profile-slurm-kchzj","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"init-entrypoint.sh","path":"init-entrypoint.sh"},{"key":"entrypoint.sh","path":"entrypoint.sh"},{"key":"script","mode":493,"path":"script"}],"name":"xpk-def-app-profile-slurm-kchzj"},"name":"slurm-scripts"},{"emptyDir":{},"name":"slurm-env"}]}}}'
+ jq '.template.spec.containers | length'
+ grep 3
3
+ log_group_end 'Check created job'
+ set +x
======================= End of 'Check created job' =======================

======================= Start of 'Get job info for the last job created on the cluster' =======================
+ python xpk.py job info --kind-cluster xpk-def-app-profile-slurm-kchzj
+ grep -e 'Entrypoint environment variables template:' -e 'Job name:' -e Labels: -e Mounts: -e Pods: -e Profile: -e 'Script name:'
+ wc -l
+ grep 7
       7
+ log_group_end 'Get job info for the last job created on the cluster'
+ set +x
======================= End of 'Get job info for the last job created on the cluster' =======================

======================= Start of 'Cancel the batch job on the cluster' =======================
+ python xpk.py job cancel xpk-def-app-profile-slurm-kchzj --kind-cluster --cluster xpk-kind-test
+ grep 'job.batch/xpk-def-app-profile-slurm-kchzj deleted'
job.batch/xpk-def-app-profile-slurm-kchzj deleted
+ log_group_end 'Cancel the batch job on the cluster'
+ set +x
======================= End of 'Cancel the batch job on the cluster' =======================

======================= Start of 'Create shell and exit it immediately' =======================
+ cat
+ chmod +x ./create-shell.exp
+ expect ./create-shell.exp
spawn python ./xpk.py shell
[XPK] Starting xpk
[XPK] Task: `Get existing interactive shell pod name.` is implemented by `kubectl get pods --no-headers --field-selector status.phase=Running -o custom-columns=":metadata.name"`, hiding output unless there is an error.
[XPK] Task: `Creating new interactive shell and entering it` is implemented by `kubectl-kjob create interactive --profile xpk-def-app-profile --pod-running-timeout 30s`. Streaming output and input live.
[XPK] To exit the shell input "exit".
pod/xpk-def-app-profile-interactive-tckk6 created
waiting for pod "xpk-def-app-profile-interactive-tckk6" to be running...
+ log_group_end 'Create shell and exit it immediately'
+ set +x
======================= End of 'Create shell and exit it immediately' =======================

======================= Start of 'Stop the shell' =======================
+ python xpk.py shell stop
[XPK] Starting xpk
[XPK] Task: `Get existing interactive shell pod name.` is implemented by `kubectl get pods --no-headers --field-selector status.phase=Running -o custom-columns=":metadata.name"`, hiding output unless there is an error.
[XPK] Task: `Deleting the existing shell.` is implemented by `kubectl delete pod xpk-def-app-profile-interactive-tckk6`, streaming output live.
[XPK] Waiting for `Deleting the existing shell.`, for 0 seconds
pod "xpk-def-app-profile-interactive-tckk6" deleted
[XPK] Waiting for `Deleting the existing shell.`, for 1 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 2 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 3 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 4 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 5 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 6 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 7 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 8 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 9 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 10 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 11 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 12 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 13 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 14 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 15 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 16 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 17 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 18 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 19 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 20 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 21 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 22 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 23 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 24 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 25 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 26 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 27 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 28 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 29 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 30 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 31 seconds
[XPK] Waiting for `Deleting the existing shell.`, for 32 seconds
[XPK] Task: `Deleting the existing shell.` terminated with code `0`
[XPK] The shell was deleted successfully.
[XPK] Exiting XPK cleanly
+ log_group_end 'Stop the shell'
+ set +x
======================= End of 'Stop the shell' =======================

[XPK] Starting xpk
[XPK] Starting cluster delete for cluster: xpk-kind-test
[XPK] Task: `Cluster Delete` is implemented by `kind delete cluster --name=xpk-kind-test`, streaming output live.
[XPK] Waiting for `Cluster Delete`, for 0 seconds
Deleting cluster "xpk-kind-test" ...
[XPK] Waiting for `Cluster Delete`, for 1 seconds
[XPK] Waiting for `Cluster Delete`, for 2 seconds
Deleted nodes: ["xpk-kind-test-worker" "xpk-kind-test-control-plane"]
[XPK] Task: `Cluster Delete` terminated with code `0`
[XPK] Kind commands done! Cluster xpk-kind-test deleted.
[XPK] Exiting XPK cleanly

[IrvingMg]

cc: @mbobrovskyi @pawloch00

[pawloch00]

@sharabiani please take a look

[sharabiani]

Your branch is behind from develop branch that will have an impact on your changes.
1- Please sync it with develop branch
2- When synced you'll have new method in core/core setup_k8s_env which I believe will make problems for your testing flow
3- Also get_cluster_credentials in core is a repeatition to commans/common set_cluster_command which needs to be addressed.

I have made some changes to address point no. 3 which I will try to merge it to develop in 1-2 days. Meanwhile you can address point no. 1 and we can address no. 2 after our changes both.