You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using MPI framework and using more nodes in azureml the shared memory (shm_size) is only 2gb by default,
In my usecase, It is giving the memory issue.
I want to increase the shm_size, which was not possible with kedro azureml
But we can pass the shm_size, If we use normal azureml sdk v2 without kedro, while creating the job in the job schema
attaching the example job schema
If I want to have the same in kedro azureml, What is the way?
The text was updated successfully, but these errors were encountered:
@distributed_job(Framework.MPI, num_nodes=2, processes_per_node=1)
When using MPI framework and using more nodes in azureml the shared memory (shm_size) is only 2gb by default,
![image](https://private-user-images.githubusercontent.com/40229861/328948613-ff7507d0-827c-4510-ab56-029be4073788.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkzNzM0ODksIm5iZiI6MTczOTM3MzE4OSwicGF0aCI6Ii80MDIyOTg2MS8zMjg5NDg2MTMtZmY3NTA3ZDAtODI3Yy00NTEwLWFiNTYtMDI5YmU0MDczNzg4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEyVDE1MTMwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRlNzI4MDZmZTFhMWRkZjQ4YTVmMmYyOWEwNDk2OTNjNTUxN2U2NTlmMzM5YjdkYmUwMjk2MmY3ZTQxMDE3ODYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.R3ZtFsoV3AsobeuMNOrybWHXxT_rKk759zHXgIgFqOg)
In my usecase, It is giving the memory issue.
I want to increase the shm_size, which was not possible with kedro azureml
But we can pass the shm_size, If we use normal azureml sdk v2 without kedro, while creating the job in the job schema
![image](https://private-user-images.githubusercontent.com/40229861/328947742-8d9e53f7-67fa-4978-acdb-c57df9525ba4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkzNzM0ODksIm5iZiI6MTczOTM3MzE4OSwicGF0aCI6Ii80MDIyOTg2MS8zMjg5NDc3NDItOGQ5ZTUzZjctNjdmYS00OTc4LWFjZGItYzU3ZGY5NTI1YmE0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEyVDE1MTMwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBkZWFmNjUzNWZhNzFjNTZkMGU3ZTI0MjY3ODgxYjlhY2M5MmMyZGE1YzBhODdlZDY1YTVjZDI0OTM5NzliYmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.yXWoxuvTx0E_eB11loMilba2hwJANwrqg1E1DotX5Fw)
attaching the example job schema
If I want to have the same in kedro azureml, What is the way?
The text was updated successfully, but these errors were encountered: