Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSAN race in //tests:export_back_compat_test_cpu under free-threading #26272

Open
hawkinsp opened this issue Feb 3, 2025 · 2 comments
Open

TSAN race in //tests:export_back_compat_test_cpu under free-threading #26272

hawkinsp opened this issue Feb 3, 2025 · 2 comments
Assignees
@vfdev-5
Labels
bug Something isn't working free threading Issues found in free threading builds

Comments

@hawkinsp
Copy link
Collaborator

hawkinsp commented Feb 3, 2025

Description

This 313t free-threading CI run:
https://github.com/jax-ml/jax/actions/runs/13111654401/job/36576699494?pr=26261

contains this race in MLIR code:

WARNING: ThreadSanitizer: data race (pid=57866)
  Read of size 8 at 0x729000596b98 by thread T64 (mutexes: read M0):
    #0 (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo::isEqual((anonymous namespace)::ParametricStorageUniquer::HashedStorage const&, (anonymous namespace)::ParametricStorageUniquer::HashedStorage const&) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:67:18 (libjaxlib_mlir_capi.so+0x34173cb) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #1 (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo::isEqual((anonymous namespace)::ParametricStorageUniquer::LookupKey const&, (anonymous namespace)::ParametricStorageUniquer::HashedStorage const&) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:70:11 (libjaxlib_mlir_capi.so+0x34173cb)
    #2 bool llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::LookupBucketFor<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::LookupKey const&, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>*&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:654:11 (libjaxlib_mlir_capi.so+0x34173cb)
    #3 std::pair<llvm::DenseMapIterator<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>, false>, bool> llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>(std::pair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty>&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:280:9 (libjaxlib_mlir_capi.so+0x34173cb)
    #4 std::pair<llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::Iterator, bool> llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseSet.h:232:19 (libjaxlib_mlir_capi.so+0x34173cb)
    #5 (anonymous namespace)::ParametricStorageUniquer::getOrCreateUnsafe((anonymous namespace)::ParametricStorageUniquer::Shard&, (anonymous namespace)::ParametricStorageUniquer::LookupKey&, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:95:37 (libjaxlib_mlir_capi.so+0x3414f71) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #6 (anonymous namespace)::ParametricStorageUniquer::getOrCreate(bool, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:140:14 (libjaxlib_mlir_capi.so+0x3414f71)
    #7 mlir::detail::StorageUniquerImpl::getOrCreate(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:270:27 (libjaxlib_mlir_capi.so+0x3414f71)
    #8 mlir::StorageUniquer::getParametricStorageTypeImpl(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:375:16 (libjaxlib_mlir_capi.so+0x3414b4f) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #9 mlir::detail::StringAttrStorage* mlir::StorageUniquer::get<mlir::detail::StringAttrStorage, llvm::StringRef, mlir::NoneType>(llvm::function_ref<void (mlir::detail::StringAttrStorage*)>, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/Support/StorageUniquer.h:218:9 (libjaxlib_mlir_capi.so+0x33bd378) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #10 std::enable_if<!std::is_same<mlir::StringAttr::ImplType, mlir::AttributeStorage>::value, mlir::StringAttr>::type mlir::detail::AttributeUniquer::getWithTypeID<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:239:39 (libjaxlib_mlir_capi.so+0x33bd378)
    #11 mlir::StringAttr mlir::detail::AttributeUniquer::get<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:220:12 (libjaxlib_mlir_capi.so+0x33bd378)
    #12 mlir::StringAttr mlir::detail::StorageUserBase<mlir::StringAttr, mlir::Attribute, mlir::detail::StringAttrStorage, mlir::detail::AttributeUniquer, mlir::TypedAttr::Trait>::get<llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:181:12 (libjaxlib_mlir_capi.so+0x33bd378)
    #13 mlir::StringAttr::get(mlir::MLIRContext*, llvm::Twine const&) /proc/self/cwd/external/llvm-project/mlir/lib/IR/BuiltinAttributes.cpp:275:10 (libjaxlib_mlir_capi.so+0x33bd378)
    #14 mlir::Operation::setAttr(llvm::StringRef, mlir::Attribute) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/Operation.h:594:13 (libjaxlib_mlir_capi.so+0x25d0e57) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #15 mlirOperationSetAttributeByName /proc/self/cwd/external/llvm-project/mlir/lib/CAPI/IR/IR.cpp:683:15 (libjaxlib_mlir_capi.so+0x25d0e57)
    #16 (anonymous namespace)::PyOpAttributeMap::dunderSetItem(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&) /proc/self/cwd/external/llvm-project/mlir/lib/Bindings/Python/IRCore.cpp:2634:5 (_mlir.so+0xc6b97) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #17 void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&)::operator()((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:376:20 (_mlir.so+0xc7685) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #18 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), void, (anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, 0ul, 1ul, 2ul, nanobind::scope, nanobind::name, nanobind::is_method>((anonymous namespace)::PyOpAttributeMap&&, void (*)(nanobind::scope, nanobind::name, nanobind::is_method), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:264:13 (_mlir.so+0xc7685)
    #19 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), void, (anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, 0ul, 1ul, 2ul, nanobind::scope, nanobind::name, nanobind::is_method>((anonymous namespace)::PyOpAttributeMap&&, void (*)(nanobind::scope, nanobind::name, nanobind::is_method), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (_mlir.so+0xc7685)
    #20 nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:915:26 (_mlir.so+0x16e312) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #21 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x2f4627) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #22 vectorcall_unbound /__w/jax/jax/cpython/Objects/typeobject.c:2572:12 (python3.13+0x2f4627)
    #23 vectorcall_method /__w/jax/jax/cpython/Objects/typeobject.c:2603:24 (python3.13+0x2f4627)
    #24 slot_mp_ass_subscript /__w/jax/jax/cpython/Objects/typeobject.c (python3.13+0x2ff750) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #25 PyObject_SetItem /__w/jax/jax/cpython/Objects/abstract.c:232:19 (python3.13+0x1b9488) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #26 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:5777:27 (python3.13+0x3f5ddb) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #27 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #28 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #29 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #30 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef440) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #31 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef440)
    #32 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #33 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #34 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #35 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #36 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #37 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #38 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #39 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1e9f3d) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #40 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1eba37) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #41 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9539:15 (python3.13+0x2f8928) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #42 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea1ac) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #43 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eae08) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #44 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eae08)
    #45 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #46 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #47 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #48 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #49 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #50 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #51 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #52 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #53 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #54 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a72) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #55 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd77) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)

  Previous write of size 8 at 0x729000596b98 by thread T67 (mutexes: read M0):
    #0 (anonymous namespace)::ParametricStorageUniquer::getOrCreateUnsafe((anonymous namespace)::ParametricStorageUniquer::Shard&, (anonymous namespace)::ParametricStorageUniquer::LookupKey&, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:98:15 (libjaxlib_mlir_capi.so+0x3414fbe) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #1 (anonymous namespace)::ParametricStorageUniquer::getOrCreate(bool, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:140:14 (libjaxlib_mlir_capi.so+0x3414fbe)
    #2 mlir::detail::StorageUniquerImpl::getOrCreate(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:270:27 (libjaxlib_mlir_capi.so+0x3414fbe)
    #3 mlir::StorageUniquer::getParametricStorageTypeImpl(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:375:16 (libjaxlib_mlir_capi.so+0x3414b4f) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #4 mlir::detail::StringAttrStorage* mlir::StorageUniquer::get<mlir::detail::StringAttrStorage, llvm::StringRef, mlir::NoneType>(llvm::function_ref<void (mlir::detail::StringAttrStorage*)>, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/Support/StorageUniquer.h:218:9 (libjaxlib_mlir_capi.so+0x33bd378) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #5 std::enable_if<!std::is_same<mlir::StringAttr::ImplType, mlir::AttributeStorage>::value, mlir::StringAttr>::type mlir::detail::AttributeUniquer::getWithTypeID<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:239:39 (libjaxlib_mlir_capi.so+0x33bd378)
    #6 mlir::StringAttr mlir::detail::AttributeUniquer::get<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:220:12 (libjaxlib_mlir_capi.so+0x33bd378)
    #7 mlir::StringAttr mlir::detail::StorageUserBase<mlir::StringAttr, mlir::Attribute, mlir::detail::StringAttrStorage, mlir::detail::AttributeUniquer, mlir::TypedAttr::Trait>::get<llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:181:12 (libjaxlib_mlir_capi.so+0x33bd378)
    #8 mlir::StringAttr::get(mlir::MLIRContext*, llvm::Twine const&) /proc/self/cwd/external/llvm-project/mlir/lib/IR/BuiltinAttributes.cpp:275:10 (libjaxlib_mlir_capi.so+0x33bd378)
    #9 mlir::Operation::setAttr(llvm::StringRef, mlir::Attribute) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/Operation.h:594:13 (libjaxlib_mlir_capi.so+0x25d0e57) (BuildId: 4a2d544bd6db4a8210062f683bf6bb298c5bbcf8)
    #10 mlirOperationSetAttributeByName /proc/self/cwd/external/llvm-project/mlir/lib/CAPI/IR/IR.cpp:683:15 (libjaxlib_mlir_capi.so+0x25d0e57)
    #11 (anonymous namespace)::PyOpAttributeMap::dunderSetItem(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&) /proc/self/cwd/external/llvm-project/mlir/lib/Bindings/Python/IRCore.cpp:2634:5 (_mlir.so+0xc6b97) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #12 void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&)::operator()((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:376:20 (_mlir.so+0xc7685) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #13 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), void, (anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, 0ul, 1ul, 2ul, nanobind::scope, nanobind::name, nanobind::is_method>((anonymous namespace)::PyOpAttributeMap&&, void (*)(nanobind::scope, nanobind::name, nanobind::is_method), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:264:13 (_mlir.so+0xc7685)
    #14 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<(anonymous namespace)::PyOpAttributeMap, void, (anonymous namespace)::PyOpAttributeMap, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, nanobind::scope, nanobind::name, nanobind::is_method>(void ((anonymous namespace)::PyOpAttributeMap::*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'((anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&), void, (anonymous namespace)::PyOpAttributeMap*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, mlir::python::PyAttribute const&, 0ul, 1ul, 2ul, nanobind::scope, nanobind::name, nanobind::is_method>((anonymous namespace)::PyOpAttributeMap&&, void (*)(nanobind::scope, nanobind::name, nanobind::is_method), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (_mlir.so+0xc7685)
    #15 nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:915:26 (_mlir.so+0x16e312) (BuildId: 15beaf78f4e0ba2c1cb26690729f5d8e660c2f73)
    #16 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x2f4627) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #17 vectorcall_unbound /__w/jax/jax/cpython/Objects/typeobject.c:2572:12 (python3.13+0x2f4627)
    #18 vectorcall_method /__w/jax/jax/cpython/Objects/typeobject.c:2603:24 (python3.13+0x2f4627)
    #19 slot_mp_ass_subscript /__w/jax/jax/cpython/Objects/typeobject.c (python3.13+0x2ff750) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #20 PyObject_SetItem /__w/jax/jax/cpython/Objects/abstract.c:232:19 (python3.13+0x1b9488) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #21 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:5777:27 (python3.13+0x3f5ddb) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #22 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #23 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #24 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #25 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef440) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #26 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef440)
    #27 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #28 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #29 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #30 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4902) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #31 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #32 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #33 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #34 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1e9f3d) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #35 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1eba37) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #36 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9539:15 (python3.13+0x2f8928) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #37 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea1ac) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #38 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eae08) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #39 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eae08)
    #40 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e271b) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #41 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de84a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #42 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de84a)
    #43 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #44 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #45 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #46 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #47 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #48 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #49 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a72) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #50 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd77) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)

PTAL?

System info (python version, jaxlib version, accelerator, etc.)

Python 3.13t
@hawkinsp hawkinsp added bug Something isn't working free threading Issues found in free threading builds labels Feb 3, 2025
@vfdev-5
Copy link
Collaborator

vfdev-5 commented Feb 4, 2025 •
edited
Loading

Another similar race report

Click 
WARNING: ThreadSanitizer: data race (pid=468047)
  Read of size 8 at 0x7290000c2168 by thread T72 (mutexes: read M0):
    #0 (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo::isEqual((anonymous namespace)::ParametricStorageUniquer::HashedStorage const&, (anonymous namespace)::ParametricStorageUniquer::HashedStorage const&) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:67:18 (libjaxlib_mlir_capi.so+0x341a3eb) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #1 (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo::isEqual((anonymous namespace)::ParametricStorageUniquer::LookupKey const&, (anonymous namespace)::ParametricStorageUniquer::HashedStorage const&) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:70:11 (libjaxlib_mlir_capi.so+0x341a3eb)
    #2 bool llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::LookupBucketFor<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::LookupKey const&, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>*&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:654:11 (libjaxlib_mlir_capi.so+0x341a3eb)
    #3 std::pair<llvm::DenseMapIterator<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>, false>, bool> llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>(std::pair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty>&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:280:9 (libjaxlib_mlir_capi.so+0x341a3eb)
    #4 std::pair<llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::Iterator, bool> llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseSet.h:232:19 (libjaxlib_mlir_capi.so+0x341a3eb)
    #5 (anonymous namespace)::ParametricStorageUniquer::getOrCreateUnsafe((anonymous namespace)::ParametricStorageUniquer::Shard&, (anonymous namespace)::ParametricStorageUniquer::LookupKey&, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:95:37 (libjaxlib_mlir_capi.so+0x3417f91) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #6 (anonymous namespace)::ParametricStorageUniquer::getOrCreate(bool, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:140:14 (libjaxlib_mlir_capi.so+0x3417f91)
    #7 mlir::detail::StorageUniquerImpl::getOrCreate(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:270:27 (libjaxlib_mlir_capi.so+0x3417f91)
    #8 mlir::StorageUniquer::getParametricStorageTypeImpl(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:375:16 (libjaxlib_mlir_capi.so+0x3417b6f) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #9 mlir::detail::StringAttrStorage* mlir::StorageUniquer::get<mlir::detail::StringAttrStorage, llvm::StringRef, mlir::NoneType>(llvm::function_ref<void (mlir::detail::StringAttrStorage*)>, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/Support/StorageUniquer.h:218:9 (libjaxlib_mlir_capi.so+0x33c0398) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #10 std::enable_if<!std::is_same<mlir::StringAttr::ImplType, mlir::AttributeStorage>::value, mlir::StringAttr>::type mlir::detail::AttributeUniquer::getWithTypeID<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:239:39 (libjaxlib_mlir_capi.so+0x33c0398)
    #11 mlir::StringAttr mlir::detail::AttributeUniquer::get<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:220:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #12 mlir::StringAttr mlir::detail::StorageUserBase<mlir::StringAttr, mlir::Attribute, mlir::detail::StringAttrStorage, mlir::detail::AttributeUniquer, mlir::TypedAttr::Trait>::get<llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:181:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #13 mlir::StringAttr::get(mlir::MLIRContext*, llvm::Twine const&) /proc/self/cwd/external/llvm-project/mlir/lib/IR/BuiltinAttributes.cpp:275:10 (libjaxlib_mlir_capi.so+0x33c0398)
    #14 mlirStringAttrGet /proc/self/cwd/external/llvm-project/mlir/lib/CAPI/IR/BuiltinAttributes.cpp:242:26 (libjaxlib_mlir_capi.so+0x25c39e5) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #15 (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext) const /proc/self/cwd/external/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:780:15 (_mlir.so+0x14813f) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #16 _object* nanobind::detail::func_create<false, true, (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), (anonymous namespace)::PyStringAttribute, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext, 0ul, 1ul, nanobind::scope, nanobind::name, nanobind::arg, nanobind::arg_v, char [32]>((anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)&&, (anonymous namespace)::PyStringAttribute (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), std::integer_sequence<unsigned long, 0ul, 1ul>, nanobind::scope const&, nanobind::name const&, nanobind::arg const&, nanobind::arg_v const&, char const (&) [32])::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:275:24 (_mlir.so+0x14813f)
    #17 _object* nanobind::detail::func_create<false, true, (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), (anonymous namespace)::PyStringAttribute, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext, 0ul, 1ul, nanobind::scope, nanobind::name, nanobind::arg, nanobind::arg_v, char [32]>((anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)&&, (anonymous namespace)::PyStringAttribute (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), std::integer_sequence<unsigned long, 0ul, 1ul>, nanobind::scope const&, nanobind::name const&, nanobind::arg const&, nanobind::arg_v const&, char const (&) [32])::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (_mlir.so+0x14813f)
    #18 nanobind::detail::nb_func_vectorcall_complex(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:823:26 (_mlir.so+0x16de42) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #19 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #20 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #21 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #22 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #23 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #24 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #25 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef6e0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #26 method_vectorcall /project/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef6e0)
    #27 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #28 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #29 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #30 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #31 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #32 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #33 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #34 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:135:15 (python3.13+0x1ea1dd) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #35 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #36 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #37 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #38 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #39 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #40 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #41 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #42 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #43 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #44 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #45 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #46 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #47 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #48 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #49 thread_run /project/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a12) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #50 pythread_wrapper /project/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd97) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

  Previous write of size 8 at 0x7290000c2168 by thread T54 (mutexes: read M0):
    #0 __tsan_memcpy <null> (python3.13+0xda10f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #1 llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>* llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::InsertIntoBucketWithLookup<(anonymous namespace)::ParametricStorageUniquer::LookupKey const>(llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>*, (anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, llvm::detail::DenseSetEmpty&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:555:27 (libjaxlib_mlir_capi.so+0x341a52f) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #2 std::pair<llvm::DenseMapIterator<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>, false>, bool> llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>(std::pair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty>&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:289:17 (libjaxlib_mlir_capi.so+0x341a52f)
    #3 std::pair<llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::Iterator, bool> llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseSet.h:232:19 (libjaxlib_mlir_capi.so+0x341a52f)
    #4 (anonymous namespace)::ParametricStorageUniquer::getOrCreateUnsafe((anonymous namespace)::ParametricStorageUniquer::Shard&, (anonymous namespace)::ParametricStorageUniquer::LookupKey&, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:95:37 (libjaxlib_mlir_capi.so+0x3417f91) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #5 (anonymous namespace)::ParametricStorageUniquer::getOrCreate(bool, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:140:14 (libjaxlib_mlir_capi.so+0x3417f91)
    #6 mlir::detail::StorageUniquerImpl::getOrCreate(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:270:27 (libjaxlib_mlir_capi.so+0x3417f91)
    #7 mlir::StorageUniquer::getParametricStorageTypeImpl(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:375:16 (libjaxlib_mlir_capi.so+0x3417b6f) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #8 mlir::detail::StringAttrStorage* mlir::StorageUniquer::get<mlir::detail::StringAttrStorage, llvm::StringRef, mlir::NoneType>(llvm::function_ref<void (mlir::detail::StringAttrStorage*)>, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/Support/StorageUniquer.h:218:9 (libjaxlib_mlir_capi.so+0x33c0398) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #9 std::enable_if<!std::is_same<mlir::StringAttr::ImplType, mlir::AttributeStorage>::value, mlir::StringAttr>::type mlir::detail::AttributeUniquer::getWithTypeID<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:239:39 (libjaxlib_mlir_capi.so+0x33c0398)
    #10 mlir::StringAttr mlir::detail::AttributeUniquer::get<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:220:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #11 mlir::StringAttr mlir::detail::StorageUserBase<mlir::StringAttr, mlir::Attribute, mlir::detail::StringAttrStorage, mlir::detail::AttributeUniquer, mlir::TypedAttr::Trait>::get<llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:181:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #12 mlir::StringAttr::get(mlir::MLIRContext*, llvm::Twine const&) /proc/self/cwd/external/llvm-project/mlir/lib/IR/BuiltinAttributes.cpp:275:10 (libjaxlib_mlir_capi.so+0x33c0398)
    #13 mlirStringAttrGet /proc/self/cwd/external/llvm-project/mlir/lib/CAPI/IR/BuiltinAttributes.cpp:242:26 (libjaxlib_mlir_capi.so+0x25c39e5) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #14 (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext) const /proc/self/cwd/external/llvm-project/mlir/lib/Bindings/Python/IRAttributes.cpp:780:15 (_mlir.so+0x14813f) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #15 _object* nanobind::detail::func_create<false, true, (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), (anonymous namespace)::PyStringAttribute, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext, 0ul, 1ul, nanobind::scope, nanobind::name, nanobind::arg, nanobind::arg_v, char [32]>((anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)&&, (anonymous namespace)::PyStringAttribute (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), std::integer_sequence<unsigned long, 0ul, 1ul>, nanobind::scope const&, nanobind::name const&, nanobind::arg const&, nanobind::arg_v const&, char const (&) [32])::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:275:24 (_mlir.so+0x14813f)
    #16 _object* nanobind::detail::func_create<false, true, (anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), (anonymous namespace)::PyStringAttribute, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext, 0ul, 1ul, nanobind::scope, nanobind::name, nanobind::arg, nanobind::arg_v, char [32]>((anonymous namespace)::PyStringAttribute::bindDerived(nanobind::class_<(anonymous namespace)::PyStringAttribute, mlir::python::PyAttribute>&)::'lambda'(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext)&&, (anonymous namespace)::PyStringAttribute (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, mlir::python::DefaultingPyMlirContext), std::integer_sequence<unsigned long, 0ul, 1ul>, nanobind::scope const&, nanobind::name const&, nanobind::arg const&, nanobind::arg_v const&, char const (&) [32])::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (_mlir.so+0x14813f)
    #17 nanobind::detail::nb_func_vectorcall_complex(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:823:26 (_mlir.so+0x16de42) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #18 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #19 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #20 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #21 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #22 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #23 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #24 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef6e0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #25 method_vectorcall /project/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef6e0)
    #26 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #27 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #28 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #29 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #30 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #31 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #32 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #33 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:135:15 (python3.13+0x1ea1dd) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #34 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #35 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #36 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #37 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #38 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #39 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #40 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #41 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #42 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #43 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #44 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #45 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #46 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #47 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #48 thread_run /project/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a12) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #49 pythread_wrapper /project/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd97) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

  Location is heap block of size 8192 at 0x7290000c2000 allocated by thread T54:
    #0 posix_memalign <null> (python3.13+0xddba9) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #1 operator new(unsigned long, std::align_val_t) <null> (libstdc++.so.6+0xbb9f5) (BuildId: ca77dae775ec87540acd7218fa990c40d1c94ab1)
    #2 llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::allocateBuckets(unsigned int) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:872:9 (libjaxlib_mlir_capi.so+0x341c142) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #3 llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::grow(unsigned int) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:817:5 (libjaxlib_mlir_capi.so+0x341c142)
    #4 llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::grow(unsigned int) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:536:64 (libjaxlib_mlir_capi.so+0x341c142)
    #5 llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>* llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::InsertIntoBucketImpl<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::LookupKey const&, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>*) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h (libjaxlib_mlir_capi.so+0x341a5ab) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #6 llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>* llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::InsertIntoBucketWithLookup<(anonymous namespace)::ParametricStorageUniquer::LookupKey const>(llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>*, (anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, llvm::detail::DenseSetEmpty&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:553:17 (libjaxlib_mlir_capi.so+0x341a5ab)
    #7 std::pair<llvm::DenseMapIterator<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>, false>, bool> llvm::DenseMapBase<llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>(std::pair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty>&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseMap.h:289:17 (libjaxlib_mlir_capi.so+0x341a5ab)
    #8 std::pair<llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::Iterator, bool> llvm::detail::DenseSetImpl<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::DenseMap<(anonymous namespace)::ParametricStorageUniquer::HashedStorage, llvm::detail::DenseSetEmpty, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo, llvm::detail::DenseSetPair<(anonymous namespace)::ParametricStorageUniquer::HashedStorage>>, (anonymous namespace)::ParametricStorageUniquer::StorageKeyInfo>::insert_as<(anonymous namespace)::ParametricStorageUniquer::LookupKey>((anonymous namespace)::ParametricStorageUniquer::HashedStorage&&, (anonymous namespace)::ParametricStorageUniquer::LookupKey const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/DenseSet.h:232:19 (libjaxlib_mlir_capi.so+0x341a5ab)
    #9 (anonymous namespace)::ParametricStorageUniquer::getOrCreateUnsafe((anonymous namespace)::ParametricStorageUniquer::Shard&, (anonymous namespace)::ParametricStorageUniquer::LookupKey&, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:95:37 (libjaxlib_mlir_capi.so+0x3418822) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #10 (anonymous namespace)::ParametricStorageUniquer::getOrCreate(bool, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* ()>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:159:24 (libjaxlib_mlir_capi.so+0x3418822)
    #11 mlir::detail::StorageUniquerImpl::getOrCreate(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:270:27 (libjaxlib_mlir_capi.so+0x3418822)
    #12 mlir::StorageUniquer::getParametricStorageTypeImpl(mlir::TypeID, unsigned int, llvm::function_ref<bool (mlir::StorageUniquer::BaseStorage const*)>, llvm::function_ref<mlir::StorageUniquer::BaseStorage* (mlir::StorageUniquer::StorageAllocator&)>) /proc/self/cwd/external/llvm-project/mlir/lib/Support/StorageUniquer.cpp:375:16 (libjaxlib_mlir_capi.so+0x3417b6f) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #13 mlir::detail::StringAttrStorage* mlir::StorageUniquer::get<mlir::detail::StringAttrStorage, llvm::StringRef, mlir::NoneType>(llvm::function_ref<void (mlir::detail::StringAttrStorage*)>, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/Support/StorageUniquer.h:218:9 (libjaxlib_mlir_capi.so+0x33c0398) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #14 std::enable_if<!std::is_same<mlir::StringAttr::ImplType, mlir::AttributeStorage>::value, mlir::StringAttr>::type mlir::detail::AttributeUniquer::getWithTypeID<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, mlir::TypeID, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:239:39 (libjaxlib_mlir_capi.so+0x33c0398)
    #15 mlir::StringAttr mlir::detail::AttributeUniquer::get<mlir::StringAttr, llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/AttributeSupport.h:220:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #16 mlir::StringAttr mlir::detail::StorageUserBase<mlir::StringAttr, mlir::Attribute, mlir::detail::StringAttrStorage, mlir::detail::AttributeUniquer, mlir::TypedAttr::Trait>::get<llvm::StringRef, mlir::NoneType>(mlir::MLIRContext*, llvm::StringRef&&, mlir::NoneType&&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:181:12 (libjaxlib_mlir_capi.so+0x33c0398)
    #17 mlir::StringAttr::get(mlir::MLIRContext*, llvm::Twine const&) /proc/self/cwd/external/llvm-project/mlir/lib/IR/BuiltinAttributes.cpp:275:10 (libjaxlib_mlir_capi.so+0x33c0398)
    #18 mlir::OperationName::Impl::Impl(llvm::StringRef, mlir::Dialect*, mlir::TypeID, mlir::detail::InterfaceMap) /proc/self/cwd/external/llvm-project/mlir/lib/IR/MLIRContext.cpp:786:12 (libjaxlib_mlir_capi.so+0x33340b9) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #19 mlir::RegisteredOperationName::Model<mlir::NVVM::CpAsyncBulkTensorGlobalToSharedClusterOp>::Model(mlir::Dialect*) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/OperationSupport.h:532:11 (libjaxlib_mlir_capi.so+0x1b0c16f) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #20 std::__detail::_MakeUniq<mlir::RegisteredOperationName::Model<mlir::NVVM::CpAsyncBulkTensorGlobalToSharedClusterOp>>::__single_object std::make_unique<mlir::RegisteredOperationName::Model<mlir::NVVM::CpAsyncBulkTensorGlobalToSharedClusterOp>, mlir::Dialect*>(mlir::Dialect*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/unique_ptr.h:1076:34 (libjaxlib_mlir_capi.so+0x1a4ee25) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #21 void mlir::RegisteredOperationName::insert<mlir::NVVM::CpAsyncBulkTensorGlobalToSharedClusterOp>(mlir::Dialect&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/OperationSupport.h:686:12 (libjaxlib_mlir_capi.so+0x1a4ee25)
    #22 void mlir::Dialect::addOperations<mlir::NVVM::Barrier0Op, mlir::NVVM::BarrierArriveOp, mlir::NVVM::BarrierOp, mlir::NVVM::BlockDimXOp, mlir::NVVM::BlockDimYOp, mlir::NVVM::BlockDimZOp, mlir::NVVM::BlockIdXOp, mlir::NVVM::BlockIdYOp, mlir::NVVM::BlockIdZOp, mlir::NVVM::BlockInClusterIdXOp, mlir::NVVM::BlockInClusterIdYOp, mlir::NVVM::BlockInClusterIdZOp, mlir::NVVM::Breakpoint, mlir::NVVM::Clock64Op, mlir::NVVM::ClockOp, mlir::NVVM::ClusterArriveOp, mlir::NVVM::ClusterArriveRelaxedOp, mlir::NVVM::ClusterDim, mlir::NVVM::ClusterDimBlocksXOp, mlir::NVVM::ClusterDimBlocksYOp, mlir::NVVM::ClusterDimBlocksZOp, mlir::NVVM::ClusterDimXOp, mlir::NVVM::ClusterDimYOp, mlir::NVVM::ClusterDimZOp, mlir::NVVM::ClusterId, mlir::NVVM::ClusterIdXOp, mlir::NVVM::ClusterIdYOp, mlir::NVVM::ClusterIdZOp, mlir::NVVM::ClusterWaitOp, mlir::NVVM::CpAsyncBulkCommitGroupOp, mlir::NVVM::CpAsyncBulkGlobalToSharedClusterOp, mlir::NVVM::CpAsyncBulkSharedCTAToGlobalOp, mlir::NVVM::CpAsyncBulkSharedCTAToSharedClusterOp, mlir::NVVM::CpAsyncBulkTensorGlobalToSharedClusterOp, mlir::NVVM::CpAsyncBulkTensorPrefetchOp, mlir::NVVM::CpAsyncBulkTensorReduceOp, mlir::NVVM::CpAsyncBulkTensorSharedCTAToGlobalOp, mlir::NVVM::CpAsyncBulkWaitGroupOp, mlir::NVVM::CpAsyncCommitGroupOp, mlir::NVVM::CpAsyncMBarrierArriveOp, mlir::NVVM::CpAsyncMBarrierArriveSharedOp, mlir::NVVM::CpAsyncOp, mlir::NVVM::CpAsyncWaitGroupOp, mlir::NVVM::CvtFloatToTF32Op, mlir::NVVM::ElectSyncOp, mlir::NVVM::EnvReg0Op, mlir::NVVM::EnvReg10Op, mlir::NVVM::EnvReg11Op, mlir::NVVM::EnvReg12Op, mlir::NVVM::EnvReg13Op, mlir::NVVM::EnvReg14Op, mlir::NVVM::EnvReg15Op, mlir::NVVM::EnvReg16Op, mlir::NVVM::EnvReg17Op, mlir::NVVM::EnvReg18Op, mlir::NVVM::EnvReg19Op, mlir::NVVM::EnvReg1Op, mlir::NVVM::EnvReg20Op, mlir::NVVM::EnvReg21Op, mlir::NVVM::EnvReg22Op, mlir::NVVM::EnvReg23Op, mlir::NVVM::EnvReg24Op, mlir::NVVM::EnvReg25Op, mlir::NVVM::EnvReg26Op, mlir::NVVM::EnvReg27Op, mlir::NVVM::EnvReg28Op, mlir::NVVM::EnvReg29Op, mlir::NVVM::EnvReg2Op, mlir::NVVM::EnvReg30Op, mlir::NVVM::EnvReg31Op, mlir::NVVM::EnvReg3Op, mlir::NVVM::EnvReg4Op, mlir::NVVM::EnvReg5Op, mlir::NVVM::EnvReg6Op, mlir::NVVM::EnvReg7Op, mlir::NVVM::EnvReg8Op, mlir::NVVM::EnvReg9Op, mlir::NVVM::Exit, mlir::NVVM::FenceMbarrierInitOp, mlir::NVVM::FenceProxyAcquireOp, mlir::NVVM::FenceProxyOp, mlir::NVVM::FenceProxyReleaseOp, mlir::NVVM::FenceScClusterOp, mlir::NVVM::GlobalTimerOp, mlir::NVVM::GridDimXOp, mlir::NVVM::GridDimYOp, mlir::NVVM::GridDimZOp, mlir::NVVM::GridIdOp, mlir::NVVM::GriddepcontrolLaunchDependentsOp, mlir::NVVM::GriddepcontrolWaitOp, mlir::NVVM::LaneIdOp, mlir::NVVM::LaneMaskEqOp, mlir::NVVM::LaneMaskGeOp, mlir::NVVM::LaneMaskGtOp, mlir::NVVM::LaneMaskLeOp, mlir::NVVM::LaneMaskLtOp, mlir::NVVM::LdMatrixOp, mlir::NVVM::MBarrierArriveExpectTxOp, mlir::NVVM::MBarrierArriveExpectTxSharedOp, mlir::NVVM::MBarrierArriveNocompleteOp, mlir::NVVM::MBarrierArriveNocompleteSharedOp, mlir::NVVM::MBarrierArriveOp, mlir::NVVM::MBarrierArriveSharedOp, mlir::NVVM::MBarrierInitOp, mlir::NVVM::MBarrierInitSharedOp, mlir::NVVM::MBarrierInvalOp, mlir::NVVM::MBarrierInvalSharedOp, mlir::NVVM::MBarrierTestWaitOp, mlir::NVVM::MBarrierTestWaitSharedOp, mlir::NVVM::MBarrierTryWaitParityOp, mlir::NVVM::MBarrierTryWaitParitySharedOp, mlir::NVVM::MapaOp, mlir::NVVM::MmaOp, mlir::NVVM::PrefetchTensorMapOp, mlir::NVVM::RcpApproxFtzF32Op, mlir::NVVM::ReduxOp, mlir::NVVM::SetMaxRegisterOp, mlir::NVVM::ShflOp, mlir::NVVM::SmDimOp, mlir::NVVM::SmIdOp, mlir::NVVM::StMatrixOp, mlir::NVVM::SyncWarpOp, mlir::NVVM::ThreadIdXOp, mlir::NVVM::ThreadIdYOp, mlir::NVVM::ThreadIdZOp, mlir::NVVM::VoteBallotOp, mlir::NVVM::WMMALoadOp, mlir::NVVM::WMMAMmaOp, mlir::NVVM::WMMAStoreOp, mlir::NVVM::WarpDimOp, mlir::NVVM::WarpIdOp, mlir::NVVM::WarpSizeOp, mlir::NVVM::WgmmaFenceAlignedOp, mlir::NVVM::WgmmaGroupSyncAlignedOp, mlir::NVVM::WgmmaMmaAsyncOp, mlir::NVVM::WgmmaWaitGroupSyncOp>() /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/Dialect.h:283:13 (libjaxlib_mlir_capi.so+0x1a4ee25)
    #23 mlir::NVVM::NVVMDialect::initialize() /proc/self/cwd/external/llvm-project/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp:1261:3 (libjaxlib_mlir_capi.so+0x1a3801d) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #24 mlir::NVVM::NVVMDialect::NVVMDialect(mlir::MLIRContext*) /proc/self/cwd/bazel-out/k8-opt/bin/external/llvm-project/mlir/include/mlir/Dialect/LLVMIR/NVVMOpsDialect.cpp.inc:19:3 (libjaxlib_mlir_capi.so+0x1a3801d)
    #25 mlir::NVVM::NVVMDialect* mlir::MLIRContext::getOrLoadDialect<mlir::NVVM::NVVMDialect>()::'lambda'()::operator()() const /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/MLIRContext.h:100:42 (libjaxlib_mlir_capi.so+0x6ee22b) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #26 std::unique_ptr<mlir::Dialect, std::default_delete<mlir::Dialect>> llvm::function_ref<std::unique_ptr<mlir::Dialect, std::default_delete<mlir::Dialect>> ()>::callback_fn<mlir::NVVM::NVVMDialect* mlir::MLIRContext::getOrLoadDialect<mlir::NVVM::NVVMDialect>()::'lambda'()>(long) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12 (libjaxlib_mlir_capi.so+0x6ee22b)
    #27 llvm::function_ref<std::unique_ptr<mlir::Dialect, std::default_delete<mlir::Dialect>> ()>::operator()() const /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12 (libjaxlib_mlir_capi.so+0x33322b3) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #28 mlir::MLIRContext::getOrLoadDialect(llvm::StringRef, mlir::TypeID, llvm::function_ref<std::unique_ptr<mlir::Dialect, std::default_delete<mlir::Dialect>> ()>) /proc/self/cwd/external/llvm-project/mlir/lib/IR/MLIRContext.cpp:476:49 (libjaxlib_mlir_capi.so+0x33322b3)
    #29 mlir::NVVM::NVVMDialect* mlir::MLIRContext::getOrLoadDialect<mlir::NVVM::NVVMDialect>() /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/MLIRContext.h:99:9 (libjaxlib_mlir_capi.so+0x6ee176) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #30 void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)::operator()(mlir::MLIRContext*) const /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/DialectRegistry.h:154:26 (libjaxlib_mlir_capi.so+0x6ee176)
    #31 mlir::NVVM::NVVMDialect* std::__invoke_impl<mlir::NVVM::NVVMDialect*, void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)&, mlir::MLIRContext*>(std::__invoke_other, void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)&, mlir::MLIRContext*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (libjaxlib_mlir_capi.so+0x6ee176)
    #32 std::enable_if<is_invocable_r_v<mlir::Dialect*, void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)&, mlir::MLIRContext*>, mlir::Dialect*>::type std::__invoke_r<mlir::Dialect*, void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)&, mlir::MLIRContext*>(void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)&, mlir::MLIRContext*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:114:9 (libjaxlib_mlir_capi.so+0x6ee176)
    #33 std::_Function_handler<mlir::Dialect* (mlir::MLIRContext*), void mlir::DialectRegistry::insert<mlir::NVVM::NVVMDialect>()::'lambda'(mlir::MLIRContext*)>::_M_invoke(std::_Any_data const&, mlir::MLIRContext*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (libjaxlib_mlir_capi.so+0x6ee176)
    #34 std::function<mlir::Dialect* (mlir::MLIRContext*)>::operator()(mlir::MLIRContext*) const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (libjaxlib_mlir_capi.so+0x338bc19) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #35 mlir::Dialect* llvm::function_ref<mlir::Dialect* (mlir::MLIRContext*)>::callback_fn<std::function<mlir::Dialect* (mlir::MLIRContext*)> const>(long, mlir::MLIRContext*) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:46:12 (libjaxlib_mlir_capi.so+0x338bc19)
    #36 llvm::function_ref<mlir::Dialect* (mlir::MLIRContext*)>::operator()(mlir::MLIRContext*) const /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:69:12 (libjaxlib_mlir_capi.so+0x3332d3b) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #37 mlir::MLIRContext::getOrLoadDialect(llvm::StringRef) /proc/self/cwd/external/llvm-project/mlir/lib/IR/MLIRContext.cpp:447:22 (libjaxlib_mlir_capi.so+0x3332d3b)
    #38 mlir::MLIRContext::loadAllAvailableDialects() /proc/self/cwd/external/llvm-project/mlir/lib/IR/MLIRContext.cpp:555:5 (libjaxlib_mlir_capi.so+0x3332d3b)
    #39 mlirContextLoadAllAvailableDialects /proc/self/cwd/external/llvm-project/mlir/lib/CAPI/IR/IR.cpp:108:20 (libjaxlib_mlir_capi.so+0x25cf2b9) (BuildId: c1220b9d9c837a0ef2acf3164a01c78d11fc72c5)
    #40 mlir::python::populateIRCore(nanobind::module_&)::$_14::operator()(mlir::python::PyMlirContext&) const /proc/self/cwd/external/llvm-project/mlir/lib/Bindings/Python/IRCore.cpp:2826:9 (_mlir.so+0xd1866) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #41 _object* nanobind::detail::func_create<false, true, mlir::python::populateIRCore(nanobind::module_&)::$_14, void, mlir::python::PyMlirContext&, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(mlir::python::populateIRCore(nanobind::module_&)::$_14&&, void (*)(mlir::python::PyMlirContext&), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:264:13 (_mlir.so+0xd1866)
    #42 _object* nanobind::detail::func_create<false, true, mlir::python::populateIRCore(nanobind::module_&)::$_14, void, mlir::python::PyMlirContext&, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(mlir::python::populateIRCore(nanobind::module_&)::$_14&&, void (*)(mlir::python::PyMlirContext&), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (_mlir.so+0xd1866)
    #43 nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:915:26 (_mlir.so+0x16e312) (BuildId: 294ceac9eacc32a1492f5bd0c3df61de64f742a3)
    #44 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #45 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #46 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #47 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #48 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #49 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #50 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:146:15 (python3.13+0x1ea1a0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #51 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #52 slot_tp_init /project/cpython/Objects/typeobject.c:9779:15 (python3.13+0x2fa37c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #53 type_call /project/cpython/Objects/typeobject.c:1993:19 (python3.13+0x2eb21c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #54 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #55 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #56 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #57 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e556d) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #58 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #59 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #60 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #61 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #62 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #63 nanobind::detail::obj_vectorcall(_object*, _object* const*, unsigned long, _object*, bool) /proc/self/cwd/external/nanobind/src/common.cpp:319:11 (xla_extension.so+0xd4dcc8d) (BuildId: ad17ba2d590fe83ae27f23ddc7e83e4cc2a0da71)
    #64 nanobind::object nanobind::detail::api<nanobind::handle>::operator()<(nanobind::rv_policy)1, nanobind::object&, nanobind::detail::args_proxy, nanobind::detail::kwargs_proxy>(nanobind::object&, nanobind::detail::args_proxy&&, nanobind::detail::kwargs_proxy&&) const /proc/self/cwd/external/nanobind/include/nanobind/nb_call.h:129:9 (xla_extension.so+0x937b52) (BuildId: ad17ba2d590fe83ae27f23ddc7e83e4cc2a0da71)
    #65 jax::WeakrefLRUCache::Call(nanobind::object, nanobind::args, nanobind::kwargs) /proc/self/cwd/external/xla/xla/python/weakref_lru_cache.cc:269:25 (xla_extension.so+0x934c22) (BuildId: ad17ba2d590fe83ae27f23ddc7e83e4cc2a0da71)
    #66 void nanobind::cpp_function_def<jax::WeakrefLRUCache, nanobind::object, jax::WeakrefLRUCache, nanobind::object, nanobind::args, nanobind::kwargs, nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self>(nanobind::object (jax::WeakrefLRUCache::*)(nanobind::object, nanobind::args, nanobind::kwargs), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&, nanobind::lock_self const&)::'lambda'(jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs)::operator()(jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:376:20 (xla_extension.so+0x93c10d) (BuildId: ad17ba2d590fe83ae27f23ddc7e83e4cc2a0da71)
    #67 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<jax::WeakrefLRUCache, nanobind::object, jax::WeakrefLRUCache, nanobind::object, nanobind::args, nanobind::kwargs, nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self>(nanobind::object (jax::WeakrefLRUCache::*)(nanobind::object, nanobind::args, nanobind::kwargs), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&, nanobind::lock_self const&)::'lambda'(jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs), nanobind::object, jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs, 0ul, 1ul, 2ul, 3ul, nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self>(jax::WeakrefLRUCache&&, nanobind::object (*)(nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&, nanobind::lock_self const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:275:24 (xla_extension.so+0x93c10d)
    #68 _object* nanobind::detail::func_create<false, true, void nanobind::cpp_function_def<jax::WeakrefLRUCache, nanobind::object, jax::WeakrefLRUCache, nanobind::object, nanobind::args, nanobind::kwargs, nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self>(nanobind::object (jax::WeakrefLRUCache::*)(nanobind::object, nanobind::args, nanobind::kwargs), nanobind::scope const&, nanobind::name const&, nanobind::is_method const&, nanobind::lock_self const&)::'lambda'(jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs), nanobind::object, jax::WeakrefLRUCache*, nanobind::object, nanobind::args, nanobind::kwargs, 0ul, 1ul, 2ul, 3ul, nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self>(jax::WeakrefLRUCache&&, nanobind::object (*)(nanobind::scope, nanobind::name, nanobind::is_method, nanobind::lock_self), std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&, nanobind::lock_self const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (xla_extension.so+0x93c10d)
    #69 nanobind::detail::nb_func_vectorcall_complex(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:823:26 (xla_extension.so+0xd4f6c32) (BuildId: ad17ba2d590fe83ae27f23ddc7e83e4cc2a0da71)
    #70 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:146:15 (python3.13+0x1ea1a0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #71 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #72 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #73 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #74 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #75 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #76 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e556d) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #77 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #78 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #79 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #80 _PyVectorcall_Call /project/cpython/Objects/call.c:285:24 (python3.13+0x1eb226) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #81 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb226)
    #82 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #83 partial_call /project/cpython/./Modules/_functoolsmodule.c:353:21 (python3.13+0x570f4b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #84 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #85 partial_vectorcall_fallback /project/cpython/./Modules/_functoolsmodule.c:226:12 (python3.13+0x572535) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #86 partial_vectorcall /project/cpython/./Modules/_functoolsmodule.c:238:16 (python3.13+0x572186) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #87 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #88 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #89 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #90 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #91 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #92 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #93 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef6e0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #94 method_vectorcall /project/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef6e0)
    #95 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #96 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #97 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #98 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #99 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #100 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #101 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #102 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:135:15 (python3.13+0x1ea1dd) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #103 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #104 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #105 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #106 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #107 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #108 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #109 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #110 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #111 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #112 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #113 method_vectorcall /project/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #114 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #115 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #116 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #117 thread_run /project/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a12) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #118 pythread_wrapper /project/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdd97) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

  Mutex M0 (0x7fffb7ba3778) created at:
    #0 __tsan_mutex_create <null> (python3.13+0x130623) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #1 absl::lts_20230802::Mutex::Mutex() /proc/self/cwd/external/com_google_absl/absl/synchronization/mutex.h:1022:3 (utils.so+0xac65) (BuildId: 8a7f0bc5213f57da9cd44587ba2607bf66b72f71)
    #2 void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)::operator()(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>) const /proc/self/cwd/external/nanobind/include/nanobind/nb_class.h:362:36 (utils.so+0xac65)
    #3 _object* nanobind::detail::func_create<false, true, void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), void, nanobind::pointer_and_handle<absl::lts_20230802::Mutex>, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)&&, void (*)(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::operator()(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) const /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:264:13 (utils.so+0xac65)
    #4 _object* nanobind::detail::func_create<false, true, void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), void, nanobind::pointer_and_handle<absl::lts_20230802::Mutex>, 0ul, nanobind::scope, nanobind::name, nanobind::is_method>(void nanobind::init<>::execute<nanobind::class_<absl::lts_20230802::Mutex>>(nanobind::class_<absl::lts_20230802::Mutex>&)::'lambda'(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>)&&, void (*)(nanobind::pointer_and_handle<absl::lts_20230802::Mutex>), std::integer_sequence<unsigned long, 0ul>, nanobind::scope const&, nanobind::name const&, nanobind::is_method const&)::'lambda'(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*)::__invoke(void*, _object**, unsigned char*, nanobind::rv_policy, nanobind::detail::cleanup_list*) /proc/self/cwd/external/nanobind/include/nanobind/nb_func.h:219:14 (utils.so+0xac65)
    #5 nanobind::detail::nb_func_vectorcall_simple(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_func.cpp:915:26 (utils.so+0x22242) (BuildId: 8a7f0bc5213f57da9cd44587ba2607bf66b72f71)
    #6 nanobind::detail::nb_type_vectorcall(_object*, _object* const*, unsigned long, _object*) /proc/self/cwd/external/nanobind/src/nb_type.cpp:1030:9 (utils.so+0x17f7e) (BuildId: 8a7f0bc5213f57da9cd44587ba2607bf66b72f71)
    #7 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafea) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #8 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eafea)
    #9 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #10 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de6f2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #11 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3de6f2)
    #12 PyEval_EvalCode /project/cpython/Python/ceval.c:597:21 (python3.13+0x3de6f2)
    #13 builtin_exec_impl /project/cpython/Python/bltinmodule.c:1145:17 (python3.13+0x3d8dd8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #14 builtin_exec /project/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d8dd8)
    #15 cfunction_vectorcall_FASTCALL_KEYWORDS /project/cpython/Objects/methodobject.c:441:24 (python3.13+0x289f20) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #16 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #17 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #18 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #19 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #20 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #21 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #22 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #23 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed690) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #24 object_vacall /project/cpython/Objects/call.c:819:14 (python3.13+0x1ed690)
    #25 PyObject_CallMethodObjArgs /project/cpython/Objects/call.c:880:24 (python3.13+0x1ed33b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #26 import_find_and_load /project/cpython/Python/import.c:3692:11 (python3.13+0x4607de) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #27 PyImport_ImportModuleLevelObject /project/cpython/Python/import.c:3774:15 (python3.13+0x4607de)
    #28 builtin___import___impl /project/cpython/Python/bltinmodule.c:277:12 (python3.13+0x3d70b4) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #29 builtin___import__ /project/cpython/Python/clinic/bltinmodule.c.h:107:20 (python3.13+0x3d70b4)
    #30 cfunction_vectorcall_FASTCALL_KEYWORDS /project/cpython/Objects/methodobject.c:441:24 (python3.13+0x289f20) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #31 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #32 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #33 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #34 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #35 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #36 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #37 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #38 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed690) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #39 object_vacall /project/cpython/Objects/call.c:819:14 (python3.13+0x1ed690)
    #40 PyObject_CallMethodObjArgs /project/cpython/Objects/call.c:880:24 (python3.13+0x1ed33b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #41 PyImport_ImportModuleLevelObject /project/cpython/Python/import.c:3842:25 (python3.13+0x46088b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #42 import_name /project/cpython/Python/ceval.c:2711:16 (python3.13+0x3ec777) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #43 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:3201:19 (python3.13+0x3ec777)
    #44 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de6f2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #45 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3de6f2)
    #46 PyEval_EvalCode /project/cpython/Python/ceval.c:597:21 (python3.13+0x3de6f2)
    #47 builtin_exec_impl /project/cpython/Python/bltinmodule.c:1145:17 (python3.13+0x3d8dd8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #48 builtin_exec /project/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d8dd8)
    #49 cfunction_vectorcall_FASTCALL_KEYWORDS /project/cpython/Objects/methodobject.c:441:24 (python3.13+0x289f20) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #50 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #51 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #52 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #53 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #54 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #55 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #56 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #57 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed690) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #58 object_vacall /project/cpython/Objects/call.c:819:14 (python3.13+0x1ed690)
    #59 PyObject_CallMethodObjArgs /project/cpython/Objects/call.c:880:24 (python3.13+0x1ed33b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #60 import_find_and_load /project/cpython/Python/import.c:3692:11 (python3.13+0x4607de) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #61 PyImport_ImportModuleLevelObject /project/cpython/Python/import.c:3774:15 (python3.13+0x4607de)
    #62 builtin___import___impl /project/cpython/Python/bltinmodule.c:277:12 (python3.13+0x3d70b4) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #63 builtin___import__ /project/cpython/Python/clinic/bltinmodule.c.h:107:20 (python3.13+0x3d70b4)
    #64 cfunction_vectorcall_FASTCALL_KEYWORDS /project/cpython/Objects/methodobject.c:441:24 (python3.13+0x289f20) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #65 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #66 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #67 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #68 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #69 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #70 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #71 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #72 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ed690) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #73 object_vacall /project/cpython/Objects/call.c:819:14 (python3.13+0x1ed690)
    #74 PyObject_CallMethodObjArgs /project/cpython/Objects/call.c:880:24 (python3.13+0x1ed33b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #75 PyImport_ImportModuleLevelObject /project/cpython/Python/import.c:3842:25 (python3.13+0x46088b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #76 import_name /project/cpython/Python/ceval.c:2711:16 (python3.13+0x3ec777) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #77 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:3201:19 (python3.13+0x3ec777)
    #78 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de6f2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #79 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3de6f2)
    #80 PyEval_EvalCode /project/cpython/Python/ceval.c:597:21 (python3.13+0x3de6f2)
    #81 run_eval_code_obj /project/cpython/Python/pythonrun.c:1337:9 (python3.13+0x4a0a5e) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #82 run_mod /project/cpython/Python/pythonrun.c:1422:19 (python3.13+0x4a0185) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #83 pyrun_file /project/cpython/Python/pythonrun.c:1255:15 (python3.13+0x49c280) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #84 _PyRun_SimpleFileObject /project/cpython/Python/pythonrun.c:490:13 (python3.13+0x49c280)
    #85 _PyRun_AnyFileObject /project/cpython/Python/pythonrun.c:77:15 (python3.13+0x49b948) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #86 pymain_run_file_obj /project/cpython/Modules/main.c:410:15 (python3.13+0x4d7e6f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #87 pymain_run_file /project/cpython/Modules/main.c:429:15 (python3.13+0x4d7e6f)
    #88 pymain_run_python /project/cpython/Modules/main.c:697:21 (python3.13+0x4d70bc) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #89 Py_RunMain /project/cpython/Modules/main.c:776:5 (python3.13+0x4d70bc)
    #90 pymain_main /project/cpython/Modules/main.c:806:12 (python3.13+0x4d74f8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #91 Py_BytesMain /project/cpython/Modules/main.c:830:12 (python3.13+0x4d757b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #92 main /project/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

  Thread T72 (tid=469314, running) created by main thread at:
    #0 pthread_create <null> (python3.13+0xde1df) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #1 do_start_joinable_thread /project/cpython/Python/thread_pthread.h:290:14 (python3.13+0x4bcc48) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #2 PyThread_start_joinable_thread /project/cpython/Python/thread_pthread.h:314:9 (python3.13+0x4bca6a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #3 ThreadHandle_start /project/cpython/./Modules/_threadmodule.c:422:9 (python3.13+0x5645a7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #4 do_start_new_thread /project/cpython/./Modules/_threadmodule.c:1849:9 (python3.13+0x5645a7)
    #5 thread_PyThread_start_joinable_thread /project/cpython/./Modules/_threadmodule.c:1972:14 (python3.13+0x563641) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #6 cfunction_call /project/cpython/Objects/methodobject.c:540:18 (python3.13+0x28abb7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #7 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #8 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #9 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #10 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e556d) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #11 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #12 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #13 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #14 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef6e0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #15 method_vectorcall /project/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef6e0)
    #16 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #17 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #18 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #19 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #20 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #21 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #22 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #23 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:135:15 (python3.13+0x1ea1dd) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #24 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #25 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #26 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #27 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #28 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #29 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #30 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #31 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #32 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #33 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:146:15 (python3.13+0x1ea1a0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #34 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #35 slot_tp_init /project/cpython/Objects/typeobject.c:9779:15 (python3.13+0x2fa37c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #36 type_call /project/cpython/Objects/typeobject.c:1993:19 (python3.13+0x2eb21c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #37 _PyObject_Call /project/cpython/Objects/call.c:361:18 (python3.13+0x1eb28b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #38 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #39 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #40 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de6f2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #41 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3de6f2)
    #42 PyEval_EvalCode /project/cpython/Python/ceval.c:597:21 (python3.13+0x3de6f2)
    #43 run_eval_code_obj /project/cpython/Python/pythonrun.c:1337:9 (python3.13+0x4a0a5e) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #44 run_mod /project/cpython/Python/pythonrun.c:1422:19 (python3.13+0x4a0185) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #45 pyrun_file /project/cpython/Python/pythonrun.c:1255:15 (python3.13+0x49c280) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #46 _PyRun_SimpleFileObject /project/cpython/Python/pythonrun.c:490:13 (python3.13+0x49c280)
    #47 _PyRun_AnyFileObject /project/cpython/Python/pythonrun.c:77:15 (python3.13+0x49b948) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #48 pymain_run_file_obj /project/cpython/Modules/main.c:410:15 (python3.13+0x4d7e6f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #49 pymain_run_file /project/cpython/Modules/main.c:429:15 (python3.13+0x4d7e6f)
    #50 pymain_run_python /project/cpython/Modules/main.c:697:21 (python3.13+0x4d70bc) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #51 Py_RunMain /project/cpython/Modules/main.c:776:5 (python3.13+0x4d70bc)
    #52 pymain_main /project/cpython/Modules/main.c:806:12 (python3.13+0x4d74f8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #53 Py_BytesMain /project/cpython/Modules/main.c:830:12 (python3.13+0x4d757b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #54 main /project/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

  Thread T54 (tid=469282, running) created by main thread at:
    #0 pthread_create <null> (python3.13+0xde1df) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #1 do_start_joinable_thread /project/cpython/Python/thread_pthread.h:290:14 (python3.13+0x4bcc48) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #2 PyThread_start_joinable_thread /project/cpython/Python/thread_pthread.h:314:9 (python3.13+0x4bca6a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #3 ThreadHandle_start /project/cpython/./Modules/_threadmodule.c:422:9 (python3.13+0x5645a7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #4 do_start_new_thread /project/cpython/./Modules/_threadmodule.c:1849:9 (python3.13+0x5645a7)
    #5 thread_PyThread_start_joinable_thread /project/cpython/./Modules/_threadmodule.c:1972:14 (python3.13+0x563641) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #6 cfunction_call /project/cpython/Objects/methodobject.c:540:18 (python3.13+0x28abb7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #7 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #8 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #9 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #10 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1502:19 (python3.13+0x3e556d) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #11 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #12 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #13 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #14 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef6e0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #15 method_vectorcall /project/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef6e0)
    #16 _PyVectorcall_Call /project/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #17 _PyObject_Call /project/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #18 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #19 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #20 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #21 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #22 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #23 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:135:15 (python3.13+0x1ea1dd) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #24 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #25 slot_tp_call /project/cpython/Objects/typeobject.c:9533:15 (python3.13+0x2f8bb8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #26 _PyObject_MakeTpCall /project/cpython/Objects/call.c:242:18 (python3.13+0x1ea44c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #27 _PyObject_VectorcallTstate /project/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb0a8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #28 PyObject_Vectorcall /project/cpython/Objects/call.c:327:12 (python3.13+0x1eb0a8)
    #29 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e28eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #30 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea1a) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #31 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea1a)
    #32 _PyFunction_Vectorcall /project/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #33 _PyObject_VectorcallDictTstate /project/cpython/Objects/call.c:146:15 (python3.13+0x1ea1a0) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #34 _PyObject_Call_Prepend /project/cpython/Objects/call.c:504:24 (python3.13+0x1ebcd7) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #35 slot_tp_init /project/cpython/Objects/typeobject.c:9779:15 (python3.13+0x2fa37c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #36 type_call /project/cpython/Objects/typeobject.c:1993:19 (python3.13+0x2eb21c) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #37 _PyObject_Call /project/cpython/Objects/call.c:361:18 (python3.13+0x1eb28b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #38 PyObject_Call /project/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #39 _PyEval_EvalFrameDefault /project/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4ad2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #40 _PyEval_EvalFrame /project/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de6f2) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #41 _PyEval_Vector /project/cpython/Python/ceval.c:1807:12 (python3.13+0x3de6f2)
    #42 PyEval_EvalCode /project/cpython/Python/ceval.c:597:21 (python3.13+0x3de6f2)
    #43 run_eval_code_obj /project/cpython/Python/pythonrun.c:1337:9 (python3.13+0x4a0a5e) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #44 run_mod /project/cpython/Python/pythonrun.c:1422:19 (python3.13+0x4a0185) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #45 pyrun_file /project/cpython/Python/pythonrun.c:1255:15 (python3.13+0x49c280) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #46 _PyRun_SimpleFileObject /project/cpython/Python/pythonrun.c:490:13 (python3.13+0x49c280)
    #47 _PyRun_AnyFileObject /project/cpython/Python/pythonrun.c:77:15 (python3.13+0x49b948) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #48 pymain_run_file_obj /project/cpython/Modules/main.c:410:15 (python3.13+0x4d7e6f) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #49 pymain_run_file /project/cpython/Modules/main.c:429:15 (python3.13+0x4d7e6f)
    #50 pymain_run_python /project/cpython/Modules/main.c:697:21 (python3.13+0x4d70bc) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #51 Py_RunMain /project/cpython/Modules/main.c:776:5 (python3.13+0x4d70bc)
    #52 pymain_main /project/cpython/Modules/main.c:806:12 (python3.13+0x4d74f8) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #53 Py_BytesMain /project/cpython/Modules/main.c:830:12 (python3.13+0x4d757b) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)
    #54 main /project/cpython/./Programs/python.c:15:12 (python3.13+0x15c7eb) (BuildId: 5572a851d2acb326869407b965b3c37f6628c49d)

To reproduce:

bazel test     --test_env=JAX_NUM_GENERATED_CASES=$JAX_NUM_GENERATED_CASES     --test_env=JAX_ENABLE_X64=$JAX_ENABLE_X64     --test_env=JAX_SKIP_SLOW_TESTS=$JAX_SKIP_SLOW_TESTS     --test_env=PYTHON_GIL=0     --test_env=TSAN_OPTIONS=halt_on_error=1,suppressions=$PWD/.github/workflows/tsan-suppressions.txt     --test_env=JAX_TEST_NUM_THREADS=8     --test_output=all     --local_test_jobs=32     --test_timeout=600     --nocache_test_results --runs_per_test=100  --test_filter=CompatTest  //tests:export_back_compat_test_cpu

@hawkinsp
Copy link
Collaborator Author

hawkinsp commented Feb 4, 2025

A perhaps silly question: when we compile the MLIR C++ code, do we compile it with threading enabled? I seem to recall some locking there is only enabled if you build with threading enabled.

Perhaps also we need to enable threading on our MLIR contexts? IIRC right now we disable it because we don't want MLIR to create thread pools, but that's orthogonal to "is thread safe".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vfdev-5 vfdev-5

Labels
bug Something isn't working free threading Issues found in free threading builds
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hawkinsp @vfdev-5