Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package does not use hardware acceleration #90

Open
jakobnissen opened this issue Nov 3, 2023 · 2 comments
Open

Package does not use hardware acceleration #90

jakobnissen opened this issue Nov 3, 2023 · 2 comments

Comments

@jakobnissen
Copy link
Contributor

Python's hashlib.sha256(data).hexdigest() is 10x faster than SHA.jl's sha256(data) on my computer:

In [8]: %timeit hashlib.sha256(data).hexdigest()
53.8 µs ± 6.88 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
julia> @benchmark sha256(data)
BenchmarkTools.Trial: 8884 samples with 1 evaluation.
 Range (min  max):  514.644 μs  913.010 μs  ┊ GC (min  max): 0.00%  0.00%
 Time  (median):     518.341 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   561.483 μs ±  99.395 μs  ┊ GC (mean ± σ):  0.00% ± 0.00%

This is presumably (haven't checked though) because Python's underlying C library uses SHA-specific CPU instructions, where as this package does not.
This might not be possible to fix in current versions of Julia, but given that we see a 10x performance difference, this is something Julia needs to support eventually (and probably will). When that time comes, this package should be updated.

This issue was discovered by @jonalm

@inkydragon
Copy link
Collaborator

Python is currently migrating some of the hash algorithms to use the HACL* project, and the rest of the algorithms use the OpensSSL implementation.

python/cpython#99108

Changed in version 3.9: Hashlib now uses SHA3 and SHAKE from OpenSSL if it provides it.

Changed in version 3.12: For any of the MD5, SHA1, SHA2, or SHA3 algorithms that the linked OpenSSL does not provide we fall back to a verified implementation from the HACL* project.
https://docs.python.org/3/library/hashlib.html#hash-algorithms

HACL* is a formally verified library of modern cryptographic algorithms written in a subset of F* called Low* and compiled to C using a compiler called KaRaMeL.

HACL* is released in pure C, so we can wrap it and test the performance.


I'd prefer to keep the pure julia implementation and then have the fast impl. as a separate package, maybe called HashlibAcc.jl. The default is to use the pure julia impl., and if users need a faster impl., they can import the accelerated package.

The main concern is that introducing a third-party dependency may increase the maintenance burden.

@jakobnissen
Copy link
Contributor Author

Would it not be best of both worlds to use hardware acceleration in pure Julia?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants