/sys/modules/opencrypto/

t logo'/> index : netbsd
NetBSD fork for lockdoc analysismerlin@scholz.ruhr
summaryrefslogtreecommitdiff
path: root/sys/crypto/chacha
AgeCommit message (Collapse)Author
2022-11-05Make aes and chacha prints debug only.jmcneill
2020-10-10Fix detection of NEON features. ID_AA64PFR0_EL1_ADV_SIMD_NONE means SIMDjmcneill
is not available, and any other value means it is.
2020-09-08Acknowledge clang warning for NEON cipher code on aarch64ebjakllsch
We've already made the nonportable vector initializations portable; the code works on aarch64eb.
2020-09-08use correct conditionjakllsch
2020-09-07Fix vgetq_lane_u32 for aarch64eb with GCCjakllsch
Fixes NEON AES on aarch64eb
2020-09-07Use a working macro to detect big endian aarch64.jakllsch
Fixes aarch64eb NEON ChaCha.
2020-08-23Adjust sp, not fp, to allocate a 32-byte temporary.riastradh
Costs another couple MOV instructions, but we can't skimp on this -- there's no red zone below sp for interrupts on arm, so we can't touch anything there. So just use fp to save sp and then adjust sp itself, rather than using fp as a temporary register to point just below sp. Should fix PR port-arm/55598 -- previously the ChaCha self-test failed 33/10000 trials triggered by sysctl during running system; with the patch it has failed 0/10000 trials. (Presumably it happened more often at boot time, leading to 5/26 failures in the test bed, because we just enabled interrupts and some devices are starting to deliver interrupts.)
2020-08-09Fix some clang neon intrinsics.riastradh
Compile-tested only, with -Wno-nonportable-vector-initializers. Need to address -- and test -- this stuff properly but this is progress.
2020-08-09Use vshlq_n_s32 rather than vsliq_n_s32 with zero destination.riastradh
Not sure why I reached for vsliq_n_s32 at first -- probably so I wouldn't have to deal with a new intrinsic in arm_neon.h!
2020-08-09Fix mistake in big-endian arm clang.riastradh
Swapped the two halves (only gcc does that, I think) and wrote j,i backwards, oops. (I don't have a big-endian arm clang build handy to test; hoping this works.)
2020-08-08Fix ARM NEON implementations of AES and ChaCha on big-endian ARM.riastradh
New macros such as VQ_N_U32(a,b,c,d) for NEON vector initializers. Needed because GCC and Clang disagree on the ordering of lanes, depending on whether it's 64-bit big-endian, 32-bit big-endian, or little-endian -- and, bizarrely, both of them disagree with the architectural numbering of lanes. Experimented with using static const uint8_t x8[16] = {...}; uint8x16_t x = vld1q_u8(x8); which doesn't require knowing anything about the ordering of lanes, but this generates considerably worse code and apparently confuses GCC into not recognizing the constant value of x8. Fix some clang mistakes while here too.
2020-07-29Issue three more swaps to save eight stores.riastradh
Reduces code size and yields a small (~2%) cgd throughput boost. Remove duplicate comment while here.
2020-07-28Implement 4-way vectorization of ChaCha for armv7 NEON.riastradh
cgd performance is not as good as I was hoping (~4% improvement over chacha_ref.c) but it should improve substantially more if we let the cgd worker thread keep fpu state so we don't have to pay the cost of isb and zero-the-fpu on every 512-byte cgd block.
2020-07-28Fix big-endian build with appropriate casts around vrev32q_u8.riastradh
2020-07-28Fix typo in comment.riastradh
2020-07-27Note that VSRI seems to hurt here.riastradh
2020-07-27Take advantage of REV32 and TBL for 16-bit and 8-bit rotations.riastradh
However, disable use of (V)TBL on armv7/aarch32 for now, because for some reason GCC spills things to the stack despite having plenty of free registers, which hurts performance more than it helps at least on ARM Cortex-A8.
2020-07-27Add RCSIDs to the AES and ChaCha .S sources.riastradh
2020-07-27Align critical-path loops in AES and ChaCha.riastradh
2020-07-27Enable ChaCha NEON code on armv7 too.riastradh
The 4-blocks-at-a-time assembly helper is disabled for now; adapting it to armv7 is going to be a little annoying with only 16 128-bit vector registers. (Should also do a fifth block in the integer registers for 320 bytes at a time.)
2020-07-27Use <aarch64/asm.h> rather than copying things from it here.riastradh
Vestige from userland build on netbsd-9 during development.
2020-07-27Simplify ChaCha selection and allow it to be used much earlier.riastradh
This way we can use it for cprng_fast early on. ChaCha is easy because there's no data formats that must be preserved from call to call but vary from implementation to implementation -- we could even make it a sysctl knob to dynamically select it with negligible cost. (In contrast, different AES implementations use different expanded key formats which must be preserved from aes_setenckey to aes_enc, for example, which means a considerably greater burden on dynamic selection that's not really worth it.)
2020-07-27Reduce some duplication.riastradh
Shouldn't substantively hurt performance -- the comparison that has been moved into the loop was essentially the former loop condition -- and may improve performance by reducing code size since there's only one inline call to chacha_permute instead of two.
2020-07-27New sysctl subtree kern.crypto.riastradh
kern.crypto.aes.selected (formerly hw.aes_impl) kern.crypto.chacha.selected (formerly hw.chacha_impl) XXX Should maybe deduplicate creation of kern.crypto.
2020-07-25Implement ChaCha with NEON on ARM.riastradh
XXX Needs performance measurement. XXX Needs adaptation to arm32 neon which has half the registers.
2020-07-25Implement ChaCha with SSE2 on x86 machines.riastradh
Slightly disappointed that it only doubles, rather than quadruples, throughput on my Ivy Bridge laptop. Worth investigating.
2020-07-25New ChaCha API in kernel.riastradh
This will enable us to adopt MD vectorized implementations of ChaCha.