summaryrefslogtreecommitdiff
path: root/sys/crypto
AgeCommit message (Collapse)Author
2022-11-05Make aes and chacha prints debug only.jmcneill
2022-09-01cprng_fast(9): Assert not in pserialize read section.riastradh
This may sleep to take the global entropy lock in case it needs to be reseeded. If that happens we can't be in a pserialize read section.
2022-06-26arm/aes_neon: Fix formatting of self-test failure message.riastradh
Discovered by code inspection. Remarkably, a combination of errors made this fail to be a stack buffer overrun. Verified by booting with ARMv8.0-AES disabled and with the self-test artificially made to fail.
2022-06-01cprng(9): cprng_fast is no longer used from interrupt context.riastradh
Rip out logic to defer reseeding to softint.
2022-04-16fix various typos in comments and log messages.andvar
2021-12-05s/folllowing/following/msaitoh
2021-10-17Upgrade self-test passed messages from verbose to debug.jmcneill
2021-10-15fix typos in comments.andvar
2021-09-04Fix typos in comments and add missing KERNEL_RCSIDgutteridge
2021-04-14use an enum instead of constant variables so that they work in CTASSERT.christos
2020-11-21Fix build with clang for earmv7hf; loadroundkey() is used only for __aarch64__.rin
2020-10-10Fix detection of NEON features. ID_AA64PFR0_EL1_ADV_SIMD_NONE means SIMDjmcneill
is not available, and any other value means it is.
2020-09-10aes neon: Gather mc_forward/backward so we can load 256 bits at once.riastradh
2020-09-10aes neon: Hoist dsbd/dsbe address calculation out of loop.riastradh
2020-09-10aes neon: Tweak register usage.riastradh
- Call r12 by its usual name, ip. - No need for r7 or r11=fp at the moment.
2020-09-10aes neon: Write vtbl with {qN} rather than {d(2N)-d(2N+1)}.riastradh
Cosmetic; no functional change.
2020-09-10aes neon: Issue 256-bit loads rather than pairs of 128-bit loads.riastradh
Not sure why I didn't realize you could do this before! Saves some temporary registers that can now be allocated to shave off a few cycles.
2020-09-08aesarmv8: Reallocate registers to shave off unnecessary MOV.riastradh
2020-09-08aesarmv8: Issue two 4-register ld/st, not four 2-register ld/st.riastradh
2020-09-08aesarmv8: Adapt aes_armv8_64.S to big-endian.riastradh
Patch mainly from (and tested by) jakllsch@ with minor tweaks by me.
2020-09-08aes(9): Fix edge case in bitsliced SSE2 AES-CBC decryption.riastradh
Make sure self-tests exercise this edge case. Discovered by confusion over code inspection of jak's adaptation of aes_armv8_64.S for big-endian.
2020-09-08Acknowledge clang warning for NEON cipher code on aarch64ebjakllsch
We've already made the nonportable vector initializations portable; the code works on aarch64eb.
2020-09-08use correct conditionjakllsch
2020-09-07Fix vgetq_lane_u32 for aarch64eb with GCCjakllsch
Fixes NEON AES on aarch64eb
2020-09-07Use a working macro to detect big endian aarch64.jakllsch
Fixes aarch64eb NEON ChaCha.
2020-09-05x86: fix several CPUID flagsmaxv
- Rename: CPUID_PN -> CPUID_PSN CPUID_CFLUSH -> CPUID_CLFSH CPUID_SBF -> CPUID_PBE CPUID_LZCNT -> CPUID_ABM CPUID_P1GB -> CPUID_PAGE1GB CPUID2_PCLMUL -> CPUID2_PCLMULQDQ CPUID2_CID -> CPUID2_CNXTID CPUID2_xTPR -> CPUID2_XTPR CPUID2_AES -> CPUID2_AESNI To match the x86 specification and the other OSes. - Remove: CPUID_B10, CPUID_B20, CPUID_IA64. They do not exist.
2020-08-26Instead of returning 0 when sysctl kern.expose_address=0, return a randomchristos
hashed value of the data. This allows sockstat to work without exposing kernel addresses or being setgid kmem.
2020-08-23Adjust sp, not fp, to allocate a 32-byte temporary.riastradh
Costs another couple MOV instructions, but we can't skimp on this -- there's no red zone below sp for interrupts on arm, so we can't touch anything there. So just use fp to save sp and then adjust sp itself, rather than using fp as a temporary register to point just below sp. Should fix PR port-arm/55598 -- previously the ChaCha self-test failed 33/10000 trials triggered by sysctl during running system; with the patch it has failed 0/10000 trials. (Presumably it happened more often at boot time, leading to 5/26 failures in the test bed, because we just enabled interrupts and some devices are starting to deliver interrupts.)
2020-08-20Import small BLAKE2s implementation.riastradh
2020-08-20[ozaki-r] libsodium glueriastradh
2020-08-16Fix AES NEON code for big-endian softfp ARM.riastradh
...which is how the kernel runs. Switch to using __SOFTFP__ for consistency with how it gets exposed to C, although I'm not sure how to get it defined automagically in the toolchain for .S files so that's set manually in files.aesneon for now.
2020-08-10Add hack to compile aes_ccm_tag() with -O0 for m68k for GCC8.rin
GCC 8 miscompiles aes_ccm_tag() for m68k with optimization level -O[12], which results in failure in aes_ccm_selftest(): | aes_ccm_selftest: tag 0: 8 bytes @ 0x4d3e38 | 03 80 5f 08 22 6f cb fe | .._."o.. | aes_ccm_selftest: verify 0 failed | ... | WARNING: module error: built-in module aes_ccm failed its MODULE_CMD_INIT, error 5 This is observed for amiga (A1200, 68060), mac68k (Quadra 840AV, 68040), and luna68k (nono, 68030 emulator). However, it is not for sun3 (TME, 68020 emulator) and sun2 (TME, 68010 emulator). At the moment, it is unclear whether this is due to differences b/w 68010-20 vs 68030-60, or something wrong with TME.
2020-08-09Fix some clang neon intrinsics.riastradh
Compile-tested only, with -Wno-nonportable-vector-initializers. Need to address -- and test -- this stuff properly but this is progress.
2020-08-09Use vshlq_n_s32 rather than vsliq_n_s32 with zero destination.riastradh
Not sure why I reached for vsliq_n_s32 at first -- probably so I wouldn't have to deal with a new intrinsic in arm_neon.h!
2020-08-09Nix outdated comment.riastradh
I implemented this parallelism a couple weeks ago.
2020-08-09Fix mistake in big-endian arm clang.riastradh
Swapped the two halves (only gcc does that, I think) and wrote j,i backwards, oops. (I don't have a big-endian arm clang build handy to test; hoping this works.)
2020-08-08Fix ARM NEON implementations of AES and ChaCha on big-endian ARM.riastradh
New macros such as VQ_N_U32(a,b,c,d) for NEON vector initializers. Needed because GCC and Clang disagree on the ordering of lanes, depending on whether it's 64-bit big-endian, 32-bit big-endian, or little-endian -- and, bizarrely, both of them disagree with the architectural numbering of lanes. Experimented with using static const uint8_t x8[16] = {...}; uint8x16_t x = vld1q_u8(x8); which doesn't require knowing anything about the ordering of lanes, but this generates considerably worse code and apparently confuses GCC into not recognizing the constant value of x8. Fix some clang mistakes while here too.
2020-07-29Issue three more swaps to save eight stores.riastradh
Reduces code size and yields a small (~2%) cgd throughput boost. Remove duplicate comment while here.
2020-07-28Rewrite cprng_fast in terms of new ChaCha API.riastradh
2020-07-28Draft 2x vectorized neon vpaes for aarch64.riastradh
Gives a modest speed boost on rk3399 (Cortex-A53/A72), around 20% in cgd tests, for parallelizable operations like CBC decryption; same improvement should probably carry over to rpi4 CPU which lacks ARMv8.0-AES.
2020-07-28Implement 4-way vectorization of ChaCha for armv7 NEON.riastradh
cgd performance is not as good as I was hoping (~4% improvement over chacha_ref.c) but it should improve substantially more if we let the cgd worker thread keep fpu state so we don't have to pay the cost of isb and zero-the-fpu on every 512-byte cgd block.
2020-07-28Fix big-endian build with appropriate casts around vrev32q_u8.riastradh
2020-07-28Fix typo in comment.riastradh
2020-07-28Initialize authctr in both branches.riastradh
I guess I didn't test the unaligned case, weird.
2020-07-27Note that VSRI seems to hurt here.riastradh
2020-07-27Take advantage of REV32 and TBL for 16-bit and 8-bit rotations.riastradh
However, disable use of (V)TBL on armv7/aarch32 for now, because for some reason GCC spills things to the stack despite having plenty of free registers, which hurts performance more than it helps at least on ARM Cortex-A8.
2020-07-27Add RCSIDs to the AES and ChaCha .S sources.riastradh
2020-07-27Issue aese/aesmc and aesd/aesimc in pairs.riastradh
Advised by the aarch64 optimization guide; increases cgd throughput by about 10%.
2020-07-27Align critical-path loops in AES and ChaCha.riastradh
2020-07-27PIC for aes_neon_32.S.riastradh
Without this, tests/sys/crypto/aes/t_aes fails to start on armv7 because of R_ARM_ABS32 relocations in a nonwritable text segment for a PIE -- which atf quietly ignores in the final report! Yikes.