summaryrefslogtreecommitdiff
path: root/sys/crypto/aes
AgeCommit message (Collapse)Author
2022-11-05Make aes and chacha prints debug only.jmcneill
2022-06-26arm/aes_neon: Fix formatting of self-test failure message.riastradh
Discovered by code inspection. Remarkably, a combination of errors made this fail to be a stack buffer overrun. Verified by booting with ARMv8.0-AES disabled and with the self-test artificially made to fail.
2021-12-05s/folllowing/following/msaitoh
2021-10-17Upgrade self-test passed messages from verbose to debug.jmcneill
2020-11-21Fix build with clang for earmv7hf; loadroundkey() is used only for __aarch64__.rin
2020-10-10Fix detection of NEON features. ID_AA64PFR0_EL1_ADV_SIMD_NONE means SIMDjmcneill
is not available, and any other value means it is.
2020-09-10aes neon: Gather mc_forward/backward so we can load 256 bits at once.riastradh
2020-09-10aes neon: Hoist dsbd/dsbe address calculation out of loop.riastradh
2020-09-10aes neon: Tweak register usage.riastradh
- Call r12 by its usual name, ip. - No need for r7 or r11=fp at the moment.
2020-09-10aes neon: Write vtbl with {qN} rather than {d(2N)-d(2N+1)}.riastradh
Cosmetic; no functional change.
2020-09-10aes neon: Issue 256-bit loads rather than pairs of 128-bit loads.riastradh
Not sure why I didn't realize you could do this before! Saves some temporary registers that can now be allocated to shave off a few cycles.
2020-09-08aesarmv8: Reallocate registers to shave off unnecessary MOV.riastradh
2020-09-08aesarmv8: Issue two 4-register ld/st, not four 2-register ld/st.riastradh
2020-09-08aesarmv8: Adapt aes_armv8_64.S to big-endian.riastradh
Patch mainly from (and tested by) jakllsch@ with minor tweaks by me.
2020-09-08aes(9): Fix edge case in bitsliced SSE2 AES-CBC decryption.riastradh
Make sure self-tests exercise this edge case. Discovered by confusion over code inspection of jak's adaptation of aes_armv8_64.S for big-endian.
2020-09-08Acknowledge clang warning for NEON cipher code on aarch64ebjakllsch
We've already made the nonportable vector initializations portable; the code works on aarch64eb.
2020-09-07Fix vgetq_lane_u32 for aarch64eb with GCCjakllsch
Fixes NEON AES on aarch64eb
2020-09-05x86: fix several CPUID flagsmaxv
- Rename: CPUID_PN -> CPUID_PSN CPUID_CFLUSH -> CPUID_CLFSH CPUID_SBF -> CPUID_PBE CPUID_LZCNT -> CPUID_ABM CPUID_P1GB -> CPUID_PAGE1GB CPUID2_PCLMUL -> CPUID2_PCLMULQDQ CPUID2_CID -> CPUID2_CNXTID CPUID2_xTPR -> CPUID2_XTPR CPUID2_AES -> CPUID2_AESNI To match the x86 specification and the other OSes. - Remove: CPUID_B10, CPUID_B20, CPUID_IA64. They do not exist.
2020-08-16Fix AES NEON code for big-endian softfp ARM.riastradh
...which is how the kernel runs. Switch to using __SOFTFP__ for consistency with how it gets exposed to C, although I'm not sure how to get it defined automagically in the toolchain for .S files so that's set manually in files.aesneon for now.
2020-08-10Add hack to compile aes_ccm_tag() with -O0 for m68k for GCC8.rin
GCC 8 miscompiles aes_ccm_tag() for m68k with optimization level -O[12], which results in failure in aes_ccm_selftest(): | aes_ccm_selftest: tag 0: 8 bytes @ 0x4d3e38 | 03 80 5f 08 22 6f cb fe | .._."o.. | aes_ccm_selftest: verify 0 failed | ... | WARNING: module error: built-in module aes_ccm failed its MODULE_CMD_INIT, error 5 This is observed for amiga (A1200, 68060), mac68k (Quadra 840AV, 68040), and luna68k (nono, 68030 emulator). However, it is not for sun3 (TME, 68020 emulator) and sun2 (TME, 68010 emulator). At the moment, it is unclear whether this is due to differences b/w 68010-20 vs 68030-60, or something wrong with TME.
2020-08-09Fix some clang neon intrinsics.riastradh
Compile-tested only, with -Wno-nonportable-vector-initializers. Need to address -- and test -- this stuff properly but this is progress.
2020-08-09Use vshlq_n_s32 rather than vsliq_n_s32 with zero destination.riastradh
Not sure why I reached for vsliq_n_s32 at first -- probably so I wouldn't have to deal with a new intrinsic in arm_neon.h!
2020-08-09Nix outdated comment.riastradh
I implemented this parallelism a couple weeks ago.
2020-08-09Fix mistake in big-endian arm clang.riastradh
Swapped the two halves (only gcc does that, I think) and wrote j,i backwards, oops. (I don't have a big-endian arm clang build handy to test; hoping this works.)
2020-08-08Fix ARM NEON implementations of AES and ChaCha on big-endian ARM.riastradh
New macros such as VQ_N_U32(a,b,c,d) for NEON vector initializers. Needed because GCC and Clang disagree on the ordering of lanes, depending on whether it's 64-bit big-endian, 32-bit big-endian, or little-endian -- and, bizarrely, both of them disagree with the architectural numbering of lanes. Experimented with using static const uint8_t x8[16] = {...}; uint8x16_t x = vld1q_u8(x8); which doesn't require knowing anything about the ordering of lanes, but this generates considerably worse code and apparently confuses GCC into not recognizing the constant value of x8. Fix some clang mistakes while here too.
2020-07-28Draft 2x vectorized neon vpaes for aarch64.riastradh
Gives a modest speed boost on rk3399 (Cortex-A53/A72), around 20% in cgd tests, for parallelizable operations like CBC decryption; same improvement should probably carry over to rpi4 CPU which lacks ARMv8.0-AES.
2020-07-28Initialize authctr in both branches.riastradh
I guess I didn't test the unaligned case, weird.
2020-07-27Add RCSIDs to the AES and ChaCha .S sources.riastradh
2020-07-27Issue aese/aesmc and aesd/aesimc in pairs.riastradh
Advised by the aarch64 optimization guide; increases cgd throughput by about 10%.
2020-07-27Align critical-path loops in AES and ChaCha.riastradh
2020-07-27PIC for aes_neon_32.S.riastradh
Without this, tests/sys/crypto/aes/t_aes fails to start on armv7 because of R_ARM_ABS32 relocations in a nonwritable text segment for a PIE -- which atf quietly ignores in the final report! Yikes.
2020-07-27New sysctl subtree kern.crypto.riastradh
kern.crypto.aes.selected (formerly hw.aes_impl) kern.crypto.chacha.selected (formerly hw.chacha_impl) XXX Should maybe deduplicate creation of kern.crypto.
2020-07-27Gather auth[16] and ctr[16] into one authctr[32].riastradh
Should appease clang.
2020-07-26Ensure aes_ccm module init runs after aes module init.riastradh
Otherwise the AES implementation might not be selected early enough.
2020-07-25Add some Intel intrinsics for ChaCha.riastradh
_mm_load1_ps _mm_loadu_si128 _mm_movelh_ps _mm_slli_epi32 _mm_storeu_si128 _mm_unpackhi_epi32 _mm_unpacklo_epi32
2020-07-25Fix target attribute on _mm_movehl_ps, fix clang _mm_unpacklo_epi64.riastradh
- _mm_movehl_ps is available in SSE2, no need for SSSE3. - _mm_unpacklo_epi64 operates on v2di, not v4si; fix.
2020-07-25Add 32-bit load, store, and shift intrinsics.riastradh
vld1q_u32 vst1q_u32 vshlq_n_u32 vshrq_n_u32
2020-07-25Fix missing clang big-endian case.riastradh
2020-07-25Make aes boot message verbose-only.riastradh
2020-07-25Remove now-unused legacy rijndael API.riastradh
2020-07-25Remove now-needless AES-CCM fallback logic.riastradh
These paths are no longer exercised because all of the aes_impls now do the AES-CCM operations.
2020-07-25Implement AES-CCM with NEON.riastradh
2020-07-25Implement AES-CCM with ARMv8.5-AES.riastradh
2020-07-25Invert some loops to save a branch instruction on every iteration.riastradh
2020-07-25Implement AES-CCM with VIA ACE.riastradh
2020-07-25Implement AES-CCM with SSSE3.riastradh
2020-07-25Implement AES-CCM with SSE2.riastradh
2020-07-25Implement AES-CCM with x86 AES-NI.riastradh
2020-07-25Implement AES-CCM with BearSSL's bitsliced 32-bit aes_ct.riastradh
2020-07-25Push CBC-MAC and CCM block updates into the aes_impl API.riastradh
This should help reduce the setup and teardown overhead (enabling and disabling fpu, or expanding bitsliced keys) for CCM, as used in 802.11 WPA2 CCMP. But all the fiddly formatting details remain in aes_ccm.c to reduce the effort of implementing it -- at the cost of a handful additional setups and teardowns per message. Not yet implemented by any of the aes_impls, so leave a fallback that just calls aes_enc for now. This should be removed when all of the aes_impls provide CBC-MAC and CCM block updates.