summaryrefslogtreecommitdiff
path: root/sys/crypto/aes/arch
AgeCommit message (Collapse)Author
2022-06-26arm/aes_neon: Fix formatting of self-test failure message.riastradh
Discovered by code inspection. Remarkably, a combination of errors made this fail to be a stack buffer overrun. Verified by booting with ARMv8.0-AES disabled and with the self-test artificially made to fail.
2020-11-21Fix build with clang for earmv7hf; loadroundkey() is used only for __aarch64__.rin
2020-10-10Fix detection of NEON features. ID_AA64PFR0_EL1_ADV_SIMD_NONE means SIMDjmcneill
is not available, and any other value means it is.
2020-09-10aes neon: Gather mc_forward/backward so we can load 256 bits at once.riastradh
2020-09-10aes neon: Hoist dsbd/dsbe address calculation out of loop.riastradh
2020-09-10aes neon: Tweak register usage.riastradh
- Call r12 by its usual name, ip. - No need for r7 or r11=fp at the moment.
2020-09-10aes neon: Write vtbl with {qN} rather than {d(2N)-d(2N+1)}.riastradh
Cosmetic; no functional change.
2020-09-10aes neon: Issue 256-bit loads rather than pairs of 128-bit loads.riastradh
Not sure why I didn't realize you could do this before! Saves some temporary registers that can now be allocated to shave off a few cycles.
2020-09-08aesarmv8: Reallocate registers to shave off unnecessary MOV.riastradh
2020-09-08aesarmv8: Issue two 4-register ld/st, not four 2-register ld/st.riastradh
2020-09-08aesarmv8: Adapt aes_armv8_64.S to big-endian.riastradh
Patch mainly from (and tested by) jakllsch@ with minor tweaks by me.
2020-09-08aes(9): Fix edge case in bitsliced SSE2 AES-CBC decryption.riastradh
Make sure self-tests exercise this edge case. Discovered by confusion over code inspection of jak's adaptation of aes_armv8_64.S for big-endian.
2020-09-08Acknowledge clang warning for NEON cipher code on aarch64ebjakllsch
We've already made the nonportable vector initializations portable; the code works on aarch64eb.
2020-09-07Fix vgetq_lane_u32 for aarch64eb with GCCjakllsch
Fixes NEON AES on aarch64eb
2020-09-05x86: fix several CPUID flagsmaxv
- Rename: CPUID_PN -> CPUID_PSN CPUID_CFLUSH -> CPUID_CLFSH CPUID_SBF -> CPUID_PBE CPUID_LZCNT -> CPUID_ABM CPUID_P1GB -> CPUID_PAGE1GB CPUID2_PCLMUL -> CPUID2_PCLMULQDQ CPUID2_CID -> CPUID2_CNXTID CPUID2_xTPR -> CPUID2_XTPR CPUID2_AES -> CPUID2_AESNI To match the x86 specification and the other OSes. - Remove: CPUID_B10, CPUID_B20, CPUID_IA64. They do not exist.
2020-08-16Fix AES NEON code for big-endian softfp ARM.riastradh
...which is how the kernel runs. Switch to using __SOFTFP__ for consistency with how it gets exposed to C, although I'm not sure how to get it defined automagically in the toolchain for .S files so that's set manually in files.aesneon for now.
2020-08-09Fix some clang neon intrinsics.riastradh
Compile-tested only, with -Wno-nonportable-vector-initializers. Need to address -- and test -- this stuff properly but this is progress.
2020-08-09Use vshlq_n_s32 rather than vsliq_n_s32 with zero destination.riastradh
Not sure why I reached for vsliq_n_s32 at first -- probably so I wouldn't have to deal with a new intrinsic in arm_neon.h!
2020-08-09Nix outdated comment.riastradh
I implemented this parallelism a couple weeks ago.
2020-08-09Fix mistake in big-endian arm clang.riastradh
Swapped the two halves (only gcc does that, I think) and wrote j,i backwards, oops. (I don't have a big-endian arm clang build handy to test; hoping this works.)
2020-08-08Fix ARM NEON implementations of AES and ChaCha on big-endian ARM.riastradh
New macros such as VQ_N_U32(a,b,c,d) for NEON vector initializers. Needed because GCC and Clang disagree on the ordering of lanes, depending on whether it's 64-bit big-endian, 32-bit big-endian, or little-endian -- and, bizarrely, both of them disagree with the architectural numbering of lanes. Experimented with using static const uint8_t x8[16] = {...}; uint8x16_t x = vld1q_u8(x8); which doesn't require knowing anything about the ordering of lanes, but this generates considerably worse code and apparently confuses GCC into not recognizing the constant value of x8. Fix some clang mistakes while here too.
2020-07-28Draft 2x vectorized neon vpaes for aarch64.riastradh
Gives a modest speed boost on rk3399 (Cortex-A53/A72), around 20% in cgd tests, for parallelizable operations like CBC decryption; same improvement should probably carry over to rpi4 CPU which lacks ARMv8.0-AES.
2020-07-28Initialize authctr in both branches.riastradh
I guess I didn't test the unaligned case, weird.
2020-07-27Add RCSIDs to the AES and ChaCha .S sources.riastradh
2020-07-27Issue aese/aesmc and aesd/aesimc in pairs.riastradh
Advised by the aarch64 optimization guide; increases cgd throughput by about 10%.
2020-07-27Align critical-path loops in AES and ChaCha.riastradh
2020-07-27PIC for aes_neon_32.S.riastradh
Without this, tests/sys/crypto/aes/t_aes fails to start on armv7 because of R_ARM_ABS32 relocations in a nonwritable text segment for a PIE -- which atf quietly ignores in the final report! Yikes.
2020-07-25Add some Intel intrinsics for ChaCha.riastradh
_mm_load1_ps _mm_loadu_si128 _mm_movelh_ps _mm_slli_epi32 _mm_storeu_si128 _mm_unpackhi_epi32 _mm_unpacklo_epi32
2020-07-25Fix target attribute on _mm_movehl_ps, fix clang _mm_unpacklo_epi64.riastradh
- _mm_movehl_ps is available in SSE2, no need for SSSE3. - _mm_unpacklo_epi64 operates on v2di, not v4si; fix.
2020-07-25Add 32-bit load, store, and shift intrinsics.riastradh
vld1q_u32 vst1q_u32 vshlq_n_u32 vshrq_n_u32
2020-07-25Fix missing clang big-endian case.riastradh
2020-07-25Implement AES-CCM with NEON.riastradh
2020-07-25Implement AES-CCM with ARMv8.5-AES.riastradh
2020-07-25Invert some loops to save a branch instruction on every iteration.riastradh
2020-07-25Implement AES-CCM with VIA ACE.riastradh
2020-07-25Implement AES-CCM with SSSE3.riastradh
2020-07-25Implement AES-CCM with SSE2.riastradh
2020-07-25Implement AES-CCM with x86 AES-NI.riastradh
2020-07-25Split aes_impl declarations out into aes_impl.h.riastradh
This will make it less painful to add more operations to struct aes_impl without having to recompile everything that just uses the block cipher directly or similar.
2020-07-25Invert some loops to save a jmp instruction on each iteration.riastradh
No semantic change intended.
2020-07-23fix build with llvm/clang.ryo
2020-07-22Fix register name in comment.riastradh
Some time ago I reallocated the registers to avoid inadvertently clobbering the callee-saves v9, but neglected to update the comment.
2020-07-19fix build with clang/llvm.ryo
clang aarch64 assembler doesn't accept optional number of lanes of vector register. (but ARMARM says that an assembler must accept it)
2020-06-30Reallocate registers to avoid abusing callee-saves registers, v8-v15.riastradh
Forgot to consult the AAPCS before committing this before -- oops! While here, take advantage of the 32 aarch64 simd registers to avoid all stack spills.
2020-06-30Use `.arch_extension aes' for aese/aesmc/aesd/aesimc.riastradh
Unlike `.arch_extension crypto', this works with clang; both work with gas, so we'll go with this. Clang still can't handle aes_armv8_64.S yet -- it gets confused by dup and mov on lanes, but this makes progress.
2020-06-30Use .p2align rather than .align.riastradh
Apparently on arm, .align is actually an alias for .p2align, taking a power of two rather than a number of bytes, so aes_armv8_64.o was bloated to 32KB with obscene alignment when it only needed to be barely past 4KB. Do the same for the x86 aes_ni_64.S -- even though .align takes a number of bytes rather than a power of two on x86, let's just stay away from the temptations of the evil .align directive.
2020-06-30Tweak clang neon intrinsics so they build.riastradh
(this file is still a kludge)
2020-06-30New test sys/crypto/aes/t_aes.riastradh
Runs aes_selftest on all kernel AES implementations supported on the current hardware, not just the preferred one.
2020-06-30Limit aes_neon to cpu_cortex | aarch64.riastradh
We won't use it on any other systems, and it doesn't build without NEON anyway. Verified earmv7hf GENERIC, aarch64 GENERIC64, and earmv6 RPI2 all build with this.
2020-06-29Provide hand-written AES NEON assembly for arm32.riastradh
gcc does a lousy job at compiling 128-bit NEON intrinsics on arm32; hand-writing it made it about 12x faster, by avoiding a zillion loads and stores to spill everything and the kitchen sink onto the stack. (But gcc does fine on aarch64, presumably because it has twice as many registers and doesn't have to deal with q2=d4/d5 overlapping.)