| Age | Commit message (Collapse) | Author |
|
|
|
Discovered by code inspection. Remarkably, a combination of errors
made this fail to be a stack buffer overrun. Verified by booting
with ARMv8.0-AES disabled and with the self-test artificially made to
fail.
|
|
|
|
|
|
|
|
is not available, and any other value means it is.
|
|
|
|
|
|
- Call r12 by its usual name, ip.
- No need for r7 or r11=fp at the moment.
|
|
Cosmetic; no functional change.
|
|
Not sure why I didn't realize you could do this before!
Saves some temporary registers that can now be allocated to shave off
a few cycles.
|
|
|
|
|
|
Patch mainly from (and tested by) jakllsch@ with minor tweaks by me.
|
|
Make sure self-tests exercise this edge case.
Discovered by confusion over code inspection of jak's adaptation of
aes_armv8_64.S for big-endian.
|
|
We've already made the nonportable vector initializations portable; the
code works on aarch64eb.
|
|
Fixes NEON AES on aarch64eb
|
|
- Rename: CPUID_PN -> CPUID_PSN
CPUID_CFLUSH -> CPUID_CLFSH
CPUID_SBF -> CPUID_PBE
CPUID_LZCNT -> CPUID_ABM
CPUID_P1GB -> CPUID_PAGE1GB
CPUID2_PCLMUL -> CPUID2_PCLMULQDQ
CPUID2_CID -> CPUID2_CNXTID
CPUID2_xTPR -> CPUID2_XTPR
CPUID2_AES -> CPUID2_AESNI
To match the x86 specification and the other OSes.
- Remove: CPUID_B10, CPUID_B20, CPUID_IA64. They do not exist.
|
|
...which is how the kernel runs. Switch to using __SOFTFP__ for
consistency with how it gets exposed to C, although I'm not sure how
to get it defined automagically in the toolchain for .S files so
that's set manually in files.aesneon for now.
|
|
GCC 8 miscompiles aes_ccm_tag() for m68k with optimization level -O[12],
which results in failure in aes_ccm_selftest():
| aes_ccm_selftest: tag 0: 8 bytes @ 0x4d3e38
| 03 80 5f 08 22 6f cb fe | .._."o..
| aes_ccm_selftest: verify 0 failed
| ...
| WARNING: module error: built-in module aes_ccm failed its MODULE_CMD_INIT, error 5
This is observed for amiga (A1200, 68060), mac68k (Quadra 840AV, 68040),
and luna68k (nono, 68030 emulator). However, it is not for sun3 (TME, 68020
emulator) and sun2 (TME, 68010 emulator). At the moment, it is unclear
whether this is due to differences b/w 68010-20 vs 68030-60, or something
wrong with TME.
|
|
Compile-tested only, with -Wno-nonportable-vector-initializers. Need
to address -- and test -- this stuff properly but this is progress.
|
|
Not sure why I reached for vsliq_n_s32 at first -- probably so I
wouldn't have to deal with a new intrinsic in arm_neon.h!
|
|
I implemented this parallelism a couple weeks ago.
|
|
Swapped the two halves (only gcc does that, I think) and wrote j,i
backwards, oops.
(I don't have a big-endian arm clang build handy to test; hoping this
works.)
|
|
New macros such as VQ_N_U32(a,b,c,d) for NEON vector initializers.
Needed because GCC and Clang disagree on the ordering of lanes,
depending on whether it's 64-bit big-endian, 32-bit big-endian, or
little-endian -- and, bizarrely, both of them disagree with the
architectural numbering of lanes.
Experimented with using
static const uint8_t x8[16] = {...};
uint8x16_t x = vld1q_u8(x8);
which doesn't require knowing anything about the ordering of lanes,
but this generates considerably worse code and apparently confuses
GCC into not recognizing the constant value of x8.
Fix some clang mistakes while here too.
|
|
Gives a modest speed boost on rk3399 (Cortex-A53/A72), around 20% in
cgd tests, for parallelizable operations like CBC decryption; same
improvement should probably carry over to rpi4 CPU which lacks
ARMv8.0-AES.
|
|
I guess I didn't test the unaligned case, weird.
|
|
|
|
Advised by the aarch64 optimization guide; increases cgd throughput
by about 10%.
|
|
|
|
Without this, tests/sys/crypto/aes/t_aes fails to start on armv7
because of R_ARM_ABS32 relocations in a nonwritable text segment for
a PIE -- which atf quietly ignores in the final report! Yikes.
|
|
kern.crypto.aes.selected (formerly hw.aes_impl)
kern.crypto.chacha.selected (formerly hw.chacha_impl)
XXX Should maybe deduplicate creation of kern.crypto.
|
|
Should appease clang.
|
|
Otherwise the AES implementation might not be selected early enough.
|
|
_mm_load1_ps
_mm_loadu_si128
_mm_movelh_ps
_mm_slli_epi32
_mm_storeu_si128
_mm_unpackhi_epi32
_mm_unpacklo_epi32
|
|
- _mm_movehl_ps is available in SSE2, no need for SSSE3.
- _mm_unpacklo_epi64 operates on v2di, not v4si; fix.
|
|
vld1q_u32
vst1q_u32
vshlq_n_u32
vshrq_n_u32
|
|
|
|
|
|
|
|
These paths are no longer exercised because all of the aes_impls now
do the AES-CCM operations.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This should help reduce the setup and teardown overhead (enabling and
disabling fpu, or expanding bitsliced keys) for CCM, as used in
802.11 WPA2 CCMP. But all the fiddly formatting details remain in
aes_ccm.c to reduce the effort of implementing it -- at the cost of a
handful additional setups and teardowns per message.
Not yet implemented by any of the aes_impls, so leave a fallback that
just calls aes_enc for now. This should be removed when all of the
aes_impls provide CBC-MAC and CCM block updates.
|