summaryrefslogtreecommitdiff
path: root/common/lib/libc/arch/arm/atomic
AgeCommit message (Collapse)Author
2022-04-09Introduce membar_acquire/release. Deprecate membar_enter/exit.riastradh
The names membar_enter/exit were unclear, and the documentation of membar_enter has disagreed with the implementations on sparc, powerpc, and even x86(!) for the entire time it has been in NetBSD. The terms `acquire' and `release' are ubiquitous in the literature today, and have been adopted in the C and C++ standards to mean load-before-load/store and load/store-before-store, respectively, which are exactly the orderings required by acquiring and releasing a mutex, as well as other useful applications like decrementing a reference count and then freeing the underlying object if it went to zero. Originally I proposed changing one word in the documentation for membar_enter to make it load-before-load/store instead of store-before-load/store, i.e., to make it an acquire barrier. I proposed this on the grounds that (a) all implementations guarantee load-before-load/store, (b) some implementations fail to guarantee store-before-load/store, and (c) all uses in-tree assume load-before-load/store. I verified parts (a) and (b) (except, for (a), powerpc didn't even guarantee load-before-load/store -- isync isn't necessarily enough; need lwsync in general -- but it _almost_ did, and it certainly didn't guarantee store-before-load/store). Part (c) might not be correct, however: under the mistaken assumption that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then membar-r/rw, I only audited the cases of membar_enter that _aren't_ immediately after an atomic-r/m/w. All of those cases assume load-before-load/store. But my assumption was wrong -- there are cases of atomic-r/m/w then membar-w/rw that would be broken by changing to atomic-r/m/w then membar-r/rw: https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html Furthermore, the name membar_enter has been adopted in other places like OpenBSD where it actually does follow the documentation and guarantee store-before-load/store, even if that order is not useful. So the name membar_enter currently lives in a bad place where it means either of two things -- r/rw or w/rw. With this change, we deprecate membar_enter/exit, introduce membar_acquire/release as better names for the useful pair (r/rw and rw/w), and make sure the implementation of membar_enter guarantees both what was documented _and_ what was implemented, making it an alias for membar_sync. While here, rework all of the membar_* definitions and aliases. The new logic follows a rule to make it easier to audit: membar_X is defined as an alias for membar_Y iff membar_X is guaranteed by membar_Y. The `no stronger than' relation is (the transitive closure of): - membar_consumer (r/r) is guaranteed by membar_acquire (r/rw) - membar_producer (w/w) is guaranteed by membar_release (rw/w) - membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw) - membar_release (rw/w) is guaranteed by membar_sync (rw/rw) And, for the deprecated membars: - membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by membar_sync (rw/rw) - membar_exit (rw/w) is guaranteed by membar_release (rw/w) (membar_exit is identical to membar_release, but the name is deprecated.) Finally, while here, annotate some of the instructions with their semantics. For powerpc, leave an essay with citations on the unfortunate but -- as far as I can tell -- necessary decision to use lwsync, not isync, for membar_acquire and membar_consumer. Also add membar(3) and atomic(3) man page links.
2021-08-01s/overwriten/overwritten/ in comments.andvar
2021-07-28#define<tab> consistency.simonb
2021-07-28Remove memory barriers from the atomic_ops(3) atomic operations. They'reskrll
not needed for correctness. Add the correct memory barriers to the gcc legacy __sync built-in functions for atomic memory access. From the gcc documentation: In most cases, these built-in functions are considered a full barrier. That is, no memory operand is moved across the operation, either forward or backward. Further, instructions are issued as necessary to prevent the processor from speculating loads across the operation and from queuing stores after the operation. type __sync_lock_test_and_set (type *ptr, type value, ...) This built-in function is not a full barrier, but rather an acquire barrier. This means that references after the operation cannot move to (or be speculated to) before the operation, but previous memory stores may not be globally visible yet, and previous memory loads may not yet be satisfied. void __sync_lock_release (type *ptr, ...) This built-in function is not a full barrier, but rather a release barrier. This means that all previous memory stores are globally visible, and all previous memory loads have been satisfied, but following memory reads are not prevented from being speculated to before the barrier.
2021-07-10s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.skrll
2021-06-29Whitespaceskrll
2021-06-28Whitespaceskrll
2021-04-27Improve the membar_ops barriers - no need to use dsb and wait forskrll
completion. Also, we only to act on the inner shareability domain.
2021-04-26Add the appropriate memory barrier before the lock is cleared inskrll
__sync_lock_release_{1,2,4,8}. That is, all reads and write for in inner shareability domain before the lock clear store.
2021-04-24Trailing whitespaceskrll
2021-04-24Fix __sync_lock_release_4 to actually zeroise the whole 4bytes/32bits.skrll
2020-03-09Give the thumb atomic ops a chance of workingskrll
2019-09-16Traiing whitespace.skrll
2019-09-15__sync_{,x}or_and_fetch_8 should return new value... make it so.skrll
2019-02-18Add some atomic_cas_64_ni aliasesmartin
2015-12-11Use gcc 4.4 and later operation for nand, i.e.skrll
*ptr = ~(tmp & value) instead of *ptr = ~tmp & value There was also another bug in sync_fetch_and_nand_8 which I've also fixed. PR port-arm32/50513: Incorrect logic for atomic_nand_xx.S
2015-06-07Back out last change; fixed in the correct place.matt
2015-06-07Require ARMv5TE to assemble.joerg
2015-05-17Move arm sync_* changes to Makefile.incjustin
2015-05-17Do not build arm toolchain symbols in the rump kerneljustin
These symbols will be provided at link time and will be duplicate symbols in rump kernel and libc if defined. Many have been fixed previously, but these were missed as did not have a test.
2015-04-17Use the right register in previous. Spotted by matt@skrll
2015-04-17ARM ARM D7.3.2 - ensure all previous accesses are observed beforeskrll
the lock is cleared
2014-10-14Provide C++ 2011 <atomic> support functions for hppa and arm.martin
2014-10-13Move the and_{16,8}_nv sources into the right (libc only) block.martin
2014-10-13Provide __sync_and_and_fetch_2 and __sync_and_and_fetch_1 for pre-ARMv6,martin
they are needed for the C++ 2011 <atomic> stuff.
2014-07-05Provide a basic implementation of __atomic_load_* and __atomic_store_*,joerg
used by GCC and LLVM as backing for C11/C++11 atomics, if the hardware is not known to have corresponding features. Include it on ARM as LLVM and libc++ hit it when compiled for ARMv4.
2014-06-28Add aliases for the C11/C++11 spelling of the CAS primitives.joerg
2014-06-23Add aliases for the builtins used to implement C11/C++11 atomics.joerg
2014-03-28Ensure SBZ register is zeroskrll
2014-03-05apcs-gnu only passes one register on the stack.matt
ldrd always loads little endian (low address, low register).
2014-03-04Don't export __sync* if _KERNEL || _STANDALONE are defined.matt
(except if _RUMPKERNEL is defined)
2014-03-04Fix #if/#endif nestingmatt
2014-03-04fix typo.matt
2014-03-04Fix non-EABI loading of argument. Deal with endian issues.matt
Fixes PR/48635
2014-03-04Add atomic_sub_64.Smatt
2014-03-04Load new value from correct stack location in _atomic_cas_64_upmatt
2014-03-04Fetch value from correct stack location. Push an even number of registersmatt
so ldrd won't fail.
2014-02-27Add atomic_cas_64 support for ARM EABI on V5TE and V5TEJ cpus.matt
(strd is atomic).
2014-02-22Missed one __sync_* op (or gcc4.8 does inline it, while 4.5 does not?)martin
2014-02-22Try to hide the C runtime implementation specific __sync_* ops from librump,martin
to avoid duplicates.
2014-02-22Move the __sync_* ops added in the previous change to a libc-only sectionmartin
2014-02-21Provide the missing __sync_* ops for earlier arm versionsmartin
2014-01-30switch to unified syntaxmatt
2014-01-27Add _atomic_cas_16_up and _atomic_cas_8_upmatt
2013-11-30Use explicit form of register pair operations by specifying both.joerg
2013-11-30Explicitly name the register pairs.joerg
2013-11-08Add support for the gcc __sync builtins.matt
Note that these need earmv6 or later to get the ldrex/strex instructions
2013-08-20Push two registers to keep stack aligned.matt
2013-08-19Rework to allow thumb armv7 compilation.matt
Add atomic_simplelock.c for thumb
2013-08-19Thumbifymatt