| Age | Commit message (Collapse) | Author |
|
I'm leaving in the conditional around the legacy membar_enters
(store-before-load, store-before-store) in kern_mutex.c and in
kern_lock.c because they may still matter: store-before-load barriers
tend to be the most expensive kind, so eliding them is probably
worthwhile on x86. (It also may not matter; I just don't care to do
measurements right now, and it's a single valid and potentially
justifiable use case in the whole tree.)
However, membar_release/acquire can be mere instruction barriers on
all TSO platforms including x86, so there's no need to go out of our
way with a bad API to conditionalize them. If the procedure call
overhead is measurable we just could change them to be macros on x86
that expand into __insn_barrier.
Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/23/msg028729.html
|
|
Can't find anything this is supposed to pair with. Pretty sure this
is just an optimistic unlocked test, not actually reliant on memory
ordering. But as it is unlocked, it needs to be coordinated with
atomic_load/store_relaxed, not ordinary loads or stores, if for no
other reason than to pacify sanitizers.
No need in vnalloc_marker or vcache_alloc because these still have
exclusive access to the vnode at that point.
XXX Should deduplicate the logic in vstate_assert_change and
vstate_change.
|
|
Also use it for extern spec_vnodeop_p, which is already there.
|
|
XXX regen sys/kern/vnode_if.c and the others
|
|
vnode state to VS_RECLAIMING, before we actually call VOP_RECLAIM(),
which will release the reference on the lower node of a stacked FS
vnode, which is likely to free the upper node's v_klist backing store.
Acquire the vnode interlock when checking for kevent interest now,
because the vp->v_klist pointer is now volatile.
PR kern/56950
|
|
like nullfs, make the upper vnode share that status with the lower vnode.
And, lo, NetBSD 9.99.99.
Fixes PR kern/56713.
|
|
|
|
This just goes through my recent reference count membar audit and
changes membar_exit to membar_release and membar_enter to
membar_acquire -- this should make everything cheaper on most CPUs
without hurting correctness, because membar_acquire is generally
cheaper than membar_enter.
|
|
Previously, it was possible for spec_node_lookup_by_dev to handle a
speconde that a concurrent spec_node_destroy is about to remove from
the hash table and then free, as soon as spec_node_lookup_by_dev
releases device_lock.
Now, the ordering is:
1. Remove specnode from hash table in spec_node_revoke. At this
point, no _new_ vnode references are possible (other than possibly
one acquired by vcache_vget under v_interlock), but there may be
existing ones.
2. Mark vnode reclaimed so vcache_vget will fail.
3. The last vrele (or equivalent logic in vcache_vget) will then free
the specnode in spec_node_destroy.
This way, _if_ a thread in spec_node_lookup_by_dev finds a specnode
in the hash table under device_lock/v_interlock, _then_ it will not
be freed until the thread completes vcache_vget.
This change requires calling spec_node_revoke unconditionally for
device special nodes, not just for active ones. Might introduce
slightly more contention on device_lock but not much because we
already have to take it in this path anyway a little later in
spec_node_destroy.
|
|
vdevgone relies on this to ensure that if there is a concurrent
revoke in progress, it will wait for that revoke to finish -- that
way, it can guarantee all I/O operations have completed and the
device is closed.
|
|
Remove unused predicates vn_locked() and vn_anylocked().
Welcome to 9.99.95
|
|
|
|
Assert "usecount == 1" instead.
|
|
If two threads are using an object that is freed when the reference
count goes to zero, we need to ensure that all memory operations
related to the object happen before freeing the object.
Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one
thread takes responsibility for freeing, but it's not enough to
ensure that the other thread's memory operations happen before the
freeing.
Consider:
Thread A Thread B
obj->foo = 42; obj->baz = 73;
mumble(&obj->bar); grumble(&obj->quux);
/* membar_exit(); */ /* membar_exit(); */
atomic_dec -- not last atomic_dec -- last
/* membar_enter(); */
KASSERT(invariant(obj->foo,
obj->bar));
free_stuff(obj);
The memory barriers ensure that
obj->foo = 42;
mumble(&obj->bar);
in thread A happens before
KASSERT(invariant(obj->foo, obj->bar));
free_stuff(obj);
in thread B. Without them, this ordering is not guaranteed.
So in general it is necessary to do
membar_exit();
if (atomic_dec_uint_nv(&obj->refcnt) != 0)
return;
membar_enter();
to release a reference, for the `last one out hit the lights' style
of reference counting. (This is in contrast to the style where one
thread blocks new references and then waits under a lock for existing
ones to drain with a condvar -- no membar needed thanks to mutex(9).)
I searched for atomic_dec to find all these. Obviously we ought to
have a better abstraction for this because there's so much copypasta.
This is a stop-gap measure to fix actual bugs until we have that. It
would be nice if an abstraction could gracefully handle the different
styles of reference counting in use -- some years ago I drafted an
API for this, but making it cover everything got a little out of hand
(particularly with struct vnode::v_usecount) and I ended up setting
it aside to work on psref/localcount instead for better scalability.
I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I
only put it on things that look performance-critical on 5sec review.
We should really adopt membar_enter_preatomic/membar_exit_postatomic
or something (except they are applicable only to atomic r/m/w, not to
atomic_load/store_*, making the naming annoying) and get rid of all
the ifdefs.
|
|
Should fix assertion "vp->v_iflag & VI_TEXT" under load.
|
|
|
|
unlock the vnode after VOP_INCATIVE().
This was the last call from vrelel() to VOP_UNLOCK() with v_interlock held.
|
|
vcache_vget() or vache_tryvget() succeeds.
Use it to rerun VOP_INACTIVE() if another thread ran a vget()..vrele()
cycle while we inactivated our last reference.
|
|
to lock or defer it. Jump straight to decrement usecount and requeue.
|
|
via kn_selnext:
- klist_init()
- klist_fini()
- klist_insert()
- klist_remove()
These provide some API insulation from the implementation details of these
lists (but not completely; see vn_knote_attach() and vn_knote_detach()).
Currently just a wrapper around SLIST(9).
This will make it significantly easier to switch kn_selnext linkage
to a different kind of list.
|
|
Adjust the KASSERT() appropriately.
|
|
- Centralize vnode kevent handling in the VOP_*() wrappers, rather than
forcing each individual file system to deal with it (except VOP_RENAME(),
because VOP_RENAME() is a mess and we currently have 2 different ways
of handling it; at least it's reasonably well-centralized in the "new"
way).
- Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ,
compatible with the same events in FreeBSD.
- Track which kevent notifications clients are interested in receiving
to avoid doing work for events no one cares about (avoiding, e.g.
taking locks and traversing the klist to send a NOTE_WRITE when
someone is merely watching for a file to be deleted, for example).
In support of the above:
- Add support in vnode_if.sh for specifying PRE- and POST-op handlers,
to be invoked before and after vop_pre() and vop_post(), respectively.
Basic idea from FreeBSD, but implemented differently.
- Add support in vnode_if.sh for specifying CONTEXT fields in the
vop_*_args structures. These context fields are used to convey information
between the file system VOP function and the VOP wrapper, but do not
occupy an argument slot in the VOP_*() call itself. These context fields
are initialized and subsequently interpreted by PRE- and POST-op handlers.
- Version VOP_REMOVE(), uses the a context field for the file system to report
back the resulting link count of the target vnode. Return this in tmpfs,
udf, nfs, chfs, ext2fs, lfs, and ufs.
NetBSD 9.99.92.
|
|
|
|
If we can't discern whether we have an exclusive or shared lock, then
just unlock and don't play fast and loose with pretending that we
have an exclusive lock will work -- it won't.
|
|
|
|
- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.
- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.
- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.
|
|
|
|
|
|
- Use cv_signal() instead of cv_broadcast(), there is only one waiter.
- No need to signal if number of vnodes doesn't increase.
- Use kpause(1) instead of yield().
|
|
that hides the details and does atomic_load_relaxed(). Signature matches
FreeBSD.
|
|
|
|
|
|
and getcwd():
- push vnode locking back as far as possible.
- do most lookups directly in the namecache, avoiding vnode locks & refs.
- don't block new refs to vnodes across VOP_INACTIVE().
- get shared locks for VOP_LOOKUP() if the file system supports it.
- correct lock types for VOP_ACCESS() / VOP_GETATTR() in a few places.
Possible future enhancements:
- make the lookups lockless.
- support dotdot lookups by being lockless and inferring absence of chroot.
- maybe make it work for layered file systems.
- avoid vnode references at the root & cwd.
|
|
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.
Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.
|
|
|
|
index becomes per-directory (initially, a red-black tree). The remaining
changes on the branch to namei()/getcwd() will be merged in the future.
|
|
split of vmobjlock & v_interlock.
|
|
- Have a stab at clustering the members of vnode_t and vnode_impl_t in a
more cache-conscious way. With that done, go back to adjusting v_usecount
with atomics and keep vi_lock directly in vnode_impl_t (saves KVA).
- Allow VOP_LOCK(LK_NONE) for the benefit of VFS_VGET() and VFS_ROOT().
Make sure LK_UPGRADE always comes with LK_NOWAIT.
- Make cwdinfo use mostly lockless.
|
|
- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.
|
|
|
|
|
|
make it not cause problems:
- Cover it with exec_lock so the updates are not racy.
- Using fileassoc is silly. Just hang a pointer off the vnode.
|
|
might need it anyway.
|
|
- Partially sort the list of per-vnode namecache entries by using a TAILQ.
Put the real name to the head, and put dot and dotdot to the tail so that
cache_lookup_reverse() doesn't have to consider them.
|
|
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.
- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).
|
|
- Stop using atomics to maniupulate v_usecount. It was a mistake to begin
with. It doesn't work as intended unless the XLOCK bit is incorporated in
v_usecount and we don't have that any more. When I introduced this 10+
years ago it was to reduce pressure on v_interlock but it doesn't do that,
it just makes stuff disappear from lockstat output and introduces problems
elsewhere. We could do atomic usecounts on vnodes but there has to be a
well thought out scheme.
- Resurrect LK_UPGRADE/LK_DOWNGRADE which will be needed to work effectively
when there is increased use of shared locks on vnodes.
- Allocate the vnode lock using rw_obj_alloc() to reduce false sharing of
struct vnode.
- Put all of the LRU lists into a single cache line, and do not requeue a
vnode if it's already on the correct list and was requeued recently (less
than a second ago).
Kernel build before and after:
119.63s real 1453.16s user 2742.57s system
115.29s real 1401.52s user 2690.94s system
|
|
valid "mnt_transinfo" and remove now unneeded flag IMNT_HAS_TRANS.
Run fstrans_start()/fstrans_done() on dead_rootmount if FSTRANS_DEAD_ENABLED.
Should become the default for DIAGNOSTIC in the future.
|
|
Now v_mount is never NULL.
|
|
pass more information about the file to create.
Welcome to 8.99.30
|
|
|