summaryrefslogtreecommitdiff
path: root/sys/kern/vfs_vnode.c
AgeCommit message (Collapse)Author
2023-02-24kern: Eliminate most __HAVE_ATOMIC_AS_MEMBAR conditionals.riastradh
I'm leaving in the conditional around the legacy membar_enters (store-before-load, store-before-store) in kern_mutex.c and in kern_lock.c because they may still matter: store-before-load barriers tend to be the most expensive kind, so eliding them is probably worthwhile on x86. (It also may not matter; I just don't care to do measurements right now, and it's a single valid and potentially justifiable use case in the whole tree.) However, membar_release/acquire can be mere instruction barriers on all TSO platforms including x86, so there's no need to go out of our way with a bad API to conditionalize them. If the procedure call overhead is measurable we just could change them to be macros on x86 that expand into __insn_barrier. Discussed on tech-kern: https://mail-index.netbsd.org/tech-kern/2023/02/23/msg028729.html
2023-02-22_vstate_assert: Use atomic_load/store_relaxed. Omit membar_enter.riastradh
Can't find anything this is supposed to pair with. Pretty sure this is just an optimistic unlocked test, not actually reliant on memory ordering. But as it is unlocked, it needs to be coordinated with atomic_load/store_relaxed, not ordinary loads or stores, if for no other reason than to pacify sanitizers. No need in vnalloc_marker or vcache_alloc because these still have exclusive access to the vnode at that point. XXX Should deduplicate the logic in vstate_assert_change and vstate_change.
2022-10-26miscfs/specfs/specdev.h: New home for extern spec_vnodeop_opv_desc.riastradh
Also use it for extern spec_vnodeop_p, which is already there.
2022-10-26miscfs/deadfs/deadfs.h: New home for deadfs-related externs.riastradh
XXX regen sys/kern/vnode_if.c and the others
2022-08-05In vcache_reclaim(), post NOTE_REVOKE immediately after changing thethorpej
vnode state to VS_RECLAIMING, before we actually call VOP_RECLAIM(), which will release the reference on the lower node of a stacked FS vnode, which is likely to free the upper node's v_klist backing store. Acquire the vnode interlock when checking for kevent interest now, because the vp->v_klist pointer is now volatile. PR kern/56950
2022-07-18Make kqueue event status for vnodes shareable, and for stacked file systemsthorpej
like nullfs, make the upper vnode share that status with the lower vnode. And, lo, NetBSD 9.99.99. Fixes PR kern/56713.
2022-04-09vfs(9): Add XXX comment about unclear membar_enter.riastradh
2022-04-09sys: Use membar_release/acquire around reference drop.riastradh
This just goes through my recent reference count membar audit and changes membar_exit to membar_release and membar_enter to membar_acquire -- this should make everything cheaper on most CPUs without hurting correctness, because membar_acquire is generally cheaper than membar_enter.
2022-03-28specfs: Remove specnode from hash table in spec_node_revoke.riastradh
Previously, it was possible for spec_node_lookup_by_dev to handle a speconde that a concurrent spec_node_destroy is about to remove from the hash table and then free, as soon as spec_node_lookup_by_dev releases device_lock. Now, the ordering is: 1. Remove specnode from hash table in spec_node_revoke. At this point, no _new_ vnode references are possible (other than possibly one acquired by vcache_vget under v_interlock), but there may be existing ones. 2. Mark vnode reclaimed so vcache_vget will fail. 3. The last vrele (or equivalent logic in vcache_vget) will then free the specnode in spec_node_destroy. This way, _if_ a thread in spec_node_lookup_by_dev finds a specnode in the hash table under device_lock/v_interlock, _then_ it will not be freed until the thread completes vcache_vget. This change requires calling spec_node_revoke unconditionally for device special nodes, not just for active ones. Might introduce slightly more contention on device_lock but not much because we already have to take it in this path anyway a little later in spec_node_destroy.
2022-03-28specfs: Let spec_node_lookup_by_dev wait for reclaim to finish.riastradh
vdevgone relies on this to ensure that if there is a concurrent revoke in progress, it will wait for that revoke to finish -- that way, it can guarantee all I/O operations have completed and the device is closed.
2022-03-19Remove now unused VV_LOCKSWORK, all file systems support locking.hannken
Remove unused predicates vn_locked() and vn_anylocked(). Welcome to 9.99.95
2022-03-19Switch spec_vnodeop vector to real vnode locking, VV_LOCKSWORK now.hannken
2022-03-15vrelel(): No need to test usecount if VGET marker is clear.hannken
Assert "usecount == 1" instead.
2022-03-12sys: Membar audit around reference count releases.riastradh
If two threads are using an object that is freed when the reference count goes to zero, we need to ensure that all memory operations related to the object happen before freeing the object. Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one thread takes responsibility for freeing, but it's not enough to ensure that the other thread's memory operations happen before the freeing. Consider: Thread A Thread B obj->foo = 42; obj->baz = 73; mumble(&obj->bar); grumble(&obj->quux); /* membar_exit(); */ /* membar_exit(); */ atomic_dec -- not last atomic_dec -- last /* membar_enter(); */ KASSERT(invariant(obj->foo, obj->bar)); free_stuff(obj); The memory barriers ensure that obj->foo = 42; mumble(&obj->bar); in thread A happens before KASSERT(invariant(obj->foo, obj->bar)); free_stuff(obj); in thread B. Without them, this ordering is not guaranteed. So in general it is necessary to do membar_exit(); if (atomic_dec_uint_nv(&obj->refcnt) != 0) return; membar_enter(); to release a reference, for the `last one out hit the lights' style of reference counting. (This is in contrast to the style where one thread blocks new references and then waits under a lock for existing ones to drain with a condvar -- no membar needed thanks to mutex(9).) I searched for atomic_dec to find all these. Obviously we ought to have a better abstraction for this because there's so much copypasta. This is a stop-gap measure to fix actual bugs until we have that. It would be nice if an abstraction could gracefully handle the different styles of reference counting in use -- some years ago I drafted an API for this, but making it cover everything got a little out of hand (particularly with struct vnode::v_usecount) and I ended up setting it aside to work on psref/localcount instead for better scalability. I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I only put it on things that look performance-critical on 5sec review. We should really adopt membar_enter_preatomic/membar_exit_postatomic or something (except they are applicable only to atomic r/m/w, not to atomic_load/store_*, making the naming annoying) and get rid of all the ifdefs.
2022-03-09vrelel(): after all locks are in place check for new reference again.hannken
Should fix assertion "vp->v_iflag & VI_TEXT" under load.
2022-02-28vrelel(): no VOP_UNLOCK() with v_interlock or vmobjlock held.hannken
2022-02-17Do the space accounting before VOP_INACTIVE() so we mayhannken
unlock the vnode after VOP_INCATIVE(). This was the last call from vrelel() to VOP_UNLOCK() with v_interlock held.
2022-02-17Add a marker VUSECOUNT_VGET to v_usecount that gets set wheneverhannken
vcache_vget() or vache_tryvget() succeeds. Use it to rerun VOP_INACTIVE() if another thread ran a vget()..vrele() cycle while we inactivated our last reference.
2022-02-17If the vnode to vrelel() is already reclaimed there is no needhannken
to lock or defer it. Jump straight to decrement usecount and requeue.
2022-02-12Add inline functions to manipulate the klists that link up knotesthorpej
via kn_selnext: - klist_init() - klist_fini() - klist_insert() - klist_remove() These provide some API insulation from the implementation details of these lists (but not completely; see vn_knote_attach() and vn_knote_detach()). Currently just a wrapper around SLIST(9). This will make it significantly easier to switch kn_selnext linkage to a different kind of list.
2022-02-08Operation vfs_suspend() returns ENOENT if the mount is gone (IMNT_GONE).hannken
Adjust the KASSERT() appropriately.
2021-10-20Overhaul of the EVFILT_VNODE kevent(2) filter:thorpej
- Centralize vnode kevent handling in the VOP_*() wrappers, rather than forcing each individual file system to deal with it (except VOP_RENAME(), because VOP_RENAME() is a mess and we currently have 2 different ways of handling it; at least it's reasonably well-centralized in the "new" way). - Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ, compatible with the same events in FreeBSD. - Track which kevent notifications clients are interested in receiving to avoid doing work for events no one cares about (avoiding, e.g. taking locks and traversing the klist to send a NOTE_WRITE when someone is merely watching for a file to be deleted, for example). In support of the above: - Add support in vnode_if.sh for specifying PRE- and POST-op handlers, to be invoked before and after vop_pre() and vop_post(), respectively. Basic idea from FreeBSD, but implemented differently. - Add support in vnode_if.sh for specifying CONTEXT fields in the vop_*_args structures. These context fields are used to convey information between the file system VOP function and the VOP wrapper, but do not occupy an argument slot in the VOP_*() call itself. These context fields are initialized and subsequently interpreted by PRE- and POST-op handlers. - Version VOP_REMOVE(), uses the a context field for the file system to report back the resulting link count of the target vnode. Return this in tmpfs, udf, nfs, chfs, ext2fs, lfs, and ufs. NetBSD 9.99.92.
2021-04-01Add a sysctl hashstat collector for vcache.simonb
2020-08-04Fix bogus fast path in vput.riastradh
If we can't discern whether we have an exclusive or shared lock, then just unlock and don't play fast and loose with pretending that we have an exclusive lock will work -- it won't.
2020-06-14If a vnode is marked with VI_EXECMAP then in all likelyhood it has pages.ad
2020-06-11Counter tweaks:ad
- Don't need to count anonpages+filepages any more; clean+unknown+dirty for each kind of page can be summed to get the totals. - Track the number of free pages with a counter so that it's one less thing for the allocator to do, which opens up further options there. - Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot. For the cheap option, give cpu_count_sync() a boolean parameter indicating that a cached value is okay, and rate limit the updates for cached values to hz.
2020-05-26Make vcache_tryvget() lockless. Reviewed by hannken@.ad
2020-05-18vrele_flush(): yield() every 100ms like we do it in vflush().hannken
2020-04-19Take some pressure from vdrain lock:hannken
- Use cv_signal() instead of cv_broadcast(), there is only one waiter. - No need to signal if number of vnodes doesn't increase. - Use kpause(1) instead of yield().
2020-04-13Replace most uses of vp->v_usecount with a call to vrefcnt(vp), a functionad
that hides the details and does atomic_load_relaxed(). Signature matches FreeBSD.
2020-04-13hardclock_ticks -> getticks()maxv
2020-04-04vrelel(): clear VV_MAPPED with the vnode still locked.ad
2020-04-04Merge the remaining changes from the ad-namecache branch, affecting namei()ad
and getcwd(): - push vnode locking back as far as possible. - do most lookups directly in the namecache, avoiding vnode locks & refs. - don't block new refs to vnodes across VOP_INACTIVE(). - get shared locks for VOP_LOOKUP() if the file system supports it. - correct lock types for VOP_ACCESS() / VOP_GETATTR() in a few places. Possible future enhancements: - make the lookups lockless. - support dotdot lookups by being lockless and inferring absence of chroot. - maybe make it work for layered file systems. - avoid vnode references at the root & cwd.
2020-03-22Process concurrent page faults on individual uvm_objects / vm_amaps inad
parallel, where the relevant pages are already in-core. Proposed on tech-kern. Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until adjustments are made to their pmaps.
2020-03-22Fix build failure.ad
2020-03-22Merge vfs_cache.c from the ad-namecache branch. With this the namecachead
index becomes per-directory (initially, a red-black tree). The remaining changes on the branch to namei()/getcwd() will be merged in the future.
2020-02-27Tighten up the locking around vp->v_iflag a little more after the recentad
split of vmobjlock & v_interlock.
2020-02-23Merge from ad-namecache:ad
- Have a stab at clustering the members of vnode_t and vnode_impl_t in a more cache-conscious way. With that done, go back to adjusting v_usecount with atomics and keep vi_lock directly in vnode_impl_t (saves KVA). - Allow VOP_LOCK(LK_NONE) for the benefit of VFS_VGET() and VFS_ROOT(). Make sure LK_UPGRADE always comes with LK_NOWAIT. - Make cwdinfo use mostly lockless.
2020-02-23UVM locking changes, proposed on tech-kern:ad
- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock. - Break v_interlock and vmobjlock apart. v_interlock remains a mutex. - Do partial PV list locking in the x86 pmap. Others to follow later.
2020-01-23Do not clean up segvguard while holding v_interlock.cad
2020-01-23#ifdef _KERNEL_OPT for previousad
2020-01-23PAX_SEGVGUARD doesn't seem to work properly in testing for me, but at leastad
make it not cause problems: - Cover it with exec_lock so the updates are not racy. - Using fileassoc is silly. Just hang a pointer off the vnode.
2020-01-12vput(): don't drop the vnode lock, carry the hold over into vrelel() whichad
might need it anyway.
2020-01-08- options NAMECACHE_ENTER_REVERSE is no more.ad
- Partially sort the list of per-vnode namecache entries by using a TAILQ. Put the real name to the head, and put dot and dotdot to the tail so that cache_lookup_reverse() doesn't have to consider them.
2019-12-16- Extend the per-CPU counters matt@ did to include all of the hot countersad
in UVM, excluding uvmexp.free, which needs special treatment and will be done with a separate commit. Cuts system time for a build by 20-25% on a 48 CPU machine w/DIAGNOSTIC. - Avoid 64-bit integer divide on every fault (for rnd_add_uint32).
2019-12-01Minor vnode locking changes:ad
- Stop using atomics to maniupulate v_usecount. It was a mistake to begin with. It doesn't work as intended unless the XLOCK bit is incorporated in v_usecount and we don't have that any more. When I introduced this 10+ years ago it was to reduce pressure on v_interlock but it doesn't do that, it just makes stuff disappear from lockstat output and introduces problems elsewhere. We could do atomic usecounts on vnodes but there has to be a well thought out scheme. - Resurrect LK_UPGRADE/LK_DOWNGRADE which will be needed to work effectively when there is increased use of shared locks on vnodes. - Allocate the vnode lock using rw_obj_alloc() to reduce false sharing of struct vnode. - Put all of the LRU lists into a single cache line, and do not requeue a vnode if it's already on the correct list and was requeued recently (less than a second ago). Kernel build before and after: 119.63s real 1453.16s user 2742.57s system 115.29s real 1401.52s user 2690.94s system
2019-02-20Attach "mnt_transinfo" to "dead_rootmount" so every mount has ahannken
valid "mnt_transinfo" and remove now unneeded flag IMNT_HAS_TRANS. Run fstrans_start()/fstrans_done() on dead_rootmount if FSTRANS_DEAD_ENABLED. Should become the default for DIAGNOSTIC in the future.
2019-02-20Assign vnode to dead_rootmount before vcache_dealloc() releases it.hannken
Now v_mount is never NULL.
2019-01-01Add "void *extra" argument to vcache_new() so a file system mayhannken
pass more information about the file to create. Welcome to 8.99.30
2017-09-22Fix non-DIAGNOSTICS build by adjusting _vstate_assert here too.joerg