| Age | Commit message (Collapse) | Author |
|
Previously the usage pattern was:
sleepq_enter(sq, l, lock); // locks l
...
sleepq_enqueue(sq, ..., sobj, ...); // assumes l locked, sets l_syncobj
... (*)
sleepq_block(...); // unlocks l
As long as l remains locked from sleepq_enter to sleepq_block,
l_syncobj is stable, and sleepq_block uses it via ktrcsw to determine
whether the sleep is on a mutex in order to avoid creating ktrace
context-switch records (which involves allocation which is forbidden
in softint context, while taking and even sleeping for a mutex is
allowed).
However, in turnstile_block, the logic at (*) also involves
turnstile_lendpri, which sometimes unlocks and relocks l. At that
point, another thread can swoop in and sleepq_remove l, which sets
l_syncobj to sched_syncobj. If that happens, ktrcsw does what is
forbidden -- tries to allocate a ktrace record for the context
switch.
As an optimization, sleepq_block or turnstile_block could stop early
if it detects that l_syncobj doesn't match -- we've already been
requested to wake up at this point so there's no need to mi_switch.
(And then it would be unnecessary to pass the syncobj through
sleepq_block, because l_syncobj would remain stable.) But I'll leave
that to another change.
Reported-by: syzbot+8b9d7b066c32dbcdc63b@syzkaller.appspotmail.com
|
|
No store-before-load ordering here -- this was obviously always
intended to be load-before-load/store all along.
|
|
via kn_selnext:
- klist_init()
- klist_fini()
- klist_insert()
- klist_remove()
These provide some API insulation from the implementation details of these
lists (but not completely; see vn_knote_attach() and vn_knote_detach()).
Currently just a wrapper around SLIST(9).
This will make it significantly easier to switch kn_selnext linkage
to a different kind of list.
|
|
|
|
true if the last knote was removed and there are no more knotes
on the selinfo.
- Use this new return value in filt_sordetach(), filt_sowdetach(),
filt_fifordetach(), and filt_fifowdetach() to know when to clear
SB_KOTE without having to know select/kqueue implementation details.
|
|
knote / kevent registration in the selinfo structure.
|
|
waits with turnstiles (not currently done).
|
|
select/poll used the FIFO method and that was for collisions which rarely
occur. Shrinks sleep_t and condvar_t.
|
|
- Update a comment.
|
|
Exceptions: when fd_refcnt <= 1, or when holding fd_lock.
While here:
- Restore KASSERT(mutex_owned(&fdp->fd_lock)) in fd_unused.
=> This is used only in fd_close and fd_abort, where it holds.
- Move bounds check assertion in fd_putfile to where it matters.
- Store fd_dt with atomic_store_release.
- Move load of fd_dt under lock in knote_fdclose.
- Omit membar_consumer in fdesc_readdir.
=> atomic_load_consume serves the same purpose now.
=> Was needed only on alpha anyway.
|
|
|
|
- Increase the maximum number of clusters from 32 to 64 for large systems.
kcpuset_t could potentially be used here but that's an excursion I don't
want to go on right now. uint32_t -> uint64_t is very simple.
- In the case of a non-blocking select/poll, or where we won't block
because there are events ready to report, stop registering interest in
the back-end objects early.
- Change the wmesg for poll back to "poll".
|
|
Later in the code selcommon() checks for proper timespec, check only
correct usec of timeval before type conversions.
|
|
|
|
Found by KUBSan.
|
|
|
|
makes poll(2) align with the Posix documentation (which allows EINVAL if
nfds > OPEN_MAX). From: Anthony Mallet
|
|
|
|
(hardcoded limit to 1000 + #<open-fds>). Changed to limit by the max of
the resource limit of open descriptors and the above.
|
|
|
|
kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()
all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.
|
|
a complicated way to call soreceive() with a sb_timeo. The only user
(netsmb) already did that anyway, so just had to delete the call to
pollsock().
|
|
the sysctl link sets are processed, and remove redundancy.
Shaves >13kB off of an amd64 GENERIC, not to mention >1k duplicate
lines of code.
|
|
Code inspection suggests that this fix is not likely to reveal any
latent problems.
|
|
|
|
NO_DIRECT_SELECT is defined or all polls return an event.
|
|
- When sel_do_scan() restarts do a full initialization with selclear() so
we start from an empty set without registered events. Defer the
evaluation of l_selret after selclear() and add the count of direct events
to the count of events.
- For selscan()/pollscan() zero the output descriptors before we poll and
for selscan() take the sc_lock before we change them.
- Change sel_setevents() to not count events already set.
Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>
Should fix PR #44763 (select/poll direct-set optimization seems racy)
and PR #45187 (select(2) sometimes doesn't wakeup)
|
|
using a signal mask. Tested by tron.
|
|
|
|
(and pollts(2)) are similar to sigsuspend(2) in that they temporarily change
the process signal mask and wait for signal delivery. Factor out and share the
code that does this.
|
|
if no polled/selected events were set; also, count the correct return
value for the select.
|
|
that scheduler locks are special in this regard - adaptive locks cannot
be in the path due to turnstiles. Randomly spotted/reported by uebayasi@.
- Remove unused lwp_relock() and replace lwp_lock_retry() by simplifying
lwp_lock() and sleepq_enter() a little.
- Give alllwp its own cache-line and mark lwp_cache pointer as read-mostly.
OK ad@
|
|
|
|
Spotted by Enami Tsugutomo.
|
|
|
|
|
|
as it might also be SEL_BLOCKING due to spurious wake-ups. That has no harm.
|
|
pipe subsystems. Avoids overhead of second selscan() on wake-up, and thus
improves performance on certain workloads (especially when polling on many
file-descriptors). Also, clean-up sys/fd_set.h header and improve macros.
Welcome to 5.99.36!
|
|
No ABI change.
|
|
|
|
open files is rather gross - the poll map isn't required to be dense.
Instead limit to a much larger value (1000 + dt_nfiles) so that user
programs cannot allocate indefinite sized blocks of kvm.
If the limit is exceeded, then return EINVAL instead of silently truncating
the list.
(The silent truncation in select isn't quite as bad - although even there
any high bits that are set ought to generate an EBADF response.)
Move the code that converts ERESTART and EWOULDBLOCK into common code.
Effectively fixes PR/17507 since the new limit is unlikely to be detected.
|
|
- Use cached curlwp->l_fd, instead of p->p_fd.
- Inline selscan/pollscan.
|
|
- Move abstimeout2timo() there too and export. Use it in lwp_park().
|
|
Avoids code duplication. XXX: pollsock() should be converted too, except
it's a bit ugly.
|
|
- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.
Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).
Discussed on <tech-kern>, reviewed by <ad>.
|
|
- Avoid atomics in more places.
- Remove the per-descriptor mutex, and just use filedesc_t::fd_lock.
It was only being used to synchronize close, and in any case we needed
to take fd_lock to free the descriptor slot.
- Optimize certain paths for the <NDFDFILE case.
- Sprinkle more comments and assertions.
- Cache more stuff in filedesc_t.
- Fix numerous minor bugs spotted along the way.
- Restructure how the open files array is maintained, for clarity and so
that we can eliminate the membar_consumer() call in fd_getfile(). This is
mostly syntactic sugar; the main functional change is that fd_nfiles now
lives alongside the open file array.
Some measurements with libmicro:
- simple file syscalls are like close() are between 1 to 10% faster.
- some nice improvements, e.g. poll(1000) which is ~50% faster.
|
|
of timeval (rides the uvm bump).
|
|
on !MP kernels, and reduces false sharing on MP ones.
|
|
|
|
|