| Age | Commit message (Collapse) | Author |
|
|
|
I had tested this change, but forgot to amend the commit before
exporting to CVS.
|
|
curcpu() might be stale by the time we're done, but it's still safe
to pass it to cpu_index, and this is just used as a best-effort
mechanism to keep I/O on queues handled by the same CPU.
bp is not always provided, and bp->b_ci is not always assigned,
e.g. when dumping. (If bp->b_ci is supposed to be always assigned,
then we need to audit all the paths into it to assign it in those
where it's not.)
Fixes dump on nvme.
|
|
|
|
|
|
|
|
for global commands is 0xffffffff. While the driver only supports 16bit
numbers (device minor & 0xffff), we need to use the full value for pass
through commands.
This fixes e.g. logpage requests on the controller level.
|
|
|
|
|
|
|
|
Nothing re-established this after suspend/resume, so attempting
suspend/resume/suspend would crash, and presumably we would miss
interrupts after resume. This keeps the establish/disestablish more
symmetric in attach/detach.
|
|
|
|
Simplify and make extensible the config_search() / config_found() /
config_attach() interfaces: rather than having different variants for
which arguments you want pass along, just have a single call that
takes a variadic list of tag-value arguments.
Adjust all call sites:
- Simplify wherever possible; don't pass along arguments that aren't
actually needed.
- Don't be explicit about what interface attribute is attaching if
the device only has one. (More simplification.)
- Add a config_probe() function to be used in indirect configuiration
situations, making is visibly easier to see when indirect config is
in play, and allowing for future change in semantics. (As of now,
this is just a wrapper around config_match(), but that is an
implementation detail.)
Remove unnecessary or redundant interface attributes where they're not
needed.
There are currently 5 "cfargs" defined:
- CFARG_SUBMATCH (submatch function for direct config)
- CFARG_SEARCH (search function for indirect config)
- CFARG_IATTR (interface attribte)
- CFARG_LOCATORS (locators array)
- CFARG_DEVHANDLE (devhandle_t - wraps OFW, ACPI, etc. handles)
...and a sentinel value CFARG_EOL.
Add some extra sanity checking to ensure that interface attributes
aren't ambiguous.
Use CFARG_DEVHANDLE in MI FDT, OFW, and ACPI code, and macppc and shark
ports to associate those device handles with device_t instance. This
will trickle trough to more places over time (need back-end for pre-OFW
Sun OBP; any others?).
|
|
visible
|
|
handle multiple nvme_rescan()s correctly by doing the
name-space identify only once per nsid.
fixes issue where modloading triggers multiple
rescans.
|
|
Now, ld(4) at nvme(4) works perfectly for aarch64eb!
Tested on ROCKPro64 and qemu-system-aarch64.
|
|
be swapped in nvme_pt_fill().
|
|
move name space availability check from ld_nvme.c:ld_nvme_attach()
to nvme.c:nvme_rescan().
this avoids allocation of ld(4) instances for every possible
name space, even if it is not usable. it also reduces the device
node flood generated from that strategy.
|
|
|
|
ok nonaka@. thanks
|
|
|
|
|
|
A new completion queue entry check incorrectly determined that there was
a Completion Queue entry for a command that was not submitted.
Fix PR kern/54275, PR kern/54503, PR kern/54532.
|
|
In normal operations with multiple queues, the nvme driver will attempt
to schedule I/O requests on the submitting CPU. This breaks down when any
one of the queues becomes full; the driver returns EAGAIN to the disk
layer, which causes the disk layer to stop submitting more requests until
the blocked request is consumed. When space becomes available in the full
queue, it pulls the next buffer from the bufq and fills the queue again,
until finally hitting EAGAIN and preventing other queues from processing
requests.
Two changes here to fix the problem:
- When processing requests from the bufq, attempt to assign them to the
queue associated with the CPU that originated the request.
- If that queue is busy, try to find another queue with available space
before returning EAGAIN. This way, only when all queues are full will
the disk layer stop submitting more requests.
Now for some real numbers. On a Rockchip RK3399 board (6 CPUs), with 6
concurrent readers:
Old code:
4294967296 bytes transferred in 52.420 secs (81933752 bytes/sec)
4294967296 bytes transferred in 53.969 secs (79582117 bytes/sec)
4294967296 bytes transferred in 55.391 secs (77539082 bytes/sec)
4294967296 bytes transferred in 55.649 secs (77179595 bytes/sec)
4294967296 bytes transferred in 56.102 secs (76556402 bytes/sec)
4294967296 bytes transferred in 72.901 secs (58915066 bytes/sec)
New code:
4294967296 bytes transferred in 37.171 secs (115546186 bytes/sec)
4294967296 bytes transferred in 37.611 secs (114194445 bytes/sec)
4294967296 bytes transferred in 37.655 secs (114061009 bytes/sec)
4294967296 bytes transferred in 38.247 secs (112295534 bytes/sec)
4294967296 bytes transferred in 38.496 secs (111569183 bytes/sec)
4294967296 bytes transferred in 38.595 secs (111282997 bytes/sec)
|
|
and also in nvme_dmamem_alloc(), allow as many DMA segment as would be
maximally needed for the size, rather than hard coding '2' for the form
and '1' for the latter.
now ld@nvme on i386 doesn't crash and i see at least 1.3GB/sec.
|
|
|
|
fix the Get Features call for DIOCGCACHE to actually retrieve the current
value properly
|
|
|
|
from FreeBSD nvmecontolr(8) r329824.
|
|
Pick from FreeBSD nvme(4) r326937.
|
|
|
|
memory corruption and possible panic on boot
PR kern/53059
|
|
should fix PR kern/53059 by Frank Kardel
|
|
for other infrequent commands
it uses single condvar for simplicity, and uses it both when waiting
for ccb or command completion - this is fine, since usually there
will be just one such command qeueued anyway
use this to finally properly implement DIOCCACHESYNC - return only after
the command is confirmed as completed by the controller
|
|
no need to maintain a counter _and_ q_ccb_list
this fixes deadlock when all ccbs happen to be taken before completion
interrupt - nvme_q_complete() increased q_nccbs_avail only after
processing all the completed commands, by then there was nothing
left to actually kick the disk queue again into action
this also fixes ccb leak on command errors e.g. with bus_dmamem_alloc()
or bus_dmamel_load() - q_nccbs_avail was never decreased on the error path
fixes PR kern/52769 by Martin Husemann, thanks to Paul Goyette
for testing
|
|
- KASSERT to prevent memory leak.
|
|
Down with externs in .c!
|
|
kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()
all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.
|
|
> NVM_ADMIN_DEL_IOCQ does not need prp1 (just as NVM_ADMIN_DEL_IOSQ).
> Remove what is likely a cut'n'paste error from the *_ADD_* code.
from OpenBSD nvme.c r1.56.
|
|
> some (broken) controllers require ordered split transfers.
> From linux a310acd7a7ea53533886c11bb7edd11ffd61a036
from OpenBSD nvme.c r1.53.
|
|
from OpenBSD nvme.c r1.52.
|
|
to be set for I/O; implement support in sd(4) and nvme(4)
discussed on tech-kern
|
|
|
|
|
|
|
|
attach/detach and for nvmectl(8), so there is actually no point having it big
|
|
if bigger then MAXPHYS
multiply the queue size by number of queues for ld(4) sc_maxqueuecnt, so
that ld_diskstart() would try to use full capacity, instead of throttling
to one queue worth of commands
|
|
in flight is actually one less then queue size, head == tail means empty
queue
|
|
"""
slightly optimize memory access - change struct nvme_queue so that the
struct dmamem members are allocated as part of it, instead of separate
kmem_alloc()s
"""
that change quite curiously caused completion queue corruption on MP systems,
regardless of MPSAFE setting for the pci/softintr interrupt
|
|
|