summaryrefslogtreecommitdiff
path: root/sys/dev/ic/nvme.c
AgeCommit message (Collapse)Author
2022-09-13nvme(4): Don't leak memory for queues on every resume.riastradh
2022-08-30nvme(4): Actually check if bp is null as commented previously.riastradh
I had tested this change, but forgot to amend the commit before exporting to CVS.
2022-08-30nvme(4): If bp is null or bp->b_ci is not assigned, use curcpu().riastradh
curcpu() might be stale by the time we're done, but it's still safe to pass it to cpu_index, and this is just used as a best-effort mechanism to keep I/O on queues handled by the same CPU. bp is not always provided, and bp->b_ci is not always assigned, e.g. when dumping. (If bp->b_ci is supposed to be always assigned, then we need to audit all the paths into it to assign it in those where it's not.) Fixes dump on nvme.
2022-08-20nvme(4): Read cqe flags and cid in that order.riastradh
2022-08-15nvme(4): KASSERT(A && B) -> KASSERT(A); KASSERT(B)riastradh
2022-08-14nvme: Make sure that q_ccb_list is always accessed with the q lock held.jmcneill
2022-07-31The namespace id is a 32bit value, in particular the "all namespaces" valuemlelstv
for global commands is 0xffffffff. While the driver only supports 16bit numbers (device minor & 0xffff), we need to use the full value for pass through commands. This fixes e.g. logpage requests on the controller level.
2022-05-07Add support for Apple silicon NVME. Ported from OpenBSD.skrll
2021-11-16Trailing whitespaceskrll
2021-08-07Merge thorpej-cfargs2.thorpej
2021-05-29nvme(4): Move disestablishment of admin q interrupt to nvme_detach.riastradh
Nothing re-established this after suspend/resume, so attempting suspend/resume/suspend would crash, and presumably we would miss interrupts after resume. This keeps the establish/disestablish more symmetric in attach/detach.
2021-05-29nvme(4): Add suspend/resume, derived from OpenBSD.riastradh
2021-04-24Merge thorpej-cfargs branch:thorpej
Simplify and make extensible the config_search() / config_found() / config_attach() interfaces: rather than having different variants for which arguments you want pass along, just have a single call that takes a variadic list of tag-value arguments. Adjust all call sites: - Simplify wherever possible; don't pass along arguments that aren't actually needed. - Don't be explicit about what interface attribute is attaching if the device only has one. (More simplification.) - Add a config_probe() function to be used in indirect configuiration situations, making is visibly easier to see when indirect config is in play, and allowing for future change in semantics. (As of now, this is just a wrapper around config_match(), but that is an implementation detail.) Remove unnecessary or redundant interface attributes where they're not needed. There are currently 5 "cfargs" defined: - CFARG_SUBMATCH (submatch function for direct config) - CFARG_SEARCH (search function for indirect config) - CFARG_IATTR (interface attribte) - CFARG_LOCATORS (locators array) - CFARG_DEVHANDLE (devhandle_t - wraps OFW, ACPI, etc. handles) ...and a sentinel value CFARG_EOL. Add some extra sanity checking to ensure that interface attributes aren't ambiguous. Use CFARG_DEVHANDLE in MI FDT, OFW, and ACPI code, and macppc and shark ports to associate those device handles with device_t instance. This will trickle trough to more places over time (need back-end for pre-OFW Sun OBP; any others?).
2020-12-27Zero DMA memory after load, and add PREREAD sync op after to ensure it is ↵jmcneill
visible
2020-12-04PR kern/55839:kardel
handle multiple nvme_rescan()s correctly by doing the name-space identify only once per nsid. fixes issue where modloading triggers multiple rescans.
2020-11-10One more fix for big-endian machines.rin
Now, ld(4) at nvme(4) works perfectly for aarch64eb! Tested on ROCKPro64 and qemu-system-aarch64.
2020-09-24no need to swap pt.cmd.cdwXX at nvme_set_number_of_queues(). cmd.cdwXX will ↵ryo
be swapped in nvme_pt_fill().
2020-09-22PR kern/55674:kardel
move name space availability check from ld_nvme.c:ld_nvme_attach() to nvme.c:nvme_rescan(). this avoids allocation of ld(4) instances for every possible name space, even if it is not usable. it also reduces the device node flood generated from that strategy.
2020-07-28say what is not configured in nvme_print()jdolecek
2020-04-07avoid "panic: LOCKDEBUG: Mutex error: rw_vector_enter,309: spin lock held"ryo
ok nonaka@. thanks
2019-11-11nvme(4): Use the SET_FEATURES command to get the number of allocated queues.nonaka
2019-09-26nvme(4): Don't attach the device, if namespace not found.nonaka
2019-09-20Don't set Phase Tag bit of Completion Queue entry at nvme_poll_done().nonaka
A new completion queue entry check incorrectly determined that there was a Completion Queue entry for a command that was not submitted. Fix PR kern/54275, PR kern/54503, PR kern/54532.
2019-06-28Fix a performance issue where one busy queue can starve all other queues.jmcneill
In normal operations with multiple queues, the nvme driver will attempt to schedule I/O requests on the submitting CPU. This breaks down when any one of the queues becomes full; the driver returns EAGAIN to the disk layer, which causes the disk layer to stop submitting more requests until the blocked request is consumed. When space becomes available in the full queue, it pulls the next buffer from the bufq and fills the queue again, until finally hitting EAGAIN and preventing other queues from processing requests. Two changes here to fix the problem: - When processing requests from the bufq, attempt to assign them to the queue associated with the CPU that originated the request. - If that queue is busy, try to find another queue with available space before returning EAGAIN. This way, only when all queues are full will the disk layer stop submitting more requests. Now for some real numbers. On a Rockchip RK3399 board (6 CPUs), with 6 concurrent readers: Old code: 4294967296 bytes transferred in 52.420 secs (81933752 bytes/sec) 4294967296 bytes transferred in 53.969 secs (79582117 bytes/sec) 4294967296 bytes transferred in 55.391 secs (77539082 bytes/sec) 4294967296 bytes transferred in 55.649 secs (77179595 bytes/sec) 4294967296 bytes transferred in 56.102 secs (76556402 bytes/sec) 4294967296 bytes transferred in 72.901 secs (58915066 bytes/sec) New code: 4294967296 bytes transferred in 37.171 secs (115546186 bytes/sec) 4294967296 bytes transferred in 37.611 secs (114194445 bytes/sec) 4294967296 bytes transferred in 37.655 secs (114061009 bytes/sec) 4294967296 bytes transferred in 38.247 secs (112295534 bytes/sec) 4294967296 bytes transferred in 38.496 secs (111569183 bytes/sec) 4294967296 bytes transferred in 38.595 secs (111282997 bytes/sec)
2019-06-14in nvme_attach() when creating the admin queue to probe the device info,mrg
and also in nvme_dmamem_alloc(), allow as many DMA segment as would be maximally needed for the size, rather than hard coding '2' for the form and '1' for the latter. now ld@nvme on i386 doesn't crash and i see at least 1.3GB/sec.
2019-04-24Expose device type. You can query it with e.g. drvctl -p ld0 disk-info/type.mlelstv
2018-12-01support DIOCSCACHE + DKCACHE_WRITE if volatile write cache is presentjdolecek
fix the Get Features call for DIOCGCACHE to actually retrieve the current value properly
2018-12-01disestablish the interrupt on failure in nvme_q_create()jdolecek
2018-04-18nvmectl(8): Add big-endian support.nonaka
from FreeBSD nvmecontolr(8) r329824.
2018-04-18nvme(4): Added some delay before check RDY bit quirk when disabling device.nonaka
Pick from FreeBSD nvme(4) r326937.
2018-03-17also remove now duplicate nvme_ccb_put() call from nvme_get_number_of_queues()jdolecek
2018-03-17fix passthrough command usage also in nvme_get_number_of_queues(), fixesjdolecek
memory corruption and possible panic on boot PR kern/53059
2018-03-17switch handling of passthrough commands to use queue, instead of pollingjdolecek
should fix PR kern/53059 by Frank Kardel
2018-03-16refactor the locking code around DIOCGCACHE handling to be reusablejdolecek
for other infrequent commands it uses single condvar for simplicity, and uses it both when waiting for ccb or command completion - this is fine, since usually there will be just one such command qeueued anyway use this to finally properly implement DIOCCACHESYNC - return only after the command is confirmed as completed by the controller
2018-03-16stop using q_nccbs_avail for deciding whether there are available ccbs;jdolecek
no need to maintain a counter _and_ q_ccb_list this fixes deadlock when all ccbs happen to be taken before completion interrupt - nvme_q_complete() increased q_nccbs_avail only after processing all the completed commands, by then there was nothing left to actually kick the disk queue again into action this also fixes ccb leak on command errors e.g. with bus_dmamem_alloc() or bus_dmamel_load() - q_nccbs_avail was never decreased on the error path fixes PR kern/52769 by Martin Husemann, thanks to Paul Goyette for testing
2018-02-27- don't leak ccb on alloc failure.christos
- KASSERT to prevent memory leak.
2017-10-28Kill some more extern struct cfdriver declarations.riastradh
Down with externs in .c!
2017-06-01remove checks for failure after memory allocation calls that cannot fail:chs
kmem_alloc() with KM_SLEEP kmem_zalloc() with KM_SLEEP percpu_alloc() pserialize_create() psref_class_create() all of these paths include an assertion that the allocation has not failed, so callers should not assert that again.
2017-05-29nvme(4): Don't set prp1 for DEL_IOCQnonaka
> NVM_ADMIN_DEL_IOCQ does not need prp1 (just as NVM_ADMIN_DEL_IOSQ). > Remove what is likely a cut'n'paste error from the *_ADD_* code. from OpenBSD nvme.c r1.56.
2017-05-29nvme(4): Do not use bus_space_{read,write}_8() even on LP64 archs.nonaka
> some (broken) controllers require ordered split transfers. > From linux a310acd7a7ea53533886c11bb7edd11ffd61a036 from OpenBSD nvme.c r1.53.
2017-05-29nvme(4): Mask non relevant bits when pritting version number.nonaka
from OpenBSD nvme.c r1.52.
2017-04-05expose disk device FUA/DPO support via DIOCGCACHE, and allow the flagsjdolecek
to be set for I/O; implement support in sd(4) and nvme(4) discussed on tech-kern
2017-02-28implement DIOCGCACHEjdolecek
2017-02-13NVMe 1.2.1nonaka
2017-02-13nvme(4): Limit the number of queues to the number allocated in HW.nonaka
2016-11-01reduce admin queue size to save memory; it's only ever used duringjdolecek
attach/detach and for nvmectl(8), so there is actually no point having it big
2016-11-01pass maxphys from device rather then assuming MAXPHYS; it's clipped in ld(4)jdolecek
if bigger then MAXPHYS multiply the queue size by number of queues for ld(4) sc_maxqueuecnt, so that ld_diskstart() would try to use full capacity, instead of throttling to one queue worth of commands
2016-11-01tighter queue control - according to spec actual cap on number of commandsjdolecek
in flight is actually one less then queue size, head == tail means empty queue
2016-10-20revert change from rev. 1.12:jdolecek
""" slightly optimize memory access - change struct nvme_queue so that the struct dmamem members are allocated as part of it, instead of separate kmem_alloc()s """ that change quite curiously caused completion queue corruption on MP systems, regardless of MPSAFE setting for the pci/softintr interrupt
2016-10-19add debug code to check for completion queue corruptionjdolecek