summaryrefslogtreecommitdiff
path: root/drivers/block/nvme-core.c
AgeCommit message (Collapse)AuthorFilesLines
2013-09-07NVMe: Merge issue on character device bring-upKeith Busch1-4/+8
A recent patch made it possible to bring up the character handle when the device is responsive but not accepting a set-features command. Another recent patch moved the initialization that requires we move where the checks for this condition occur. This patch merges these two ideas so it works much as before. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Handle ioremap failureKeith Busch1-8/+22
Decrement the number of queues required for doorbell remapping until the memory is successfully mapped for that size. Additional checks are done so that we don't call free_irq if it has already been freed. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Add pci suspend/resume driver callbacksKeith Busch1-15/+58
Used for going in and out of low power states. Resuming reuses the IO queues from the previous initialization, freeing any allocated queues that are no longer usable. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Use normal shutdownKeith Busch1-0/+26
The NVMe spec recommends using the shutdown normal sequence when safely taking the controller offline instead of hitting CC.EN on the next start-up to reset the controller. The spec recommends a minimum of 1 second for the shutdown complete. This patch waits 2 seconds to be on the safe side. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Separate controller init from disk discoveryKeith Busch1-30/+47
This combines the controller initialization into one function, removing IO queue setup from namespace discovery, and creates symetric functions for device removal. The controller start and shutdown functions can now be called from resume/suspend context as well as probe/remove. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Separate queue alloc/free from create/deleteKeith Busch1-39/+94
This separates nvme queue allocation from creation, and queue deletion from freeing. This is so that we may in the future temporarily disable queues and reuse the same memory when bringing them back online, like coming back from suspend state. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Group pci related actions in functionsKeith Busch1-46/+66
This will make it easier to reuse these outside probe/remove. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Disk stats for read/write commands onlyKeith Busch1-3/+3
Flush and discard requests would previously mess up the accounting. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-09-04NVMe: Bring up cdev on set feature failureKeith Busch1-2/+2
This patch creates the character device as long as a device's admin queues are usable so a user has an opprotunity to perform administration tasks. A device may be in a state that does not allow IO and setting the queue count feature in such a state returns an error. Previously the driver would bail and the controller would be unusable. Signed-off-by: Keith Busch <keith.busch@intel.com>
2013-09-04NVMe: Fix checkpatch issuesKeith Busch1-5/+4
Signed-off-by: Keith Busch <keith.busch@intel.com>
2013-09-04NVMe: Namespace IDs are unsignedMatthew Wilcox1-2/+5
The 'Number of Namespaces' read from the device was being treated as signed, which would cause us to not scan any namespaces for a device with more than 2 billion namespaces. That led to noticing that the namespace ID was also being treated as signed, which could lead to the result from NVME_IOCTL_ID being treated as an error code. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-06-24NVMe: Call nvme_process_cq from submission pathMatthew Wilcox1-19/+20
Since we have the queue locked, it makes sense to check if there are any completion queue entries on the queue before we release the lock. If there are, it may save an interrupt and reduce latency for the I/Os that happened to complete. This happens fairly often for some workloads. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-06-24NVMe: Remove "process_cq did something" messageMatthew Wilcox1-2/+1
I was originally intending to log the fact that the kthread had done some work since it might help us find interrupt handling problems, but that hasn't been done yet, and spamming the logs with this message is just rude. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-06-24NVMe: Return correct value from interrupt handlerMatthew Wilcox1-5/+9
The interrupt handler currently reports whether it found any new completion queue entries. If the completion queue is primarily being processed by a method other than the interrupt handler, it may return IRQ_NONE so often that Linux thinks that the interrupt is being falsely triggered. To solve this problem, report whether any completion queue entries have been seen since the last interrupt was received for this queue. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-06-20NVMe: Disk IO statisticsKeith Busch1-0/+28
Add io stats accounting for bio requests so nvme block devices show useful disk stats. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-06-20NVMe: Restructure MSI / MSI-X setupMatthew Wilcox1-21/+23
The current code copies 'nr_io_queues' into 'q_count', modifies 'nr_io_queues' during MSI-X setup, then resets 'nr_io_queues' for MSI setup. Instead, copy 'nr_io_queues' into 'vecs' and modify 'vecs' during both MSI-X and MSI setup. This lets us simplify the for-loops that set up MSI-X and MSI, and opens the possibility of using more I/O queues than we have interrupt vectors, should future benchmarking prove that to be a useful feature. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-31NVMe: Add MSI supportRamachandra Rao Gajula1-8/+32
Some devices only have support for MSI, not MSI-X. While MSI is more limited, it still provides better performance than line-based interrupts. Signed-off-by: Ramachandra Gajula <rama@fastorsystems.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-29NVMe: Use dma_set_mask() correctlyMatthew Wilcox1-2/+8
In some circumstances setting a 64-bit DMA mask can fail, as explained in Documentation/DMA-API-HOWTO.txt. Use the recommended code sequence to set a 32-bit DMA mask if setting a 64-bit DMA mask fails. Reported-by: Chayan Biswas <Chayan.Biswas@sandisk.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-23Return the result from user admin command IOCTL even in case of failureChayan Biswas1-1/+1
We copy the result to user if the command is completed from the controller even if it completes with failure (non-zero) status. A return status of < 0 indicates the command was not completed by the controller. The user application may expect the error code in the result field in case of failure. Signed-off-by: Chayan Biswas <Chayan.Biswas@sandisk.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-17NVMe: Do not cancel command multiple timesKeith Busch1-0/+2
Cancelling an already cancelled command does not do anything, so check the command context before cancelling it, continuing if had already been cancelled so we do not log the same problem every second if a device stops responding. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-17NVMe: fix error return code in nvme_submit_bio_queue()Wei Yongjun1-1/+2
nvme_submit_flush_data() might overwrite the initialisation of the return value with 0, so move the -ENOMEM setting close to the usage. Signed-off-by: Wei Yongjun <yongjun_wei@trendmicro.com.cn> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-17NVMe: check for integer overflow in nvme_map_user_pages()Dan Carpenter1-2/+3
You need to have CAP_SYS_ADMIN to trigger this overflow but it makes the static checkers complain so we should fix it. The worry is that "length" comes from copy_from_user() so we need to check that "length + offset" can't overflow. I also changed the min_t() cast to be unsigned instead of signed. Now that we cap "length" to INT_MAX it doesn't make a difference, but it's a little easier for reviewers to know that large values aren't cast to negative. Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-10NVMe: Use user defined admin ioctl timeoutKeith Busch1-1/+5
Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-08NVMe: Only clear the enable bit when disabling controllerMatthew Wilcox1-1/+4
Many of the bits in the Controller Configuration register may only be modified when the Enable bit is clear. Clearing them at the same time as the Enable bit might be OK, but let's play it safe and only touch the Enable bit. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com> Reviewed-by: Keith Busch <keith.busch@intel.com>
2013-05-08NVMe: Wait for device to acknowledge shutdownMatthew Wilcox1-19/+46
A recent update to the specification makes it clear that the host is expected to wait for the device to acknowledge the Enable bit transitioning to 0 as well as waiting for the device to acknowledge a transition to 1. Reported-by: Khosrow Panah <Khosrow.Panah@idt.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com> Reviewed-by: Keith Busch <keith.busch@intel.com>
2013-05-02NVMe: Schedule timeout for sync commandsKeith Busch1-1/+1
Schedule a timeout on sync commands in case the command times out and the device is not being polled for timeouts. This prevents device removal from hanging forever if the device has stopped responding. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Meta-data support in NVME_IOCTL_SUBMIT_IOKeith Busch1-4/+67
This adds support for namespaces with separate meta-data formats in the submit io ioctl. The meta-data buffer has to be a contiguous, so such a buffer is allocated and the mapped user pages are copied to/from this buffer for write/read commands. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Device specific stripe size handlingKeith Busch1-4/+15
We have an nvme device that has a concept of a stripe size. IO requests that do not transfer data crossing a stripe boundary has greater performance compared to IO that does cross it. This patch sets the stripe size for the device if the device and vendor ids match one with this feature and splits IO requests that cross the stripe boundary. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Split non-mergeable bio requestsKeith Busch1-30/+122
It is possible a bio request can not be submitted as a single NVMe IO command if the bio_vec is not mergeable with the NVMe PRP alignement constraints. This condition was handled by submitting an IO for the mergeable portion then submitting a follow on IO for the remaining data after the previous IO completes. The remainder to be sent was tracked by manipulating the bio->bi_idx and bio->bi_sector. This patch splits the request as many times as necessary and submits the bios together. Since submitting the bio may cause it to be requeued on split, nvme_resubmit_bios had to be modified to remove the wait queue when the bio list is empty prior to submitting the bio since a split would have added the wait queue a second time, corrupting the wait queue head task list. There are a few other benefits from doing this: it fixes a potential issue with the previous handling of a non-mergeable bio as the requeuing method could would use an unlocked nvme_queue if the callback isn't invoked on the queue's associated cpu; it will be possible to retry a failed bio if desired at some later time since it does not manipulate the original bio; the bio integrity extensions require the bio to be in its original condition for the checks to work correctly if we implement the end-to-end data protection in the future. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Remove dead code in nvme_dev_addKeith Busch1-9/+2
There is no situation that could occur where we could error out of this function and require cleaning up allocated namespaces. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Check for NULL memory in nvme_dev_addKeith Busch1-0/+2
Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Fix error clean-up on nvme_alloc_queueKeith Busch1-1/+1
The nvme_queue's depth is not set if we fail to allocate submission queue entries, which was being used to determine how much coherent memory to free on error. Use the depth variable instead. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Free admin queue on request_irq errorKeith Busch1-4/+9
Fixes a potential memory leak if requesting the admin queue irq fails. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-05-02NVMe: Set TASK_INTERRUPTIBLE before processing queuesArjan van de Ven1-2/+1
The kthread has two tasks; handling timeouts (for which it runs once per second), and submitting queued BIOs. If a BIO happens to be queued after the thread has processed the queue but before it calls schedule_timeout(), the thread will sleep for a second before submitting it, which can cause performance problems in some rare cases (that will become more common in a subsequent patch). Signed-off-by: Arjan van de Ven <arjan@linux.intel.com> Tested-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-04-16NVMe: Add a character device for each nvme deviceKeith Busch1-10/+64
Registers a miscellaneous device for each nvme controller probed. This creates character device files as /dev/nvmeN, where N is the device instance, and supports nvme admin ioctl commands so devices without namespaces can be managed. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-04-16NVMe: Fix endian-related problems in user I/O submission pathMatthew Wilcox1-4/+4
When constructing the command, dsmgmt needs to be treated as a 32-bit value, not a 16-bit value. reftag, apptag and appmask all need to be converted from native-endian to little-endian. Again, sparse's bitwise warnings caught this problem. Thanks to Keith for pointing out the correct way to fix the reftag. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com> Acked-by: Keith Busch <keith.busch@intel.com>
2013-04-16NVMe: Fix I/O cancellation status on big-endian machinesMatthew Wilcox1-1/+1
The sparse bitwise checks pointed out that I needed to shift the status before changing its endianness, not after. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-04-16NVMe: Don't fail initialisation unnecessarilyMatthew Wilcox1-1/+7
The nvme_dev_add() function currently returns the last error code that it saw, which (if everything else succeeds) happens to be the result of an optional command, so it can legitimately fail. Looking at the error path more closely reveals that we should return success from this function, even if no device namespaces are added. So once the queues are created and the device has responded to Identify, make sure that this function succeeds. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com> Acked-by: Keith Busch <keith.busch@intel.com>
2013-04-16NVMe: Abstract out sector to block number conversionMatthew Wilcox1-2/+2
Introduce nvme_block_nr() to help convert sectors to block numbers. This fixes an integer overflow in the SCSI conversion layer, and it's slightly less typing than opencoding it. Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com> Acked-by: Keith Busch <keith.busch@intel.com>
2013-04-16NVMe: Use round_jiffies_relative() for the periodic, once-per-second timerArjan van de Ven1-1/+1
The nvme driver has a "once per second" event where the management kthread wakes up the system and then reschedules itself for 1 second later. For power efficiency reasons, I'd like this timer to happen together with other wakeups in the system. This patch makes the schedule_timeout() call in the kthread use round_jiffies_relative(), causing the wakeup to at least align with other "once per X seconds" events in the kernel. Signed-off-by: Arjan van de Ven <arjan@linux.intel.com> Tested-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-03-28NVMe: Add nvme-scsi.cVishal Verma1-17/+20
Translates SCSI commands in SG_IO ioctl to NVMe commands. Uses the scsi-nvme translation spec from nvmexpress.org as reference. Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-03-27NVMe: Add definitions for format commandVishal Verma1-0/+1
The SCSI emulation has the ability to send format commands, so we need to add the definition of the command. Also add a missing error code. Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-03-27NVMe: Move structures & definitions to header fileVishal Verma1-55/+0
nvme-scsi.c uses several data structures and definitions that were previously private to nvme-core.c. Move the definitions to nvme.h, protected by __KERNEL__. Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
2013-03-26NVMe: Rename nvme.c to nvme-core.cVishal Verma1-0/+1865
In preparation for adding nvme-scsi.c It is preferable to retain the module name 'nvme' Signed-off-by: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
generated by cgit v1.2.3 (git 2.39.0) at 2024-07-23 02:04:04 +0300