From e513de9936eaac5cdf40cdbf2747bb972deea89c Mon Sep 17 00:00:00 2001 From: Mauro Carvalho Chehab Date: Mon, 2 Mar 2020 09:16:01 +0100 Subject: scsi: docs: convert scsi_eh.txt to ReST Link: https://lore.kernel.org/r/300314197f2e6a3258200711e825aa04c9e8ceaf.1583136624.git.mchehab+huawei@kernel.org Signed-off-by: Mauro Carvalho Chehab Signed-off-by: Martin K. Petersen --- Documentation/scsi/index.rst | 1 + Documentation/scsi/scsi_eh.rst | 512 +++++++++++++++++++++++++++++++++++++++++ Documentation/scsi/scsi_eh.txt | 475 -------------------------------------- 3 files changed, 513 insertions(+), 475 deletions(-) create mode 100644 Documentation/scsi/scsi_eh.rst delete mode 100644 Documentation/scsi/scsi_eh.txt (limited to 'Documentation/scsi') diff --git a/Documentation/scsi/index.rst b/Documentation/scsi/index.rst index 635a3b3c5e90..8da7c27f73b7 100644 --- a/Documentation/scsi/index.rst +++ b/Documentation/scsi/index.rst @@ -32,5 +32,6 @@ Linux SCSI Subsystem ppa qlogicfas scsi-changer + scsi_eh scsi_transport_srp/figures diff --git a/Documentation/scsi/scsi_eh.rst b/Documentation/scsi/scsi_eh.rst new file mode 100644 index 000000000000..341f22f35056 --- /dev/null +++ b/Documentation/scsi/scsi_eh.rst @@ -0,0 +1,512 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======= +SCSI EH +======= + +This document describes SCSI midlayer error handling infrastructure. +Please refer to Documentation/scsi/scsi_mid_low_api.txt for more +information regarding SCSI midlayer. + +.. TABLE OF CONTENTS + + [1] How SCSI commands travel through the midlayer and to EH + [1-1] struct scsi_cmnd + [1-2] How do scmd's get completed? + [1-2-1] Completing a scmd w/ scsi_done + [1-2-2] Completing a scmd w/ timeout + [1-3] How EH takes over + [2] How SCSI EH works + [2-1] EH through fine-grained callbacks + [2-1-1] Overview + [2-1-2] Flow of scmds through EH + [2-1-3] Flow of control + [2-2] EH through transportt->eh_strategy_handler() + [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions + [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions + [2-2-3] Things to consider + + +1. How SCSI commands travel through the midlayer and to EH +========================================================== + +1.1 struct scsi_cmnd +-------------------- + +Each SCSI command is represented with struct scsi_cmnd (== scmd). A +scmd has two list_head's to link itself into lists. The two are +scmd->list and scmd->eh_entry. The former is used for free list or +per-device allocated scmd list and not of much interest to this EH +discussion. The latter is used for completion and EH lists and unless +otherwise stated scmds are always linked using scmd->eh_entry in this +discussion. + + +1.2 How do scmd's get completed? +-------------------------------- + +Once LLDD gets hold of a scmd, either the LLDD will complete the +command by calling scsi_done callback passed from midlayer when +invoking hostt->queuecommand() or the block layer will time it out. + + +1.2.1 Completing a scmd w/ scsi_done +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For all non-EH commands, scsi_done() is the completion callback. It +just calls blk_complete_request() to delete the block layer timer and +raise SCSI_SOFTIRQ + +SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to +determine what to do with the command. scsi_decide_disposition() +looks at the scmd->result value and sense data to determine what to do +with the command. + + - SUCCESS + + scsi_finish_command() is invoked for the command. The + function does some maintenance chores and then calls + scsi_io_completion() to finish the I/O. + scsi_io_completion() then notifies the block layer on + the completed request by calling blk_end_request and + friends or figures out what to do with the remainder + of the data in case of an error. + + - NEEDS_RETRY + + - ADD_TO_MLQUEUE + + scmd is requeued to blk queue. + + - otherwise + + scsi_eh_scmd_add(scmd) is invoked for the command. See + [1-3] for details of this function. + + +1.2.2 Completing a scmd w/ timeout +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The timeout handler is scsi_times_out(). When a timeout occurs, this +function + + 1. invokes optional hostt->eh_timed_out() callback. Return value can + be one of + + - BLK_EH_RESET_TIMER + This indicates that more time is required to finish the + command. Timer is restarted. This action is counted as a + retry and only allowed scmd->allowed + 1(!) times. Once the + limit is reached, action for BLK_EH_DONE is taken instead. + + - BLK_EH_DONE + eh_timed_out() callback did not handle the command. + Step #2 is taken. + + 2. scsi_abort_command() is invoked to schedule an asynchrous abort. + Asynchronous abort are not invoked for commands which the + SCSI_EH_ABORT_SCHEDULED flag is set (this indicates that the command + already had been aborted once, and this is a retry which failed), + or when the EH deadline is expired. In these case Step #3 is taken. + + 3. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the + command. See [1-4] for more information. + +1.3 Asynchronous command aborts +------------------------------- + + After a timeout occurs a command abort is scheduled from + scsi_abort_command(). If the abort is successful the command + will either be retried (if the number of retries is not exhausted) + or terminated with DID_TIME_OUT. + + Otherwise scsi_eh_scmd_add() is invoked for the command. + See [1-4] for more information. + +1.4 How EH takes over +--------------------- + +scmds enter EH via scsi_eh_scmd_add(), which does the following. + + 1. Links scmd->eh_entry to shost->eh_cmd_q + + 2. Sets SHOST_RECOVERY bit in shost->shost_state + + 3. Increments shost->host_failed + + 4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed + +As can be seen above, once any scmd is added to shost->eh_cmd_q, +SHOST_RECOVERY shost_state bit is turned on. This prevents any new +scmd to be issued from blk queue to the host; eventually, all scmds on +the host either complete normally, fail and get added to eh_cmd_q, or +time out and get added to shost->eh_cmd_q. + +If all scmds either complete or fail, the number of in-flight scmds +becomes equal to the number of failed scmds - i.e. shost->host_busy == +shost->host_failed. This wakes up SCSI EH thread. So, once woken up, +SCSI EH thread can expect that all in-flight commands have failed and +are linked on shost->eh_cmd_q. + +Note that this does not mean lower layers are quiescent. If a LLDD +completed a scmd with error status, the LLDD and lower layers are +assumed to forget about the scmd at that point. However, if a scmd +has timed out, unless hostt->eh_timed_out() made lower layers forget +about the scmd, which currently no LLDD does, the command is still +active as long as lower layers are concerned and completion could +occur at any time. Of course, all such completions are ignored as the +timer has already expired. + +We'll talk about how SCSI EH takes actions to abort - make LLDD +forget about - timed out scmds later. + + +2. How SCSI EH works +==================== + +LLDD's can implement SCSI EH actions in one of the following two +ways. + + - Fine-grained EH callbacks + LLDD can implement fine-grained EH callbacks and let SCSI + midlayer drive error handling and call appropriate callbacks. + This will be discussed further in [2-1]. + + - eh_strategy_handler() callback + This is one big callback which should perform whole error + handling. As such, it should do all chores the SCSI midlayer + performs during recovery. This will be discussed in [2-2]. + +Once recovery is complete, SCSI EH resumes normal operation by +calling scsi_restart_operations(), which + + 1. Checks if door locking is needed and locks door. + + 2. Clears SHOST_RECOVERY shost_state bit + + 3. Wakes up waiters on shost->host_wait. This occurs if someone + calls scsi_block_when_processing_errors() on the host. + (*QUESTION* why is it needed? All operations will be blocked + anyway after it reaches blk queue.) + + 4. Kicks queues in all devices on the host in the asses + + +2.1 EH through fine-grained callbacks +------------------------------------- + +2.1.1 Overview +^^^^^^^^^^^^^^ + +If eh_strategy_handler() is not present, SCSI midlayer takes charge +of driving error handling. EH's goals are two - make LLDD, host and +device forget about timed out scmds and make them ready for new +commands. A scmd is said to be recovered if the scmd is forgotten by +lower layers and lower layers are ready to process or fail the scmd +again. + +To achieve these goals, EH performs recovery actions with increasing +severity. Some actions are performed by issuing SCSI commands and +others are performed by invoking one of the following fine-grained +hostt EH callbacks. Callbacks may be omitted and omitted ones are +considered to fail always. + +:: + + int (* eh_abort_handler)(struct scsi_cmnd *); + int (* eh_device_reset_handler)(struct scsi_cmnd *); + int (* eh_bus_reset_handler)(struct scsi_cmnd *); + int (* eh_host_reset_handler)(struct scsi_cmnd *); + +Higher-severity actions are taken only when lower-severity actions +cannot recover some of failed scmds. Also, note that failure of the +highest-severity action means EH failure and results in offlining of +all unrecovered devices. + +During recovery, the following rules are followed + + - Recovery actions are performed on failed scmds on the to do list, + eh_work_q. If a recovery action succeeds for a scmd, recovered + scmds are removed from eh_work_q. + + Note that single recovery action on a scmd can recover multiple + scmds. e.g. resetting a device recovers all failed scmds on the + device. + + - Higher severity actions are taken iff eh_work_q is not empty after + lower severity actions are complete. + + - EH reuses failed scmds to issue commands for recovery. For + timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd + before reusing it for EH commands. + +When a scmd is recovered, the scmd is moved from eh_work_q to EH +local eh_done_q using scsi_eh_finish_cmd(). After all scmds are +recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to +either retry or error-finish (notify upper layer of failure) recovered +scmds. + +scmds are retried iff its sdev is still online (not offlined during +EH), REQ_FAILFAST is not set and ++scmd->retries is less than +scmd->allowed. + + +2.1.2 Flow of scmds through EH +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + 1. Error completion / time out + + :ACTION: scsi_eh_scmd_add() is invoked for scmd + + - add scmd to shost->eh_cmd_q + - set SHOST_RECOVERY + - shost->host_failed++ + + :LOCKING: shost->host_lock + + 2. EH starts + + :ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q + is cleared. + + :LOCKING: shost->host_lock (not strictly necessary, just for + consistency) + + 3. scmd recovered + + :ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd + + - scsi_setup_cmd_retry() + - move from local eh_work_q to local eh_done_q + + :LOCKING: none + + :CONCURRENCY: at most one thread per separate eh_work_q to + keep queue manipulation lockless + + 4. EH completes + + :ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper + layer of failure. May be called concurrently but must have + a no more than one thread per separate eh_work_q to + manipulate the queue locklessly + + - scmd is removed from eh_done_q and scmd->eh_entry is cleared + - if retry is necessary, scmd is requeued using + scsi_queue_insert() + - otherwise, scsi_finish_command() is invoked for scmd + - zero shost->host_failed + + :LOCKING: queue or finish function performs appropriate locking + + +2.1.3 Flow of control +^^^^^^^^^^^^^^^^^^^^^^ + + EH through fine-grained callbacks start from scsi_unjam_host(). + +``scsi_unjam_host`` + + 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local + eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is + cleared by this action. + + 2. Invoke scsi_eh_get_sense. + + ``scsi_eh_get_sense`` + + This action is taken for each error-completed + (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most + SCSI transports/LLDDs automatically acquire sense data on + command failures (autosense). Autosense is recommended for + performance reasons and as sense information could get out of + sync between occurrence of CHECK CONDITION and this action. + + Note that if autosense is not supported, scmd->sense_buffer + contains invalid sense data when error-completing the scmd + with scsi_done(). scsi_decide_disposition() always returns + FAILED in such cases thus invoking SCSI EH. When the scmd + reaches here, sense data is acquired and + scsi_decide_disposition() is called again. + + 1. Invoke scsi_request_sense() which issues REQUEST_SENSE + command. If fails, no action. Note that taking no action + causes higher-severity recovery to be taken for the scmd. + + 2. Invoke scsi_decide_disposition() on the scmd + + - SUCCESS + scmd->retries is set to scmd->allowed preventing + scsi_eh_flush_done_q() from retrying the scmd and + scsi_eh_finish_cmd() is invoked. + + - NEEDS_RETRY + scsi_eh_finish_cmd() invoked + + - otherwise + No action. + + 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). + + ``scsi_eh_abort_cmds`` + + This action is taken for each timed out command when + no_async_abort is enabled in the host template. + hostt->eh_abort_handler() is invoked for each scmd. The + handler returns SUCCESS if it has succeeded to make LLDD and + all related hardware forget about the scmd. + + If a timedout scmd is successfully aborted and the sdev is + either offline or ready, scsi_eh_finish_cmd() is invoked for + the scmd. Otherwise, the scmd is left in eh_work_q for + higher-severity actions. + + Note that both offline and ready status mean that the sdev is + ready to process new scmds, where processing also implies + immediate failing; thus, if a sdev is in one of the two + states, no further recovery action is needed. + + Device readiness is tested using scsi_eh_tur() which issues + TEST_UNIT_READY command. Note that the scmd must have been + aborted successfully before reusing it for TEST_UNIT_READY. + + 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() + + ``scsi_eh_ready_devs`` + + This function takes four increasingly more severe measures to + make failed sdevs ready for new commands. + + 1. Invoke scsi_eh_stu() + + ``scsi_eh_stu`` + + For each sdev which has failed scmds with valid sense data + of which scsi_check_sense()'s verdict is FAILED, + START_STOP_UNIT command is issued w/ start=1. Note that + as we explicitly choose error-completed scmds, it is known + that lower layers have forgotten about the scmd and we can + reuse it for STU. + + If STU succeeds and the sdev is either offline or ready, + all failed scmds on the sdev are EH-finished with + scsi_eh_finish_cmd(). + + *NOTE* If hostt->eh_abort_handler() isn't implemented or + failed, we may still have timed out scmds at this point + and STU doesn't make lower layers forget about those + scmds. Yet, this function EH-finish all scmds on the sdev + if STU succeeds leaving lower layers in an inconsistent + state. It seems that STU action should be taken only when + a sdev has no timed out scmd. + + 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). + + ``scsi_eh_bus_device_reset`` + + This action is very similar to scsi_eh_stu() except that, + instead of issuing STU, hostt->eh_device_reset_handler() + is used. Also, as we're not issuing SCSI commands and + resetting clears all scmds on the sdev, there is no need + to choose error-completed scmds. + + 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() + + ``scsi_eh_bus_reset`` + + hostt->eh_bus_reset_handler() is invoked for each channel + with failed scmds. If bus reset succeeds, all failed + scmds on all ready or offline sdevs on the channel are + EH-finished. + + 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() + + ``scsi_eh_host_reset`` + + This is the last resort. hostt->eh_host_reset_handler() + is invoked. If host reset succeeds, all failed scmds on + all ready or offline sdevs on the host are EH-finished. + + 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() + + ``scsi_eh_offline_sdevs`` + + Take all sdevs which still have unrecovered scmds offline + and EH-finish the scmds. + + 5. Invoke scsi_eh_flush_done_q(). + + ``scsi_eh_flush_done_q`` + + At this point all scmds are recovered (or given up) and + put on eh_done_q by scsi_eh_finish_cmd(). This function + flushes eh_done_q by either retrying or notifying upper + layer of failure of the scmds. + + +2.2 EH through transportt->eh_strategy_handler() +------------------------------------------------ + +transportt->eh_strategy_handler() is invoked in the place of +scsi_unjam_host() and it is responsible for whole recovery process. +On completion, the handler should have made lower layers forget about +all failed scmds and either ready for new commands or offline. Also, +it should perform SCSI EH maintenance chores to maintain integrity of +SCSI midlayer. IOW, of the steps described in [2-1-2], all steps +except for #1 must be implemented by eh_strategy_handler(). + + +2.2.1 Pre transportt->eh_strategy_handler() SCSI midlayer conditions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + The following conditions are true on entry to the handler. + + - Each failed scmd's eh_flags field is set appropriately. + + - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. + + - SHOST_RECOVERY is set. + + - shost->host_failed == shost->host_busy + + +2.2.2 Post transportt->eh_strategy_handler() SCSI midlayer conditions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + The following conditions must be true on exit from the handler. + + - shost->host_failed is zero. + + - Each scmd is in such a state that scsi_setup_cmd_retry() on the + scmd doesn't make any difference. + + - shost->eh_cmd_q is cleared. + + - Each scmd->eh_entry is cleared. + + - Either scsi_queue_insert() or scsi_finish_command() is called on + each scmd. Note that the handler is free to use scmd->retries and + ->allowed to limit the number of retries. + + +2.2.3 Things to consider +^^^^^^^^^^^^^^^^^^^^^^^^ + + - Know that timed out scmds are still active on lower layers. Make + lower layers forget about them before doing anything else with + those scmds. + + - For consistency, when accessing/modifying shost data structure, + grab shost->host_lock. + + - On completion, each failed sdev must have forgotten about all + active scmds. + + - On completion, each failed sdev must be ready for new commands or + offline. + + +Tejun Heo +htejun@gmail.com + +11th September 2005 diff --git a/Documentation/scsi/scsi_eh.txt b/Documentation/scsi/scsi_eh.txt deleted file mode 100644 index 1b7436932a2b..000000000000 --- a/Documentation/scsi/scsi_eh.txt +++ /dev/null @@ -1,475 +0,0 @@ - -SCSI EH -====================================== - - This document describes SCSI midlayer error handling infrastructure. -Please refer to Documentation/scsi/scsi_mid_low_api.txt for more -information regarding SCSI midlayer. - -TABLE OF CONTENTS - -[1] How SCSI commands travel through the midlayer and to EH - [1-1] struct scsi_cmnd - [1-2] How do scmd's get completed? - [1-2-1] Completing a scmd w/ scsi_done - [1-2-2] Completing a scmd w/ timeout - [1-3] How EH takes over -[2] How SCSI EH works - [2-1] EH through fine-grained callbacks - [2-1-1] Overview - [2-1-2] Flow of scmds through EH - [2-1-3] Flow of control - [2-2] EH through transportt->eh_strategy_handler() - [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions - [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions - [2-2-3] Things to consider - - -[1] How SCSI commands travel through the midlayer and to EH - -[1-1] struct scsi_cmnd - - Each SCSI command is represented with struct scsi_cmnd (== scmd). A -scmd has two list_head's to link itself into lists. The two are -scmd->list and scmd->eh_entry. The former is used for free list or -per-device allocated scmd list and not of much interest to this EH -discussion. The latter is used for completion and EH lists and unless -otherwise stated scmds are always linked using scmd->eh_entry in this -discussion. - - -[1-2] How do scmd's get completed? - - Once LLDD gets hold of a scmd, either the LLDD will complete the -command by calling scsi_done callback passed from midlayer when -invoking hostt->queuecommand() or the block layer will time it out. - - -[1-2-1] Completing a scmd w/ scsi_done - - For all non-EH commands, scsi_done() is the completion callback. It -just calls blk_complete_request() to delete the block layer timer and -raise SCSI_SOFTIRQ - - SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to -determine what to do with the command. scsi_decide_disposition() -looks at the scmd->result value and sense data to determine what to do -with the command. - - - SUCCESS - scsi_finish_command() is invoked for the command. The - function does some maintenance chores and then calls - scsi_io_completion() to finish the I/O. - scsi_io_completion() then notifies the block layer on - the completed request by calling blk_end_request and - friends or figures out what to do with the remainder - of the data in case of an error. - - - NEEDS_RETRY - - ADD_TO_MLQUEUE - scmd is requeued to blk queue. - - - otherwise - scsi_eh_scmd_add(scmd) is invoked for the command. See - [1-3] for details of this function. - - -[1-2-2] Completing a scmd w/ timeout - - The timeout handler is scsi_times_out(). When a timeout occurs, this -function - - 1. invokes optional hostt->eh_timed_out() callback. Return value can - be one of - - - BLK_EH_RESET_TIMER - This indicates that more time is required to finish the - command. Timer is restarted. This action is counted as a - retry and only allowed scmd->allowed + 1(!) times. Once the - limit is reached, action for BLK_EH_DONE is taken instead. - - - BLK_EH_DONE - eh_timed_out() callback did not handle the command. - Step #2 is taken. - - 2. scsi_abort_command() is invoked to schedule an asynchrous abort. - Asynchronous abort are not invoked for commands which the - SCSI_EH_ABORT_SCHEDULED flag is set (this indicates that the command - already had been aborted once, and this is a retry which failed), - or when the EH deadline is expired. In these case Step #3 is taken. - - 3. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the - command. See [1-4] for more information. - -[1-3] Asynchronous command aborts - - After a timeout occurs a command abort is scheduled from - scsi_abort_command(). If the abort is successful the command - will either be retried (if the number of retries is not exhausted) - or terminated with DID_TIME_OUT. - Otherwise scsi_eh_scmd_add() is invoked for the command. - See [1-4] for more information. - -[1-4] How EH takes over - - scmds enter EH via scsi_eh_scmd_add(), which does the following. - - 1. Links scmd->eh_entry to shost->eh_cmd_q - - 2. Sets SHOST_RECOVERY bit in shost->shost_state - - 3. Increments shost->host_failed - - 4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed - - As can be seen above, once any scmd is added to shost->eh_cmd_q, -SHOST_RECOVERY shost_state bit is turned on. This prevents any new -scmd to be issued from blk queue to the host; eventually, all scmds on -the host either complete normally, fail and get added to eh_cmd_q, or -time out and get added to shost->eh_cmd_q. - - If all scmds either complete or fail, the number of in-flight scmds -becomes equal to the number of failed scmds - i.e. shost->host_busy == -shost->host_failed. This wakes up SCSI EH thread. So, once woken up, -SCSI EH thread can expect that all in-flight commands have failed and -are linked on shost->eh_cmd_q. - - Note that this does not mean lower layers are quiescent. If a LLDD -completed a scmd with error status, the LLDD and lower layers are -assumed to forget about the scmd at that point. However, if a scmd -has timed out, unless hostt->eh_timed_out() made lower layers forget -about the scmd, which currently no LLDD does, the command is still -active as long as lower layers are concerned and completion could -occur at any time. Of course, all such completions are ignored as the -timer has already expired. - - We'll talk about how SCSI EH takes actions to abort - make LLDD -forget about - timed out scmds later. - - -[2] How SCSI EH works - - LLDD's can implement SCSI EH actions in one of the following two -ways. - - - Fine-grained EH callbacks - LLDD can implement fine-grained EH callbacks and let SCSI - midlayer drive error handling and call appropriate callbacks. - This will be discussed further in [2-1]. - - - eh_strategy_handler() callback - This is one big callback which should perform whole error - handling. As such, it should do all chores the SCSI midlayer - performs during recovery. This will be discussed in [2-2]. - - Once recovery is complete, SCSI EH resumes normal operation by -calling scsi_restart_operations(), which - - 1. Checks if door locking is needed and locks door. - - 2. Clears SHOST_RECOVERY shost_state bit - - 3. Wakes up waiters on shost->host_wait. This occurs if someone - calls scsi_block_when_processing_errors() on the host. - (*QUESTION* why is it needed? All operations will be blocked - anyway after it reaches blk queue.) - - 4. Kicks queues in all devices on the host in the asses - - -[2-1] EH through fine-grained callbacks - -[2-1-1] Overview - - If eh_strategy_handler() is not present, SCSI midlayer takes charge -of driving error handling. EH's goals are two - make LLDD, host and -device forget about timed out scmds and make them ready for new -commands. A scmd is said to be recovered if the scmd is forgotten by -lower layers and lower layers are ready to process or fail the scmd -again. - - To achieve these goals, EH performs recovery actions with increasing -severity. Some actions are performed by issuing SCSI commands and -others are performed by invoking one of the following fine-grained -hostt EH callbacks. Callbacks may be omitted and omitted ones are -considered to fail always. - -int (* eh_abort_handler)(struct scsi_cmnd *); -int (* eh_device_reset_handler)(struct scsi_cmnd *); -int (* eh_bus_reset_handler)(struct scsi_cmnd *); -int (* eh_host_reset_handler)(struct scsi_cmnd *); - - Higher-severity actions are taken only when lower-severity actions -cannot recover some of failed scmds. Also, note that failure of the -highest-severity action means EH failure and results in offlining of -all unrecovered devices. - - During recovery, the following rules are followed - - - Recovery actions are performed on failed scmds on the to do list, - eh_work_q. If a recovery action succeeds for a scmd, recovered - scmds are removed from eh_work_q. - - Note that single recovery action on a scmd can recover multiple - scmds. e.g. resetting a device recovers all failed scmds on the - device. - - - Higher severity actions are taken iff eh_work_q is not empty after - lower severity actions are complete. - - - EH reuses failed scmds to issue commands for recovery. For - timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd - before reusing it for EH commands. - - When a scmd is recovered, the scmd is moved from eh_work_q to EH -local eh_done_q using scsi_eh_finish_cmd(). After all scmds are -recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to -either retry or error-finish (notify upper layer of failure) recovered -scmds. - - scmds are retried iff its sdev is still online (not offlined during -EH), REQ_FAILFAST is not set and ++scmd->retries is less than -scmd->allowed. - - -[2-1-2] Flow of scmds through EH - - 1. Error completion / time out - ACTION: scsi_eh_scmd_add() is invoked for scmd - - add scmd to shost->eh_cmd_q - - set SHOST_RECOVERY - - shost->host_failed++ - LOCKING: shost->host_lock - - 2. EH starts - ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q - is cleared. - LOCKING: shost->host_lock (not strictly necessary, just for - consistency) - - 3. scmd recovered - ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd - - scsi_setup_cmd_retry() - - move from local eh_work_q to local eh_done_q - LOCKING: none - CONCURRENCY: at most one thread per separate eh_work_q to - keep queue manipulation lockless - - 4. EH completes - ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper - layer of failure. May be called concurrently but must have - a no more than one thread per separate eh_work_q to - manipulate the queue locklessly - - scmd is removed from eh_done_q and scmd->eh_entry is cleared - - if retry is necessary, scmd is requeued using - scsi_queue_insert() - - otherwise, scsi_finish_command() is invoked for scmd - - zero shost->host_failed - LOCKING: queue or finish function performs appropriate locking - - -[2-1-3] Flow of control - - EH through fine-grained callbacks start from scsi_unjam_host(). - -<> - - 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local - eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is - cleared by this action. - - 2. Invoke scsi_eh_get_sense. - - <> - - This action is taken for each error-completed - (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most - SCSI transports/LLDDs automatically acquire sense data on - command failures (autosense). Autosense is recommended for - performance reasons and as sense information could get out of - sync between occurrence of CHECK CONDITION and this action. - - Note that if autosense is not supported, scmd->sense_buffer - contains invalid sense data when error-completing the scmd - with scsi_done(). scsi_decide_disposition() always returns - FAILED in such cases thus invoking SCSI EH. When the scmd - reaches here, sense data is acquired and - scsi_decide_disposition() is called again. - - 1. Invoke scsi_request_sense() which issues REQUEST_SENSE - command. If fails, no action. Note that taking no action - causes higher-severity recovery to be taken for the scmd. - - 2. Invoke scsi_decide_disposition() on the scmd - - - SUCCESS - scmd->retries is set to scmd->allowed preventing - scsi_eh_flush_done_q() from retrying the scmd and - scsi_eh_finish_cmd() is invoked. - - - NEEDS_RETRY - scsi_eh_finish_cmd() invoked - - - otherwise - No action. - - 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). - - <> - - This action is taken for each timed out command when - no_async_abort is enabled in the host template. - hostt->eh_abort_handler() is invoked for each scmd. The - handler returns SUCCESS if it has succeeded to make LLDD and - all related hardware forget about the scmd. - - If a timedout scmd is successfully aborted and the sdev is - either offline or ready, scsi_eh_finish_cmd() is invoked for - the scmd. Otherwise, the scmd is left in eh_work_q for - higher-severity actions. - - Note that both offline and ready status mean that the sdev is - ready to process new scmds, where processing also implies - immediate failing; thus, if a sdev is in one of the two - states, no further recovery action is needed. - - Device readiness is tested using scsi_eh_tur() which issues - TEST_UNIT_READY command. Note that the scmd must have been - aborted successfully before reusing it for TEST_UNIT_READY. - - 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() - - <> - - This function takes four increasingly more severe measures to - make failed sdevs ready for new commands. - - 1. Invoke scsi_eh_stu() - - <> - - For each sdev which has failed scmds with valid sense data - of which scsi_check_sense()'s verdict is FAILED, - START_STOP_UNIT command is issued w/ start=1. Note that - as we explicitly choose error-completed scmds, it is known - that lower layers have forgotten about the scmd and we can - reuse it for STU. - - If STU succeeds and the sdev is either offline or ready, - all failed scmds on the sdev are EH-finished with - scsi_eh_finish_cmd(). - - *NOTE* If hostt->eh_abort_handler() isn't implemented or - failed, we may still have timed out scmds at this point - and STU doesn't make lower layers forget about those - scmds. Yet, this function EH-finish all scmds on the sdev - if STU succeeds leaving lower layers in an inconsistent - state. It seems that STU action should be taken only when - a sdev has no timed out scmd. - - 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). - - <> - - This action is very similar to scsi_eh_stu() except that, - instead of issuing STU, hostt->eh_device_reset_handler() - is used. Also, as we're not issuing SCSI commands and - resetting clears all scmds on the sdev, there is no need - to choose error-completed scmds. - - 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() - - <> - - hostt->eh_bus_reset_handler() is invoked for each channel - with failed scmds. If bus reset succeeds, all failed - scmds on all ready or offline sdevs on the channel are - EH-finished. - - 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() - - <> - - This is the last resort. hostt->eh_host_reset_handler() - is invoked. If host reset succeeds, all failed scmds on - all ready or offline sdevs on the host are EH-finished. - - 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() - - <> - - Take all sdevs which still have unrecovered scmds offline - and EH-finish the scmds. - - 5. Invoke scsi_eh_flush_done_q(). - - <> - - At this point all scmds are recovered (or given up) and - put on eh_done_q by scsi_eh_finish_cmd(). This function - flushes eh_done_q by either retrying or notifying upper - layer of failure of the scmds. - - -[2-2] EH through transportt->eh_strategy_handler() - - transportt->eh_strategy_handler() is invoked in the place of -scsi_unjam_host() and it is responsible for whole recovery process. -On completion, the handler should have made lower layers forget about -all failed scmds and either ready for new commands or offline. Also, -it should perform SCSI EH maintenance chores to maintain integrity of -SCSI midlayer. IOW, of the steps described in [2-1-2], all steps -except for #1 must be implemented by eh_strategy_handler(). - - -[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions - - The following conditions are true on entry to the handler. - - - Each failed scmd's eh_flags field is set appropriately. - - - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. - - - SHOST_RECOVERY is set. - - - shost->host_failed == shost->host_busy - - -[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions - - The following conditions must be true on exit from the handler. - - - shost->host_failed is zero. - - - Each scmd is in such a state that scsi_setup_cmd_retry() on the - scmd doesn't make any difference. - - - shost->eh_cmd_q is cleared. - - - Each scmd->eh_entry is cleared. - - - Either scsi_queue_insert() or scsi_finish_command() is called on - each scmd. Note that the handler is free to use scmd->retries and - ->allowed to limit the number of retries. - - -[2-2-3] Things to consider - - - Know that timed out scmds are still active on lower layers. Make - lower layers forget about them before doing anything else with - those scmds. - - - For consistency, when accessing/modifying shost data structure, - grab shost->host_lock. - - - On completion, each failed sdev must have forgotten about all - active scmds. - - - On completion, each failed sdev must be ready for new commands or - offline. - - --- -Tejun Heo -htejun@gmail.com -11th September 2005 -- cgit v1.2.3