phenomenon

Running the SPDK program, the following error occurred:

starting write I/O failed, push back, reback to previous status
starting write I/O failed, push back, reback to previous status
starting write I/O failed, push back, reback to previous status
starting write I/O failed, push back, reback to previous status
starting write I/O failed, push back, reback to previous status

Causes the program to be unable to execute. What's the reason?

Analysis process

Taboos on NVME Hardware queue

Referring to the NVME protocol, you can see that hardware queue is composed of a submittion queue and
Complequeue consists of two cooperations to process IO requests:

Reference to the description in the protocol:

When host software builds a command for the controller to execute, it first checks to make sure that the appropriate Submission Queue (SQx) is not full. The Submission Queue is full when the number of entries in the queue is one less than the queue size. Once an empty slot (pFreeSlot) is available:
1. Host software builds a command at SQx[pFreeSlot] with:
a. CDW0.OPC is set to the appropriate command to be executed by the controller;
b. CDW0.FUSE is set to the appropriate value, depending on whether the command is a
fused operation;
c. CDW0.CID is set to a unique identifier for the command when combined with the
Submission Queue identifier;
d. The Namespace Identifier, CDW1.NSID, is set to the namespace the command applies to;
e. MPTR shall be filled in with the offset to the beginning of the Metadata Region, if there is a data transfer and the namespace format contains metadata as a separate buffer;
f. PRP1 and/or PRP2 (or SGL Entry 1 if SGLs are used) are set to the source/destination of data transfer, if there is a data transfer; and
g. CDW10 – CDW15 are set to any command specific information;
and
2. Host software writes the corresponding Submission Queue doorbell register (SQxTDBL)
to submit one or more commands for processing.
The write to the Submission Queue doorbell register triggers the controller to consume one or more new commands contained in the Submission Queue entry. The controller indicates the most recent SQ entry that has been consumed as part of reporting completions. Host software may use this information to determine when SQ slots may be re-used for new commands.

It can be seen that the above 3, 4, 5 and 6 steps are all completed by NVME controller hardware, while 1/27/8 is completed by host side software, among which 1/2 has strict sequence restriction and 7/8 has strict sequence restriction.

SPDK Default Kernel Binding

Based on the above processing flow, SPDK provides an API encapsulating steps 1, 2, 7, 8 above, which can be used as a function. If multiple threads call the above API to control the same set of hard ware queue s at the same time, it may lead to breaking the restriction of the above order of operations. Therefore, during initialization, SPDK threads are bound to a processor core by default.

@@ -448,7 +448,7 @@ int init(const char * dev_name) {
     spdk_env_opts_init(&opts);
     opts.name = "append_demo";
     opts.shm_id = 0;
     opts.core_mask = "0x8";
     if (spdk_env_init(&opts) < 0) {
         fprintf(stderr, "Unable to initialize Spdk env\n");
         return -1;

Notes for SPDK Threads

From the above analysis, we can see that a set of HW queue pair s can not be used by multiple threads at the same time, but different hard ware queue s are used by different threads at the same time.

Verification results

According to the above analysis, the program was modified and the error disappeared at once.