Linux storage IO stack -- an overview of SCSI subsystem

Summary

Hierarchical architecture of Linux SCSI subsystem:

Low level: it represents the actual driver of the physical interface with SCSI, such as the driver developed by each manufacturer for its specific host bus adapter (HbAS). The main function of low level driver is to discover the SCSI devices connected to the host adapter, build the data structure required by the SCSI subsystem in memory, and provide the message passing interface to interpret the receiving and sending of SCSI commands Operation of machine adapter.
High level: drivers representing various types of scsi devices, such as scsi disk drivers, scsi tape drivers, and high-level drivers claim the scsi devices found by the low-level drivers. Assign names to these devices, convert IO to scsi commands, and submit them to the low-level drivers for processing.
Middle layer: the public service function containing the scsi stack. The high level and the low level complete their functions by calling the functions in the middle level, while the middle level needs to call the callback functions registered in the high level and the low level to do some personalized processing during the execution process.

Linux SCSI model

The Linux SCSI model is an abstraction of the kernel. The host adapter connects the host IO bus (such as PCI bus) and the storage IO bus (such as SCSI bus). A computer can have multiple host adapters, and the host adapter can control one or more SCSI buses. A bus can have multiple target nodes connected to it, and a target node can have multiple logical units.

In the Linux SCSI subsystem, the target node in the kernel corresponds to the SCSI disk. There can be multiple logical units in the SCSI disk, which are controlled by the disk controller. These logical units are the storage devices that are the real IO destination. The kernel abstracts the logical units with devices. The Host in the kernel corresponds to the Host adapter (physical HbAS / raid cards, virtual iSCSI tar) Get)

The kernel uses a quad to uniquely identify the logical unit of a scsi. In sysfs, the view of sda disk < 2:0:0:0 > is as follows:

root@ubuntu16:/home/comet/Costor/bin# ls /sys/bus/scsi/devices/2\:0\:0\:0/block/sda/
alignment_offset  device             events_poll_msecs  integrity  removable  sda5    subsystem
bdi               discard_alignment  ext_range          power      ro         size    trace
capability        events             holders            queue      sda1       slaves  uevent
dev               events_async       inflight           range      sda2       stat
root@ubuntu16:/home/comet/Costor/bin# cat /sys/bus/scsi/devices/2\:0\:0\:0/block/sda/dev
8:0
root@ubuntu16:/home/comet/Costor/bin# ll /dev/sda
brw-rw---- 1 root disk 8, 0 Sep 19 11:36 /dev/sda

host: Unique number of the host adapter.
channel: In host adapter scsi Channel number, maintained by the host adapter firmware.
id: Unique identifier of the target node.
lun: Logical unit number in the target node.

SCSI command

SCSI The order is in Command Descriptor Block (CDB) Defined in. CDB Contains the operation code that defines the specific operation to be performed, as well as a large number of operation specific parameters.

command	purpose
Test unit ready	Query whether the device is ready for transmission
Inquiry	Request device basic information
Request sense	Error message for command before request
Read capacity	Request storage capacity information
Read	Read data from device
Write	Write data to device
Mode sense	Request mode page (device parameters)
Mode select	Configure device parameters on the mode page

With about 60 commands available, SCSI It can be applied to many devices (including random access devices, such as disk and sequential storage devices such as tape). SCSI Special commands are also provided to access the enclosure services (such as the current sensing and temperature inside the enclosure).

Core data structure

Host adapter template scsi_host_template

The host adapter template is the common content of the same model of host adapter, including request queue depth, SCSI Command processing callback function, error processing recovery function. When assigning the host adapter structure, you need to use the host adapter template to assign values. In writing SCSI The first step for low-level drivers is to define templates scsi_host_template，Before a template can generate a host adapter.

struct scsi_host_template {
    struct module *module;  //Point to the SCSI host implemented with this template, the low-level driver module.
    const char *name;       //Host adapter name

    int (* detect)(struct scsi_host_template *);
    int (* release)(struct Scsi_Host *);

    const char *(* info)(struct Scsi_Host *); //Return relevant information of HbAS, optional implementation

    int (* ioctl)(struct scsi_device *dev, int cmd, void __user *arg); //Optional implementation of ioctl function in user space


#ifdef CONFIG_COMPAT
    //Through this function, the user state ioctl function of 32-bit system is supported
    int (* compat_ioctl)(struct scsi_device *dev, int cmd, void __user *arg);
#endif

    //Put the scsi command into the queue driven by the lower layer, which is called by the middle layer and must be implemented
    int (* queuecommand)(struct Scsi_Host *, struct scsi_cmnd *);

    //The following five functions are error handling callback functions, which are called by the middle layer according to the severity
    int (* eh_abort_handler)(struct scsi_cmnd *);        //Abort
    int (* eh_device_reset_handler)(struct scsi_cmnd *); //Device Reset
    int (* eh_target_reset_handler)(struct scsi_cmnd *); //Target Reset
    int (* eh_bus_reset_handler)(struct scsi_cmnd *);    //Bus Reset
    int (* eh_host_reset_handler)(struct scsi_cmnd *);   //Host Reset

    //Called when a new disk is scanned, the middle layer callback function can allocate and initialize the structure required by the lower layer driver
    int (* slave_alloc)(struct scsi_device *)

//Perform relevant configuration operations after the device receives the INQUIRY command
    int (* slave_configure)(struct scsi_device *);

    //Before the scsi device is destroyed, the middle layer callback is used to release the private data allocated by slave_alloc.
    void (* slave_destroy)(struct scsi_device *);

    //When a new target is found, the middle layer calls, and the user allocates target private data
    int (* target_alloc)(struct scsi_target *);

    //Before the target is destroyed, the middle layer calls and the lower layer drives the implementation to release the data allocated by target ﹣ alloc
    void (* target_destroy)(struct scsi_target *);

    //When you need to customize the scan target logic, the middle layer circulates to check the return value until the function returns 1, indicating that the scan is complete
    int (* scan_finished)(struct Scsi_Host *, unsigned long);

    //Callback before scan start when custom scan target logic is required
    void (* scan_start)(struct Scsi_Host *);

    //Change the queue depth of the host adapter to return the set queue depth
    int (* change_queue_depth)(struct scsi_device *, int);

    //Returns the BIOS parameters of the disk, such as size, device, list (heads, sections, cylinders)
    int (* bios_param)(struct scsi_device *, struct block_device *,
            sector_t, int []);

    void (*unlock_native_capacity)(struct scsi_device *);

    //Callback of read and write operation in procfs
    int (*show_info)(struct seq_file *, struct Scsi_Host *);
    int (*write_info)(struct Scsi_Host *, char *, int);

    //Middle layer found scsi command timeout callback
    enum blk_eh_timer_return (*eh_timed_out)(struct scsi_cmnd *);

    //When reset ting host adapter through sysfs property, callback
    int (*host_reset)(struct Scsi_Host *shost, int reset_type);
#define SCSI_ADAPTER_RESET  1
#define SCSI_FIRMWARE_RESET 2

    const char *proc_name; //The name of the proc file system

    struct proc_dir_entry *proc_dir;

    int can_queue; //The number of commands the host adapter can accept at the same time

    int this_id;

    /*
     * This determines the degree to which the host adapter is capable
     * of scatter-gather.
     */  //Parameters of the hash table
    unsigned short sg_tablesize;
    unsigned short sg_prot_tablesize;

    /*
     * Set this if the host adapter has limitations beside segment count.
     */ //The maximum number of sectors that a single scsi command can access
    unsigned int max_sectors;

    /*
     * DMA scatter gather segment boundary limit. A segment crossing this
     * boundary will be split in two.
     */
    unsigned long dma_boundary; //DMA accumulation and dispersion section boundary value, exceeding which will be cut into two

#define SCSI_DEFAULT_MAX_SECTORS    1024

    short cmd_per_lun;

    /*
     * present contains counter indicating how many boards of this
     * type were found when we did the scan.
     */
    unsigned char present;

    /* If use block layer to manage tags, this is tag allocation policy */
    int tag_alloc_policy;

    /*
     * Track QUEUE_FULL events and reduce queue depth on demand.
     */
    unsigned track_queue_depth:1;

    /*
     * This specifies the mode that a LLD supports.
     */
    unsigned supported_mode:2; //Patterns supported by low-level drivers (initiator or target)

    /*
     * True if this host adapter uses unchecked DMA onto an ISA bus.
     */
    unsigned unchecked_isa_dma:1;

    unsigned use_clustering:1;

    /*
     * True for emulated SCSI host adapters (e.g. ATAPI).
     */
    unsigned emulated:1;

    /*
     * True if the low-level driver performs its own reset-settle delays.
     */
    unsigned skip_settle_delay:1;

    /* True if the controller does not support WRITE SAME */
    unsigned no_write_same:1;

    /*
     * True if asynchronous aborts are not supported
     */
    unsigned no_async_abort:1;

    /*
     * Countdown for host blocking with no commands outstanding.
     */
    unsigned int max_host_blocked; //The low threshold value of the sending queue of the host adapter allows multiple commands to be distributed at the same time

#define SCSI_DEFAULT_HOST_BLOCKED   7

    /*
     * Pointer to the sysfs class properties for this host, NULL terminated.
     */
    struct device_attribute **shost_attrs; //Host adapter class properties

    /*
     * Pointer to the SCSI device properties for this host, NULL terminated.
     */
    struct device_attribute **sdev_attrs;  //Host adapter device properties

    struct list_head legacy_hosts;

    u64 vendor_id;

    /*
     * Additional per-command data allocated for the driver.
     */  //The scsi command buffer pool and the scsi commands are pre allocated and stored in the CMD pool
    unsigned int cmd_size;
    struct scsi_host_cmd_pool *cmd_pool;

    /* temporary flag to disable blk-mq I/O path */
    bool disable_blk_mq;  //Disable universal block layer multi queue mode flag
};

Host adapter

SCSI host describes a SCSI host adapter, which is usually a PCI bus based expansion card or a SCSI controller chip. Each SCSI host adapter can have multiple channels, and one channel actually extends one SCSI bus. Each pass can connect more than one SCSI target node, and the specific number of connections is related to the SCSI bus carrying capacity, or limited by the specific SCSI protocol. The real host bus adapter is connected to the host IO bus (usually PCI bus). When the system is started, the devices attached to the PCI bus will be scanned and the host bus adapter will be assigned.
The SCSI host structure contains embedded general-purpose devices, which will be linked into the device list of SCSI bus type.

struct Scsi_Host {
    struct list_head    __devices; //device list
    struct list_head    __targets; //Target node list

    struct scsi_host_cmd_pool *cmd_pool; //scsi command buffer pool
    spinlock_t      free_list_lock;   //Protect free & list
    struct list_head    free_list; /* backup store of cmd structs, scsi Command pre assigned backup command list */
    struct list_head    starved_list; //Hunger list of scsi command

    spinlock_t      default_lock;
    spinlock_t      *host_lock;

    struct mutex        scan_mutex;/* serialize scanning activity */

    struct list_head    eh_cmd_q; //Chain list of executing wrong scsi commands
    struct task_struct    * ehandler;  /* Error recovery thread. Error recovery thread */
    struct completion     * eh_action; /* Wait for specific actions on the
                          host. */
    wait_queue_head_t       host_wait; //scsi device recovery waiting queue
    struct scsi_host_template *hostt;  //Host adapter template
    struct scsi_transport_template *transportt; //Point to SCSI transport layer template

    /*
     * Area to keep a shared tag map (if needed, will be
     * NULL if not).
     */
    union {
        struct blk_queue_tag    *bqt;
        struct blk_mq_tag_set   tag_set; //Use when SCSI supports multiple queues
    };
    //The number of scsi commands that have been sent to the host adapter (low level driver)
    atomic_t host_busy;        /* commands actually active on low-level */
    atomic_t host_blocked;  //Number of blocked scsi commands

    unsigned int host_failed;      /* commands that failed.
                          protected by host_lock */
    unsigned int host_eh_scheduled;    /* EH scheduled without command */

    unsigned int host_no;  /* Used for IOCTL_GET_IDLUN, /proc/scsi et al. Unique identification in the system */

    /* next two fields are used to bound the time spent in error handling */
    int eh_deadline;
    unsigned long last_reset; //Record last reset time


    /*
     * These three parameters can be used to allow for wide scsi,
     * and for host adapters that support multiple busses
     * The last two should be set to 1 more than the actual max id
     * or lun (e.g. 8 for SCSI parallel systems).
     */
    unsigned int max_channel; //Maximum channel number of the host adapter
    unsigned int max_id;      //Host adapter target node maximum number
    u64 max_lun;              //Host adapter lun maximum number

    unsigned int unique_id;

    /*
     * The maximum length of SCSI commands that this host can accept.
     * Probably 12 for most host adapters, but could be 16 for others.
     * or 260 if the driver supports variable length cdbs.
     * For drivers that don't set this field, a value of 12 is
     * assumed.
     */
    unsigned short max_cmd_len;  //The longest SCSI command a host adapter can accept
    //The following paragraph is also available in SCSI host template, which is assigned by the field in template
    int this_id;
    int can_queue;
    short cmd_per_lun;
    short unsigned int sg_tablesize;
    short unsigned int sg_prot_tablesize;
    unsigned int max_sectors;
    unsigned long dma_boundary;
    /*
     * In scsi-mq mode, the number of hardware queues supported by the LLD.
     *
     * Note: it is assumed that each hardware queue has a queue depth of
     * can_queue. In other words, the total queue depth per host
     * is nr_hw_queues * can_queue.
     */
    unsigned nr_hw_queues; //In the SCSI MQ mode, the number of hardware queues supported by low-level drivers
    /*
     * Used to assign serial numbers to the cmds.
     * Protected by the host lock.
     */
    unsigned long cmd_serial_number;  //Point to command serial number

    unsigned active_mode:2;           //The identity is initiator or target
    unsigned unchecked_isa_dma:1;
    unsigned use_clustering:1;

    /*
     * Host has requested that no further requests come through for the
     * time being.
     */
    unsigned host_self_blocked:1; //Indicates that the low-level driver requires blocking the host adapter. At this time, the middle level will not continue to send commands to the host adapter queue

    /*
     * Host uses correct SCSI ordering not PC ordering. The bit is
     * set for the minority of drivers whose authors actually read
     * the spec ;).
     */
    unsigned reverse_ordering:1;

    /* Task mgmt function in progress */
    unsigned tmf_in_progress:1;  //Task management function executing

    /* Asynchronous scan in progress */
    unsigned async_scan:1;       //Asynchronous scan in progress

    /* Don't resume host in EH */
    unsigned eh_noresume:1;      //Do not recover host adapter during error handling

    /* The controller does not support WRITE SAME */
    unsigned no_write_same:1;

    unsigned use_blk_mq:1;       //Whether to use SCSI multi queue mode
    unsigned use_cmd_list:1;

    /* Host responded with short (<36 bytes) INQUIRY result */
    unsigned short_inquiry:1;

    /*
     * Optional work queue to be utilized by the transport
     */
    char work_q_name[20];  //Work queues used by the scsi transport layer
    struct workqueue_struct *work_q;

    /*
     * Task management function work queue
     */
    struct workqueue_struct *tmf_work_q; //Task management function work queue

    /* The transport requires the LUN bits NOT to be stored in CDB[1] */
    unsigned no_scsi2_lun_in_cdb:1;

    /*
     * Value host_blocked counts down from
     */
    unsigned int max_host_blocked; //When the cumulative command in the dispatch queue reaches this value, the host adapter starts to wake up

    /* Protection Information */
    unsigned int prot_capabilities;
    unsigned char prot_guard_type;

    /*
     * q used for scsi_tgt msgs, async events or any other requests that
     * need to be processed in userspace
     */
    struct request_queue *uspace_req_q; //Request queues for SCSI ﹐ TGT messages, asynchronous events, or other requests that need to be processed in user space

    /* legacy crap */
    unsigned long base;
    unsigned long io_port;   //I/O port number
    unsigned char n_io_port;
    unsigned char dma_channel;
    unsigned int  irq;


    enum scsi_host_state shost_state; //state

    /* ldm bits */ //Shost ﹣ gendev: embedded general-purpose device, through which SCSI devices are linked to the device list of SCSI ﹣ bus ﹣ type
    struct device       shost_gendev, shost_dev;
    //shost_dev: embedded class device, through which SCSI devices are linked to the device list of the SCSI host adapter type (shost_class)
    /*
     * List of hosts per template.
     *
     * This is only for use by scsi_module.c for legacy templates.
     * For these access to it is synchronized implicitly by
     * module_init/module_exit.
     */
    struct list_head sht_legacy_list;

    /*
     * Points to the transport data (if any) which is allocated
     * separately
     */
    void *shost_data; //Point to independently allocated transport layer data, used by the SCSI transport layer

    /*
     * Points to the physical bus device we'd use to do DMA
     * Needed just in case we have virtual hosts.
     */
    struct device *dma_dev;

    /*
     * We should ensure that this is aligned, both for better performance
     * and also because some compilers (m68k) don't automatically force
     * alignment to a long boundary.
     */ //Host adapter specific data
    unsigned long hostdata[0]  /* Used for storage of host specific stuff */
        __attribute__ ((aligned (sizeof(unsigned long))));
};

Target node SCSI target

There is an embedded driver model device in the SCSI target structure, which is linked to the device list of SCSI bus type.

struct scsi_target {
    struct scsi_device  *starget_sdev_user; //Point to the scsi device with I/O in progress or NULL if there is no IO
    struct list_head    siblings;  //Link to host adapter target list
    struct list_head    devices;   //device linked list belonging to the target
    struct device       dev;       //General purpose device for adding device driven models
    struct kref     reap_ref; /* last put renders target invisible Reference count of this structure */
    unsigned int        channel;   //The channel number of the target
    unsigned int        id; /* target id ... replace
                     * scsi_device.id eventually */
    unsigned int        create:1; /* signal that it needs to be added */
    unsigned int        single_lun:1;   /* Indicates we should only
                         * allow I/O to one of the luns
                         * for the device at a time. */
    unsigned int        pdt_1f_for_no_lun:1;    /* PDT = 0x1f
                         * means no lun present. */
    unsigned int        no_report_luns:1;   /* Don't use
                         * REPORT LUNS for scanning. */
    unsigned int        expecting_lun_change:1; /* A device has reported
                         * a 3F/0E UA, other devices on
                         * the same target will also. */
    /* commands actually active on LLD. */
    atomic_t        target_busy;
    atomic_t        target_blocked;           //Number of currently blocked commands

    /*
     * LLDs should set this in the slave_alloc host template callout.
     * If set to zero then there is not limit.
     */
    unsigned int        can_queue;             //Number of commands processed at the same time
    unsigned int        max_target_blocked;    //Blocking command number threshold
#define SCSI_DEFAULT_TARGET_BLOCKED 3

    char            scsi_level;                //Supported SCSI specification levels
    enum scsi_target_state  state;             //target state
    void            *hostdata; /* available to low-level driver */
    unsigned long       starget_data[0]; /* for the transport SCSI Use of transport layer (middle layer) */
    /* starget_data must be the last element!!!! */
} __attribute__((aligned(sizeof(unsigned long))));

Logical device

scsi device describes the scsi logical device, which represents the logical unit lun of the scsi disk. The device represented by the scsi device descriptor may be a SATA/SAS/SCSI disk or SSD on another storage device. When the operating system scans the logical device connected to the host adapter, it creates a scsi device structure for the high-level driver of scsi to communicate with the device.

struct scsi_device {
    struct Scsi_Host *host;  //Host bus adapter
    struct request_queue *request_queue; //Request queue

    /* the next two are protected by the host->host_lock */
    struct list_head    siblings;   /* list of all devices on this host */ //Link in host bus adapter device list
    struct list_head    same_target_siblings; /* just the devices sharing same target id */ //List of devices linked to target

    atomic_t device_busy;       /* commands actually active on LLDD */
    atomic_t device_blocked;    /* Device returned QUEUE_FULL. */

    spinlock_t list_lock;
    struct list_head cmd_list;  /* queue of in use SCSI Command structures */
    struct list_head starved_entry; //"Hungry" list of chained host adapters
    struct scsi_cmnd *current_cmnd; /* currently active command */ //Currently executing command
    unsigned short queue_depth; /* How deep of a queue we want */
    unsigned short max_queue_depth; /* max queue depth */
    unsigned short last_queue_full_depth; /* These two are used by */
    unsigned short last_queue_full_count; /* scsi_track_queue_full() */
    unsigned long last_queue_full_time; /* last queue full time */
    unsigned long queue_ramp_up_period; /* ramp up period in jiffies */
#define SCSI_DEFAULT_RAMP_UP_PERIOD (120 * HZ)

    unsigned long last_queue_ramp_up;   /* last queue ramp up time */

    unsigned int id, channel; //target id and channel number of the SCSI device
    u64 lun;  //lun number of the device
    unsigned int manufacturer;  /* Manufacturer of device, for using  Manufacturer
                     * vendor-specific cmd's */
    unsigned sector_size;   /* size in bytes Sector size of hardware */

    void *hostdata;     /* available to low-level driver proprietary data */
    char type;          //SCSI device type
    char scsi_level;    //The version number of the supported SCSI specification, obtained by the inquire command
    char inq_periph_qual;   /* PQ from INQUIRY data */
    unsigned char inquiry_len;  /* valid bytes in 'inquiry' */
    unsigned char * inquiry;    /* INQUIRY response data */
    const char * vendor;        /* [back_compat] point into 'inquiry' ... */
    const char * model;     /* ... after scan; point to static string */
    const char * rev;       /* ... "nullnullnullnull" before scan */

#define SCSI_VPD_PG_LEN                255
    int vpd_pg83_len;          //sense command 0x83
    unsigned char *vpd_pg83;
    int vpd_pg80_len;          //sense command 0x80
    unsigned char *vpd_pg80;
    unsigned char current_tag;  /* current tag */
    struct scsi_target      *sdev_target;   /* used only for single_lun */

    unsigned int    sdev_bflags; /* black/white flags as also found in
                 * scsi_devinfo.[hc]. For now used only to
                 * pass settings from slave_alloc to scsi
                 * core. */
    unsigned int eh_timeout; /* Error handling timeout */
    unsigned removable:1;
    unsigned changed:1; /* Data invalid due to media change */
    unsigned busy:1;    /* Used to prevent races */
    unsigned lockable:1;    /* Able to prevent media removal */
    unsigned locked:1;      /* Media removal disabled */
    unsigned borken:1;  /* Tell the Seagate driver to be
                 * painfully slow on this device */
    unsigned disconnect:1;  /* can disconnect */
    unsigned soft_reset:1;  /* Uses soft reset option */
    unsigned sdtr:1;    /* Device supports SDTR messages Support synchronous data transmission */
    unsigned wdtr:1;    /* Device supports WDTR messages Support 16 bit wide data transmission*/
    unsigned ppr:1;     /* Device supports PPR messages Support PPR (parallel protocol request) messages*/
    unsigned tagged_supported:1;    /* Supports SCSI-II tagged queuing */
    unsigned simple_tags:1; /* simple queue tag messages are enabled */
    unsigned was_reset:1;   /* There was a bus reset on the bus for
                 * this device */
    unsigned expecting_cc_ua:1; /* Expecting a CHECK_CONDITION/UNIT_ATTN
                     * because we did a bus reset. */
    unsigned use_10_for_rw:1; /* first try 10-byte read / write */
    unsigned use_10_for_ms:1; /* first try 10-byte mode sense/select */
    unsigned no_report_opcodes:1;   /* no REPORT SUPPORTED OPERATION CODES */
    unsigned no_write_same:1;   /* no WRITE SAME command */
    unsigned use_16_for_rw:1; /* Use read/write(16) over read/write(10) */
    unsigned skip_ms_page_8:1;  /* do not use MODE SENSE page 0x08 */
    unsigned skip_ms_page_3f:1; /* do not use MODE SENSE page 0x3f */
    unsigned skip_vpd_pages:1;  /* do not read VPD pages */
    unsigned try_vpd_pages:1;   /* attempt to read VPD pages */
    unsigned use_192_bytes_for_3f:1; /* ask for 192 bytes from page 0x3f */
    unsigned no_start_on_add:1; /* do not issue start on add */
    unsigned allow_restart:1; /* issue START_UNIT in error handler */
    unsigned manage_start_stop:1;   /* Let HLD (sd) manage start/stop */
    unsigned start_stop_pwr_cond:1; /* Set power cond. in START_STOP_UNIT */
    unsigned no_uld_attach:1; /* disable connecting to upper level drivers */
    unsigned select_no_atn:1;
    unsigned fix_capacity:1;    /* READ_CAPACITY is too high by 1 */
    unsigned guess_capacity:1;  /* READ_CAPACITY might be too high by 1 */
    unsigned retry_hwerror:1;   /* Retry HARDWARE_ERROR */
    unsigned last_sector_bug:1; /* do not use multisector accesses on
                       SD_LAST_BUGGY_SECTORS */
    unsigned no_read_disc_info:1;   /* Avoid READ_DISC_INFO cmds */
    unsigned no_read_capacity_16:1; /* Avoid READ_CAPACITY_16 cmds */
    unsigned try_rc_10_first:1; /* Try READ_CAPACACITY_10 first */
    unsigned is_visible:1;  /* is the device visible in sysfs */
    unsigned wce_default_on:1;  /* Cache is ON by default */
    unsigned no_dif:1;  /* T10 PI (DIF) should be disabled */
    unsigned broken_fua:1;      /* Don't set FUA bit */
    unsigned lun_in_cdb:1;      /* Store LUN bits in CDB[1] */

    atomic_t disk_events_disable_depth; /* disable depth for disk events */

    DECLARE_BITMAP(supported_events, SDEV_EVT_MAXBITS); /* supported events */
    DECLARE_BITMAP(pending_events, SDEV_EVT_MAXBITS); /* pending events */
    struct list_head event_list;    /* asserted events */
    struct work_struct event_work;

    unsigned int max_device_blocked; /* what device_blocked counts down from  */
#define SCSI_DEFAULT_DEVICE_BLOCKED 3

    atomic_t iorequest_cnt;
    atomic_t iodone_cnt;
    atomic_t ioerr_cnt;

    struct device       sdev_gendev, //Embedded general equipment, linked to the device list of scsi bus type
                sdev_dev; //Embedded device, linked to the device list of the scsi device class

    struct execute_work ew; /* used to get process context on put */
    struct work_struct  requeue_work;

    struct scsi_device_handler *handler; //Custom device handling functions
    void            *handler_data;

    enum scsi_device_state sdev_state;  //scsi device status
    unsigned long       sdev_data[0];   //scsi transport layer usage
} __attribute__((aligned(sizeof(unsigned long))));

SCSI command structure defined by kernel

The SCSI ﹣ cmnd structure is created by the SCSI middle layer and passed to the SCSI low layer driver. Each IO request will be created with a SCSI cnmd, but SCSI cmnd is not necessarily a time IO request. SCSI cmnd is finally converted into a specific SCSI command. In addition to the command description block, SCSI ﹣ cmnd contains more information, including data buffer, sense data buffer, complete callback function and the related block device driver layer request, etc., which is the context of SCSI command execution in the SCSI middle layer.

struct scsi_cmnd {
    struct scsi_device *device;  //Pointer to the descriptor of the SCSI device to which the command belongs
    struct list_head list;  /* scsi_cmnd participates in queue lists List of commands linked to scsi devices */
    struct list_head eh_entry; /* entry for the host eh_cmd_q */
    struct delayed_work abort_work;
    int eh_eflags;      /* Used by error handlr */

    /*
     * A SCSI Command is assigned a nonzero serial_number before passed
     * to the driver's queue command function.  The serial_number is
     * cleared when scsi_done is entered indicating that the command
     * has been completed.  It is a bug for LLDDs to use this number
     * for purposes other than printk (and even that is only useful
     * for debugging).
     */
    unsigned long serial_number; //Unique sequence number of scsi command

    /*
     * This is set to jiffies as it was when the command was first
     * allocated.  It is used to time how long the command has
     * been outstanding
     */
    unsigned long jiffies_at_alloc; //jiffies at allocation time, used to calculate command processing time

    int retries;  //Number of command retries
    int allowed;  //Number of retries allowed

    unsigned char prot_op;    //Protection operation (DIF and DIX)
    unsigned char prot_type;  //DIF protection type
    unsigned char prot_flags;

    unsigned short cmd_len;   //Command length
    enum dma_data_direction sc_data_direction;  //Command transmission direction

    /* These elements define the operation we are about to perform */
    unsigned char *cmnd;  //Command string in scsi format


    /* These elements define the operation we ultimately want to perform */
    struct scsi_data_buffer sdb;        //scsi command data buffer
    struct scsi_data_buffer *prot_sdb;  //scsi command protection information buffer

    unsigned underflow; /* Return error if less than
                   this amount is transferred */

    unsigned transfersize;  /* How much we are guaranteed to  //Transmission unit
                   transfer with each SCSI transfer
                   (ie, between disconnect /
                   reconnects.   Probably == sector
                   size */

    struct request *request;    /* The command we are  Request descriptor of general block layer
                       working on */

#define SCSI_SENSE_BUFFERSIZE   96
    unsigned char *sense_buffer;    //scsi command sense data buffer
                /* obtained by REQUEST SENSE when
                 * CHECK CONDITION is received on original
                 * command (auto-sense) */

    /* Low-level done function - can be used by low-level driver to point
     *        to completion function.  Not used by mid/upper level code. */
    void (*scsi_done) (struct scsi_cmnd *); //When the scsi command is completed, the

    /*
     * The following fields can be written to by the host specific code.
     * Everything else should be left alone.
     */
    struct scsi_pointer SCp;    /* Scratchpad used by some host adapters */

    unsigned char *host_scribble;   /* The host adapter is allowed to
                     * call scsi_malloc and get some memory
                     * and hang it here.  The host adapter
                     * is also expected to call scsi_free
                     * to release this memory.  (The memory
                     * obtained by scsi_malloc is guaranteed
                     * to be at an address < 16Mb). */

    int result;     /* Status code from lower level driver */
    int flags;      /* Command flags */

    unsigned char tag;  /* SCSI-II queued command tag */
};

Driving the SCSI driver

struct scsi_driver {
    struct device_driver    gendrv;  // "Inherit" device? Driver

    void (*rescan)(struct device *); //Callback function before scanning
    int (*init_command)(struct scsi_cmnd *);
    void (*uninit_command)(struct scsi_cmnd *);
    int (*done)(struct scsi_cmnd *);  //Called when the lower driver completes a scsi command to calculate the number of bytes that have been completed
    int (*eh_action)(struct scsi_cmnd *, int); //Error handling callback
};

Device model

SCSI bus type: SCSI subsystem bus type

struct bus_type scsi_bus_type = {
        .name       = "scsi",   // Corresponding to / sys/bus/scsi
        .match      = scsi_bus_match,
    .uevent     = scsi_bus_uevent,
#ifdef CONFIG_PM
    .pm     = &scsi_bus_pm_ops,
#endif
};
EXPORT_SYMBOL_GPL(scsi_bus_type);

shost_class: scsi subsystem class

static struct class shost_class = {
    .name       = "scsi_host",  // Corresponding to / sys / class / SCSI host
    .dev_release    = scsi_host_cls_release,
};

Initialization process

When the operating system starts, the scsi subsystem will be loaded. The entry function is init? scsi. Use subsys? Initcall to define:

static int __init init_scsi(void)
{
    int error;

    error = scsi_init_queue();  //Storage pool required to initialize the hash table
    if (error)
        return error;
    error = scsi_init_procfs(); //Initialize the scsi related directory items in procfs
    if (error)
        goto cleanup_queue;
    error = scsi_init_devinfo();//Set scsi dynamic device information list
    if (error)
        goto cleanup_procfs;
    error = scsi_init_hosts();  //Register the shost ﹐ class and create the SCSI ﹐ host subdirectory under the / sys/class /
    if (error)
        goto cleanup_devlist;
    error = scsi_init_sysctl(); //Register SCSI system control table
    if (error)
        goto cleanup_hosts;
    error = scsi_sysfs_register(); //Register the SCSI bus type bus type and SDEV class class
    if (error)
        goto cleanup_sysctl;

    scsi_netlink_init(); //Initializing the SCSI transport netlink interface

    printk(KERN_NOTICE "SCSI subsystem initialized\n");
    return 0;

cleanup_sysctl:
    scsi_exit_sysctl();
cleanup_hosts:
    scsi_exit_hosts();
cleanup_devlist:
    scsi_exit_devinfo();
cleanup_procfs:
    scsi_exit_procfs();
cleanup_queue:
    scsi_exit_queue();
    printk(KERN_ERR "SCSI subsystem failed to initialize, error = %d\n",
           -error);
    return error;
}

The scsi init hosts function initializes the class of the host adapter of the scsi subsystem, shost class:

int scsi_init_hosts(void)
{
    return class_register(&shost_class);
}

The scsi sysfs register function initializes the scsi subsystem bus type and the SDEV class class to which the device belongs

int scsi_sysfs_register(void)
{
    int error;

    error = bus_register(&scsi_bus_type);
    if (!error) {
        error = class_register(&sdev_class);
        if (error)
            bus_unregister(&scsi_bus_type);
    }

    return error;
}

The scsi low-level driver is oriented to the host adapter. When the low-level driver is loaded, the host adapter needs to be added. There are two ways to add the host adapter: 1. Add it when the PCI subsystem scans the attached driver; 2. Add it manually. All host adapters based on hardware PCI interface adopt the first method. Adding a host adapter consists of two steps:
1. The data structure of the host adapter is SCSI host alloc
2. Add the host adapter to the system SCSI add host

struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
{
    struct Scsi_Host *shost;
    gfp_t gfp_mask = GFP_KERNEL;

    if (sht->unchecked_isa_dma && privsize)
        gfp_mask |= __GFP_DMA;
    //One time allocation of SCSI host and private data space
    shost = kzalloc(sizeof(struct Scsi_Host) + privsize, gfp_mask);
    if (!shost)
        return NULL;

    shost->host_lock = &shost->default_lock;
    spin_lock_init(shost->host_lock);
    shost->shost_state = SHOST_CREATED; //Update state
    INIT_LIST_HEAD(&shost->__devices);  //Initializing the scsi device list
    INIT_LIST_HEAD(&shost->__targets);  //Initialize target list
    INIT_LIST_HEAD(&shost->eh_cmd_q);   //Initializing the list of scsi commands that execute incorrectly
    INIT_LIST_HEAD(&shost->starved_list);   //Initialize scsi command hunger list
    init_waitqueue_head(&shost->host_wait);
    mutex_init(&shost->scan_mutex);

    /*
     * subtract one because we increment first then return, but we need to
     * know what the next host number was before increment
     */ //Incrementally assign host adapter number
    shost->host_no = atomic_inc_return(&scsi_host_next_hn) - 1;
    shost->dma_channel = 0xff;

    /* These three are default values which can be overridden */
    shost->max_channel = 0; //The default channel number is 0
    shost->max_id = 8;      //Default target Max
    shost->max_lun = 8;     //Default maximum number of SCSI device

    /* Give each shost a default transportt */
    shost->transportt = &blank_transport_template;  //scsi transport layer (middle layer) template

    /*
     * All drivers right now should be able to handle 12 byte
     * commands.  Every so often there are requests for 16 byte
     * commands, but individual low-level drivers need to certify that
     * they actually do something sensible with such commands.
     */
    shost->max_cmd_len = 12;  //Maximum SCSI command length
    shost->hostt = sht;       //Use host adapter template
    shost->this_id = sht->this_id;
    shost->can_queue = sht->can_queue;
    shost->sg_tablesize = sht->sg_tablesize;
    shost->sg_prot_tablesize = sht->sg_prot_tablesize;
    shost->cmd_per_lun = sht->cmd_per_lun;
    shost->unchecked_isa_dma = sht->unchecked_isa_dma;
    shost->use_clustering = sht->use_clustering;
    shost->no_write_same = sht->no_write_same;

    if (shost_eh_deadline == -1 || !sht->eh_host_reset_handler)
        shost->eh_deadline = -1;
    else if ((ulong) shost_eh_deadline * HZ > INT_MAX) {
        shost_printk(KERN_WARNING, shost,
                 "eh_deadline %u too large, setting to %u\n",
                 shost_eh_deadline, INT_MAX / HZ);
        shost->eh_deadline = INT_MAX;
    } else
        shost->eh_deadline = shost_eh_deadline * HZ;

    if (sht->supported_mode == MODE_UNKNOWN) //Specify the mode of the HbAS by the template
        /* means we didn't set it ... default to INITIATOR */
        shost->active_mode = MODE_INITIATOR;  //The default host adapter mode is initiator
    else
        shost->active_mode = sht->supported_mode;

    if (sht->max_host_blocked)
        shost->max_host_blocked = sht->max_host_blocked;
    else
        shost->max_host_blocked = SCSI_DEFAULT_HOST_BLOCKED;

    /*
     * If the driver imposes no hard sector transfer limit, start at
     * machine infinity initially.
     */
    if (sht->max_sectors)
        shost->max_sectors = sht->max_sectors;
    else
        shost->max_sectors = SCSI_DEFAULT_MAX_SECTORS;

    /*
     * assume a 4GB boundary, if not set
     */
    if (sht->dma_boundary)
        shost->dma_boundary = sht->dma_boundary;
    else
        shost->dma_boundary = 0xffffffff;  //The default DMA boundary is 4G

    shost->use_blk_mq = scsi_use_blk_mq && !shost->hostt->disable_blk_mq;

    device_initialize(&shost->shost_gendev); //Initializing the host adapter internal generic device
    dev_set_name(&shost->shost_gendev, "host%d", shost->host_no);
    shost->shost_gendev.bus = &scsi_bus_type;   //Set the bus type of the host adapter
    shost->shost_gendev.type = &scsi_host_type; //Set the device type of the host adapter

    device_initialize(&shost->shost_dev);    //Initializing the host adapter's internal class device
    shost->shost_dev.parent = &shost->shost_gendev; //The parent device of the internal class device is set as its internal common device
    shost->shost_dev.class = &shost_class;   //Set the class to which the internal class device belongs to as shost class
    dev_set_name(&shost->shost_dev, "host%d", shost->host_no);
    shost->shost_dev.groups = scsi_sysfs_shost_attr_groups;  //Set property group of class device

    shost->ehandler = kthread_run(scsi_error_handler, shost,  //Starting the error recovery kernel thread of the host adapter
            "scsi_eh_%d", shost->host_no);
    if (IS_ERR(shost->ehandler)) {
        shost_printk(KERN_WARNING, shost,
            "error handler thread failed to spawn, error = %ld\n",
            PTR_ERR(shost->ehandler));
        goto fail_kfree;
    }
    //Assign task management work queue
    shost->tmf_work_q = alloc_workqueue("scsi_tmf_%d",
                        WQ_UNBOUND | WQ_MEM_RECLAIM,
                       1, shost->host_no);
    if (!shost->tmf_work_q) {
        shost_printk(KERN_WARNING, shost,
                 "failed to create tmf workq\n");
        goto fail_kthread;
    }
    scsi_proc_hostdir_add(shost->hostt); //Add the directory of host adapter in procfs, eg. / / create / proc / SCSI / < host adapter name > directory
    return shost;

 fail_kthread:
    kthread_stop(shost->ehandler);
 fail_kfree:
    kfree(shost);
    return NULL;
}
EXPORT_SYMBOL(scsi_host_alloc);

static inline int __must_check scsi_add_host(struct Scsi_Host *host,
                         struct device *dev) //dev is the parent device
{
    return scsi_add_host_with_dma(host, dev, dev);
}

int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
               struct device *dma_dev)
{
    struct scsi_host_template *sht = shost->hostt;
    int error = -EINVAL;

    shost_printk(KERN_INFO, shost, "%s\n",
            sht->info ? sht->info(shost) : sht->name);

    if (!shost->can_queue) {
        shost_printk(KERN_ERR, shost,
                 "can_queue = 0 no longer supported\n");
        goto fail;
    }

    if (shost_use_blk_mq(shost)) {         //If the host adapter is set to use multi queue IO, the
        error = scsi_mq_setup_tags(shost); //Corresponding multi queue environment
        if (error)
            goto fail;
    } else {
        shost->bqt = blk_init_tags(shost->can_queue,
                shost->hostt->tag_alloc_policy);
        if (!shost->bqt) {
            error = -ENOMEM;
            goto fail;
        }
    }

    /*
     * Note that we allocate the freelist even for the MQ case for now,
     * as we need a command set aside for scsi_reset_provider.  Having
     * the full host freelist and one command available for that is a
     * little heavy-handed, but avoids introducing a special allocator
     * just for this.  Eventually the structure of scsi_reset_provider
     * will need a major overhaul.
     */ //Allocate the buffer for storing scsi command and sense data, and allocate the spare warehouse list for scsi command
    error = scsi_setup_command_freelist(shost);
    if (error)
        goto out_destroy_tags;

    //Set the parent device of the host adapter to determine the location of the device in sysfs, which is usually passed in PCI? Dev through the dev parameter.
    if (!shost->shost_gendev.parent)
        shost->shost_gendev.parent = dev ? dev : &platform_bus; //If dev is NULL, set to platform bus
    if (!dma_dev)
        dma_dev = shost->shost_gendev.parent;

    shost->dma_dev = dma_dev;

    error = device_add(&shost->shost_gendev);  //Add host adapter universal device to system
    if (error)
        goto out_destroy_freelist;

    pm_runtime_set_active(&shost->shost_gendev);
    pm_runtime_enable(&shost->shost_gendev);
    device_enable_async_suspend(&shost->shost_gendev); //Supports asynchronous suspend of generic devices

    scsi_host_set_state(shost, SHOST_RUNNING);  //Set host adapter status
    get_device(shost->shost_gendev.parent);     //Increase the reference count of the universal parent device

    device_enable_async_suspend(&shost->shost_dev);  //Supports asynchronous suspend devices

    error = device_add(&shost->shost_dev);    //Add host adapter class device to the system
    if (error)
        goto out_del_gendev;

    get_device(&shost->shost_gendev);

    if (shost->transportt->host_size) {  //Data space used by scsi transport layer
        shost->shost_data = kzalloc(shost->transportt->host_size,
                     GFP_KERNEL);
        if (shost->shost_data == NULL) {
            error = -ENOMEM;
            goto out_del_dev;
        }
    }

    if (shost->transportt->create_work_queue) {
        snprintf(shost->work_q_name, sizeof(shost->work_q_name),
             "scsi_wq_%d", shost->host_no);
        shost->work_q = create_singlethread_workqueue( //Assign work queues used by the scsi transport layer
                    shost->work_q_name);
        if (!shost->work_q) {
            error = -EINVAL;
            goto out_free_shost_data;
        }
    }

    error = scsi_sysfs_add_host(shost); //Add host adapter to subsystem
    if (error)
        goto out_destroy_host;

    scsi_proc_host_add(shost);  //Add host adapter information in procfs
    return error;

 out_destroy_host:
    if (shost->work_q)
        destroy_workqueue(shost->work_q);
 out_free_shost_data:
    kfree(shost->shost_data);
 out_del_dev:
    device_del(&shost->shost_dev);
 out_del_gendev:
    device_del(&shost->shost_gendev);
 out_destroy_freelist:
    scsi_destroy_command_freelist(shost);
 out_destroy_tags:
    if (shost_use_blk_mq(shost))
        scsi_mq_destroy_tags(shost);
 fail:
    return error;
}
EXPORT_SYMBOL(scsi_add_host_with_dma);

Equipment detection process

During the system startup process, the default PCI root bus will be scanned, which triggers the process of PCI device scanning, and the PCI device tree will be constructed. The SCSI host adapter is the device mounted on the PCI bus. As a PCI device, the SCSI host adapter will be scanned by the PCI bus driver layer (the scan of PCI device adopts the way of configuration space access). After scanning the SCSI host adapter, the operating system starts to load the SCSI host adapter driver, which is the low-level driver mentioned above. The SCSI host adapter driver allocates the SCSI host adapter descriptor according to the SCSI host adapter driver according to the SCSI host adapter template, and adds it to the system, and then starts the scanning process of the next level bus - SCSI bus extended through the SCSI host adapter.

The SCSI middle layer constructs the INQUIRY commands with possible ID and LUN in turn, and then submits these INQUIRY commands to the block IO subsystem. The latter will finally call the SCSI middle layer's policy routine, extract the SCSI command structure again, and then call the queuecommand callback function driven by the SCSI low layer to implement.
For the target node with given ID, if it exists on the SCSI bus, it must implement the INQUIRY response to LUN0. That is to say, if you send the INQUIRY command to LUN0 of the target node of an ID, or try to send the INQUIRY command to each LUN in turn, check whether you can receive the response, and finally the SCSI middle layer can get the connected logical devices and their information in the SCSI domain.

The specific scan mode of SCSI bus can be realized by specific host adapter firmware and host adapter driver. In this paper, we only discuss the implementation mode that the host adapter driver calls the general scan function provided by the SCSI middle layer.

void scsi_scan_host(struct Scsi_Host *shost)
{
    struct async_scan_data *data;

    if (strncmp(scsi_scan_type, "none", 4) == 0) //Check scan logic
        return;
    if (scsi_autopm_get_host(shost) < 0)
        return;

    data = scsi_prep_async_scan(shost); //Prepare asynchronous scan
    if (!data) {
        do_scsi_scan_host(shost);    //Synchronous scanning
        scsi_autopm_put_host(shost);
        return;
    }

    /* register with the async subsystem so wait_for_device_probe()
     * will flush this work
     */
    async_schedule(do_scan_async, data);  //Asynchronous scanning

    /* scsi_autopm_put_host(shost) is called in scsi_finish_async_scan() */
}
EXPORT_SYMBOL(scsi_scan_host);

The scsi scan host function is a host adapter scan function provided by the scsi middle layer. For those with host adapter driver and custom scan logic requirements, you can set the callback function of the host adapter template. The scsi scan host function calls the callback to implement the custom scan.
The SCSI? Scan? Type variable specifies the scan mode: async, sync, none. Whether the final scanning mode is synchronous or asynchronous, it is implemented by the do ﹣ SCSI ﹣ scan ﹣ host function:

static void do_scsi_scan_host(struct Scsi_Host *shost)
{
    if (shost->hostt->scan_finished) {  //Use custom scan method
        unsigned long start = jiffies;
        if (shost->hostt->scan_start)
            shost->hostt->scan_start(shost); //Custom scan start callback

        while (!shost->hostt->scan_finished(shost, jiffies - start)) //Return 1 when custom scan is complete
            msleep(10);
    } else { //Scan? Wild? Card means to scan all target s and device s
        scsi_scan_host_selected(shost, SCAN_WILD_CARD, SCAN_WILD_CARD,
                SCAN_WILD_CARD, 0);
    }
}

If the host adapter template has a custom scan function set, the do? SCSI? Scan? Host function will be called. If not, the Default scan function, SCSI > scan > host > selected, is used to perform the scan.

int scsi_scan_host_selected(struct Scsi_Host *shost, unsigned int channel,
                unsigned int id, u64 lun, int rescan)
{
    SCSI_LOG_SCAN_BUS(3, shost_printk (KERN_INFO, shost,
        "%s: <%u:%u:%llu>\n",
        __func__, channel, id, lun));
    //Check whether channel, id and lun are valid
    if (((channel != SCAN_WILD_CARD) && (channel > shost->max_channel)) ||
        ((id != SCAN_WILD_CARD) && (id >= shost->max_id)) ||
        ((lun != SCAN_WILD_CARD) && (lun >= shost->max_lun)))
        return -EINVAL;

    mutex_lock(&shost->scan_mutex);
    if (!shost->async_scan)
        scsi_complete_async_scans();
    //Check if the status of the SCSI host allows scanning
    if (scsi_host_scan_allowed(shost) && scsi_autopm_get_host(shost) == 0) {
        if (channel == SCAN_WILD_CARD)
            for (channel = 0; channel <= shost->max_channel; //Scan through all channel s
                 channel++)
                scsi_scan_channel(shost, channel, id, lun,  //Scan channel
                          rescan);
        else
            scsi_scan_channel(shost, channel, id, lun, rescan); //Scan the specified channel
        scsi_autopm_put_host(shost);
    }
    mutex_unlock(&shost->scan_mutex);

    return 0;
}

The SCSI scan host selected function scans the specified host adapter, determines whether to scan all channels or scan the specified channels according to the input parameters, and completes it through the SCSI scan channel function.

static void scsi_scan_channel(struct Scsi_Host *shost, unsigned int channel,
                  unsigned int id, u64 lun, int rescan)
{
    uint order_id;

    if (id == SCAN_WILD_CARD)
        for (id = 0; id < shost->max_id; ++id) {  //Traverse all target s
            /*
             * XXX adapter drivers when possible (FCP, iSCSI)
             * could modify max_id to match the current max,
             * not the absolute max.
             *
             * XXX add a shost id iterator, so for example,
             * the FC ID can be the same as a target id
             * without a huge overhead of sparse id's.
             */
            if (shost->reverse_ordering)
                /*
                 * Scan from high to low id.
                 */
                order_id = shost->max_id - id - 1;
            else
                order_id = id;
            __scsi_scan_target(&shost->shost_gendev, channel, //Scan the specified target
                    order_id, lun, rescan);
        }
    else
        __scsi_scan_target(&shost->shost_gendev, channel,
                id, lun, rescan);
}

__The SCSI target function specifies the lun inside the scan target.

static void __scsi_scan_target(struct device *parent, unsigned int channel,
        unsigned int id, u64 lun, int rescan)
{
    struct Scsi_Host *shost = dev_to_shost(parent);
    int bflags = 0;
    int res;
    struct scsi_target *starget;

    if (shost->this_id == id)
        /*
         * Don't scan the host adapter
         */
        return;
    //Assigns a target data structure to the specified id and initializes
    starget = scsi_alloc_target(parent, channel, id);
    if (!starget)
        return;
    scsi_autopm_get_target(starget);

    if (lun != SCAN_WILD_CARD) {
        /*
         * Scan for a specific host/chan/id/lun.
         */ //Scan the SCSI device (Lun) with the id specified in the target and add the SCSI device (Lun) to the subsystem
        scsi_probe_and_add_lun(starget, lun, NULL, NULL, rescan, NULL);
        goto out_reap;
    }

    /*
     * Scan LUN 0, if there is some response, scan further. Ideally, we
     * would not configure LUN 0 until all LUNs are scanned.
     */ //Probe target's LUN0
    res = scsi_probe_and_add_lun(starget, 0, &bflags, NULL, rescan, NULL);
    if (res == SCSI_SCAN_LUN_PRESENT || res == SCSI_SCAN_TARGET_PRESENT) {
        if (scsi_report_lun_scan(starget, bflags, rescan) != 0) //Send report to target lun 0
            /*
             * The REPORT LUN did not scan the target,
             * do a sequential scan.
             */
            scsi_sequential_lun_scan(starget, bflags,  //Detect the reported lun
                         starget->scsi_level, rescan);
    }

 out_reap:
    scsi_autopm_put_target(starget);
    /*
     * paired with scsi_alloc_target(): determine if the target has
     * any children at all and if not, nuke it
     */
    scsi_target_reap(starget);

    put_device(&starget->dev);
}

When the target is scanned, the SCSI target structure is allocated and initialized. The SCSI probe and add functions detect the lun in the target and add the discovered lun to the system.

static int scsi_probe_and_add_lun(struct scsi_target *starget,
                  u64 lun, int *bflagsp,
                  struct scsi_device **sdevp, int rescan,
                  void *hostdata)
{
    struct scsi_device *sdev;
    unsigned char *result;
    int bflags, res = SCSI_SCAN_NO_RESPONSE, result_len = 256;
    struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);

    /*
     * The rescan flag is used as an optimization, the first scan of a
     * host adapter calls into here with rescan == 0.
     */
    sdev = scsi_device_lookup_by_target(starget, lun);  //Find lun with id specified in target
    if (sdev) {   //lun already exists in target
        if (rescan || !scsi_device_created(sdev)) { //The rescan parameter requires rescan of the lun
            SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
                "scsi scan: device exists on %s\n",
                dev_name(&sdev->sdev_gendev)));
            if (sdevp)
                *sdevp = sdev;
            else
                scsi_device_put(sdev);

            if (bflagsp)
                *bflagsp = scsi_get_device_flags(sdev,
                                 sdev->vendor,
                                 sdev->model);
            return SCSI_SCAN_LUN_PRESENT;
        }
        scsi_device_put(sdev);
    } else
        sdev = scsi_alloc_sdev(starget, lun, hostdata); //lun does not exist in target, assign SCSI device
    if (!sdev)
        goto out;

    result = kmalloc(result_len, GFP_ATOMIC |
            ((shost->unchecked_isa_dma) ? __GFP_DMA : 0));
    if (!result)
        goto out_free_sdev;

    if (scsi_probe_lun(sdev, result, result_len, &bflags)) //Send INQUIRY to specific device for detection
        goto out_free_result;

    if (bflagsp)
        *bflagsp = bflags;
    /*
     * result contains valid SCSI INQUIRY data.
     */
    if (((result[0] >> 5) == 3) && !(bflags & BLIST_ATTACH_PQ3)) {
        /*
         * For a Peripheral qualifier 3 (011b), the SCSI
         * spec says: The device server is not capable of
         * supporting a physical device on this logical
         * unit.
         *
         * For disks, this implies that there is no
         * logical disk configured at sdev->lun, but there
         * is a target id responding.
         */
        SCSI_LOG_SCAN_BUS(2, sdev_printk(KERN_INFO, sdev, "scsi scan:"
                   " peripheral qualifier of 3, device not"
                   " added\n"))
        if (lun == 0) {
            SCSI_LOG_SCAN_BUS(1, {
                unsigned char vend[9];
                unsigned char mod[17];

                sdev_printk(KERN_INFO, sdev,
                    "scsi scan: consider passing scsi_mod."
                    "dev_flags=%s:%s:0x240 or 0x1000240\n",
                    scsi_inq_str(vend, result, 8, 16),
                    scsi_inq_str(mod, result, 16, 32));
            });

        }

        res = SCSI_SCAN_TARGET_PRESENT;
        goto out_free_result;
    }

    /*
     * Some targets may set slight variations of PQ and PDT to signal
     * that no LUN is present, so don't add sdev in these cases.
     * Two specific examples are:
     * 1) NetApp targets: return PQ=1, PDT=0x1f
     * 2) USB UFI: returns PDT=0x1f, with the PQ bits being "reserved"
     *    in the UFI 1.0 spec (we cannot rely on reserved bits).
     *
     * References:
     * 1) SCSI SPC-3, pp. 145-146
     * PQ=1: "A peripheral device having the specified peripheral
     * device type is not connected to this logical unit. However, the
     * device server is capable of supporting the specified peripheral
     * device type on this logical unit."
     * PDT=0x1f: "Unknown or no device type"
     * 2) USB UFI 1.0, p. 20
     * PDT=00h Direct-access device (floppy)
     * PDT=1Fh none (no FDD connected to the requested logical unit)
     */
    if (((result[0] >> 5) == 1 || starget->pdt_1f_for_no_lun) &&
        (result[0] & 0x1f) == 0x1f &&
        !scsi_is_wlun(lun)) {
        SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
                    "scsi scan: peripheral device type"
                    " of 31, no device added\n"));
        res = SCSI_SCAN_TARGET_PRESENT;
        goto out_free_result;
    }
    //Add scsi device to subsystem
    res = scsi_add_lun(sdev, result, &bflags, shost->async_scan);
    if (res == SCSI_SCAN_LUN_PRESENT) {
        if (bflags & BLIST_KEY) {
            sdev->lockable = 0;
            scsi_unlock_floptical(sdev, result);
        }
    }

 out_free_result:
    kfree(result);
 out_free_sdev:
    if (res == SCSI_SCAN_LUN_PRESENT) {
        if (sdevp) {
            if (scsi_device_get(sdev) == 0) {
                *sdevp = sdev;
            } else {
                __scsi_remove_device(sdev);
                res = SCSI_SCAN_NO_RESPONSE;
            }
        }
    } else
        __scsi_remove_device(sdev);
 out:
    return res;
}

The SCSI probe and add functions can be seen from the name to complete the probe and add operations of lun
1. Detect the logical device SCSI probe Lun and send the INQUIRY command to the specific device.
2. Add logical devices to the system SCSI add lun, and add lun to the system according to the return value of INQUIRY command.

static int scsi_probe_lun(struct scsi_device *sdev, unsigned char *inq_result,
              int result_len, int *bflags)
{
    unsigned char scsi_cmd[MAX_COMMAND_SIZE];
    int first_inquiry_len, try_inquiry_len, next_inquiry_len;
    int response_len = 0;
    int pass, count, result;
    struct scsi_sense_hdr sshdr;

    *bflags = 0;

    /* Perform up to 3 passes.  The first pass uses a conservative
     * transfer length of 36 unless sdev->inquiry_len specifies a
     * different value. */
    first_inquiry_len = sdev->inquiry_len ? sdev->inquiry_len : 36;
    try_inquiry_len = first_inquiry_len;
    pass = 1;

 next_pass:
    SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
                "scsi scan: INQUIRY pass %d length %d\n",
                pass, try_inquiry_len));

    /* Each pass gets up to three chances to ignore Unit Attention */
    for (count = 0; count < 3; ++count) {
        int resid;

        memset(scsi_cmd, 0, 6);
        scsi_cmd[0] = INQUIRY;      //The command type isINQUIRY
        scsi_cmd[4] = (unsigned char) try_inquiry_len;

        memset(inq_result, 0, try_inquiry_len);
        //Send outSCSICommand, retry3second
        result = scsi_execute_req(sdev,  scsi_cmd, DMA_FROM_DEVICE,
                      inq_result, try_inquiry_len, &sshdr,
                      HZ / 2 + HZ * scsi_inq_timeout, 3,
                      &resid);

        SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
                "scsi scan: INQUIRY %s with code 0x%x\n",
                result ? "failed" : "successful", result));

        if (result) {
            /*
             * not-ready to ready transition [asc/ascq=0x28/0x0]
             * or power-on, reset [asc/ascq=0x29/0x0], continue.
             * INQUIRY should not yield UNIT_ATTENTION
             * but many buggy devices do so anyway.
             */
            if ((driver_byte(result) & DRIVER_SENSE) &&
                scsi_sense_valid(&sshdr)) {
                if ((sshdr.sense_key == UNIT_ATTENTION) &&
                    ((sshdr.asc == 0x28) ||
                     (sshdr.asc == 0x29)) &&
                    (sshdr.ascq == 0))
                    continue;
            }
        } else {
            /*
             * if nothing was transferred, we try
             * again. It's a workaround for some USB
             * devices.
             */
            if (resid == try_inquiry_len)
                continue;
        }
        break;
    }

    if (result == 0) {
        sanitize_inquiry_string(&inq_result[8], 8);
        sanitize_inquiry_string(&inq_result[16], 16);
        sanitize_inquiry_string(&inq_result[32], 4);

        response_len = inq_result[4] + 5;
        if (response_len > 255)
            response_len = first_inquiry_len;   /* sanity */

        /*
         * Get any flags for this device.
         *
         * XXX add a bflags to scsi_device, and replace the
         * corresponding bit fields in scsi_device, so bflags
         * need not be passed as an argument.
         */
        *bflags = scsi_get_device_flags(sdev, &inq_result[8],
                &inq_result[16]);

        /* When the first pass succeeds we gain information about
         * what larger transfer lengths might work. */
        if (pass == 1) {
            if (BLIST_INQUIRY_36 & *bflags)
                next_inquiry_len = 36;
            else if (BLIST_INQUIRY_58 & *bflags)
                next_inquiry_len = 58;
            else if (sdev->inquiry_len)
                next_inquiry_len = sdev->inquiry_len;
            else
                next_inquiry_len = response_len;

            /* If more data is available perform the second pass */
            if (next_inquiry_len > try_inquiry_len) {
                try_inquiry_len = next_inquiry_len;
                pass = 2;
                goto next_pass;
            }
        }

    } else if (pass == 2) {
        sdev_printk(KERN_INFO, sdev,
                "scsi scan: %d byte inquiry failed.  "
                "Consider BLIST_INQUIRY_36 for this device\n",
                try_inquiry_len);

        /* If this pass failed, the third pass goes back and transfers
         * the same amount as we successfully got in the first pass. */
        try_inquiry_len = first_inquiry_len;
        pass = 3;
        goto next_pass;
    }

    /* If the last transfer attempt got an error, assume the
     * peripheral doesn't exist or is dead. */
    if (result)
        return -EIO;

    /* Don't report any more data than the device says is valid */
    sdev->inquiry_len = min(try_inquiry_len, response_len);

    /*
     * XXX Abort if the response length is less than 36? If less than
     * 32, the lookup of the device flags (above) could be invalid,
     * and it would be possible to take an incorrect action - we do
     * not want to hang because of a short INQUIRY. On the flip side,
     * if the device is spun down or becoming ready (and so it gives a
     * short INQUIRY), an abort here prevents any further use of the
     * device, including spin up.
     *
     * On the whole, the best approach seems to be to assume the first
     * 36 bytes are valid no matter what the device says.  That's
     * better than copying < 36 bytes to the inquiry-result buffer
     * and displaying garbage for the Vendor, Product, or Revision
     * strings.
     */
    if (sdev->inquiry_len < 36) {
        if (!sdev->host->short_inquiry) {
            shost_printk(KERN_INFO, sdev->host,
                    "scsi scan: INQUIRY result too short (%d),"
                    " using 36\n", sdev->inquiry_len);
            sdev->host->short_inquiry = 1;
        }
        sdev->inquiry_len = 36;
    }

    /*
     * Related to the above issue:
     *
     * XXX Devices (disk or all?) should be sent a TEST UNIT READY,
     * and if not ready, sent a START_STOP to start (maybe spin up) and
     * then send the INQUIRY again, since the INQUIRY can change after
     * a device is initialized.
     *
     * Ideally, start a device if explicitly asked to do so.  This
     * assumes that a device is spun up on power on, spun down on
     * request, and then spun up on request.
     */

    /*
     * The scanning code needs to know the scsi_level, even if no
     * device is attached at LUN 0 (SCSI_SCAN_TARGET_PRESENT) so
     * non-zero LUNs can be scanned.
     */
    sdev->scsi_level = inq_result[2] & 0x07;
    if (sdev->scsi_level >= 2 ||
        (sdev->scsi_level == 1 && (inq_result[3] & 0x0f) == 1))
        sdev->scsi_level++;
    sdev->sdev_target->scsi_level = sdev->scsi_level;

    /*
     * If SCSI-2 or lower, and if the transport requires it,
     * store the LUN value in CDB[1].
     */
    sdev->lun_in_cdb = 0;
    if (sdev->scsi_level <= SCSI_2 &&
        sdev->scsi_level != SCSI_UNKNOWN &&
        !sdev->host->no_scsi2_lun_in_cdb)
        sdev->lun_in_cdb = 1;

    return 0;
}


static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
        int *bflags, int async)
{
    int ret;

    /*
     * XXX do not save the inquiry, since it can change underneath us,
     * save just vendor/model/rev.
     *
     * Rather than save it and have an ioctl that retrieves the saved
     * value, have an ioctl that executes the same INQUIRY code used
     * in scsi_probe_lun, let user level programs doing INQUIRY
     * scanning run at their own risk, or supply a user level program
     * that can correctly scan.
     */

    /*
     * Copy at least 36 bytes of INQUIRY data, so that we don't
     * dereference unallocated memory when accessing the Vendor,
     * Product, and Revision strings.  Badly behaved devices may set
     * the INQUIRY Additional Length byte to a small value, indicating
     * these strings are invalid, but often they contain plausible data
     * nonetheless.  It doesn't matter if the device sent < 36 bytes
     * total, since scsi_probe_lun() initializes inq_result with 0s.
     */
    sdev->inquiry = kmemdup(inq_result,
                max_t(size_t, sdev->inquiry_len, 36),
                GFP_ATOMIC);
    if (sdev->inquiry == NULL)
        return SCSI_SCAN_NO_RESPONSE;

    sdev->vendor = (char *) (sdev->inquiry + 8); //The first8Bytes to15Bytes are vendor identification
    sdev->model = (char *) (sdev->inquiry + 16); //The first16Bytes to31Bytes are product identification
    sdev->rev = (char *) (sdev->inquiry + 32);   //The first32Bytes to35Bytes are product revision level

    if (strncmp(sdev->vendor, "ATA     ", 8) == 0) {
        /*
         * sata emulation layer device.  This is a hack to work around
         * the SATL power management specifications which state that
         * when the SATL detects the device has gone into standby
         * mode, it shall respond with NOT READY.
         */
        sdev->allow_restart = 1;
    }

    if (*bflags & BLIST_ISROM) {
        sdev->type = TYPE_ROM;
        sdev->removable = 1;
    } else {
        sdev->type = (inq_result[0] & 0x1f);
        sdev->removable = (inq_result[1] & 0x80) >> 7;

        /*
         * some devices may respond with wrong type for
         * well-known logical units. Force well-known type
         * to enumerate them correctly.
         */
        if (scsi_is_wlun(sdev->lun) && sdev->type != TYPE_WLUN) {
            sdev_printk(KERN_WARNING, sdev,
                "%s: correcting incorrect peripheral device type 0x%x for W-LUN 0x%16xhN\n",
                __func__, sdev->type, (unsigned int)sdev->lun);
            sdev->type = TYPE_WLUN;
        }

    }

    if (sdev->type == TYPE_RBC || sdev->type == TYPE_ROM) {
        /* RBC and MMC devices can return SCSI-3 compliance and yet
         * still not support REPORT LUNS, so make them act as
         * BLIST_NOREPORTLUN unless BLIST_REPORTLUN2 is
         * specifically set */
        if ((*bflags & BLIST_REPORTLUN2) == 0)
            *bflags |= BLIST_NOREPORTLUN;
    }

    /*
     * For a peripheral qualifier (PQ) value of 1 (001b), the SCSI
     * spec says: The device server is capable of supporting the
     * specified peripheral device type on this logical unit. However,
     * the physical device is not currently connected to this logical
     * unit.
     *
     * The above is vague, as it implies that we could treat 001 and
     * 011 the same. Stay compatible with previous code, and create a
     * scsi_device for a PQ of 1
     *
     * Don't set the device offline here; rather let the upper
     * level drivers eval the PQ to decide whether they should
     * attach. So remove ((inq_result[0] >> 5) & 7) == 1 check.
     */

    sdev->inq_periph_qual = (inq_result[0] >> 5) & 7;
    sdev->lockable = sdev->removable;
    sdev->soft_reset = (inq_result[7] & 1) && ((inq_result[3] & 7) == 2);

    if (sdev->scsi_level >= SCSI_3 ||
            (sdev->inquiry_len > 56 && inq_result[56] & 0x04))
        sdev->ppr = 1;
    if (inq_result[7] & 0x60)
        sdev->wdtr = 1;
    if (inq_result[7] & 0x10)
        sdev->sdtr = 1;

    sdev_printk(KERN_NOTICE, sdev, "%s %.8s %.16s %.4s PQ: %d "
            "ANSI: %d%s\n", scsi_device_type(sdev->type),
            sdev->vendor, sdev->model, sdev->rev,
            sdev->inq_periph_qual, inq_result[2] & 0x07,
            (inq_result[3] & 0x0f) == 1 ? " CCS" : "");

    if ((sdev->scsi_level >= SCSI_2) && (inq_result[7] & 2) &&
        !(*bflags & BLIST_NOTQ)) {
        sdev->tagged_supported = 1;
        sdev->simple_tags = 1;
    }

    /*
     * Some devices (Texel CD ROM drives) have handshaking problems
     * when used with the Seagate controllers. borken is initialized
     * to 1, and then set it to 0 here.
     */
    if ((*bflags & BLIST_BORKEN) == 0)
        sdev->borken = 0;

    if (*bflags & BLIST_NO_ULD_ATTACH)
        sdev->no_uld_attach = 1;

    /*
     * Apparently some really broken devices (contrary to the SCSI
     * standards) need to be selected without asserting ATN
     */
    if (*bflags & BLIST_SELECT_NO_ATN)
        sdev->select_no_atn = 1;

    /*
     * Maximum 512 sector transfer length
     * broken RA4x00 Compaq Disk Array
     */
    if (*bflags & BLIST_MAX_512)
        blk_queue_max_hw_sectors(sdev->request_queue, 512);
    /*
     * Max 1024 sector transfer length for targets that report incorrect
     * max/optimal lengths and relied on the old block layer safe default
     */
    else if (*bflags & BLIST_MAX_1024)
        blk_queue_max_hw_sectors(sdev->request_queue, 1024);

    /*
     * Some devices may not want to have a start command automatically
     * issued when a device is added.
     */
    if (*bflags & BLIST_NOSTARTONADD)
        sdev->no_start_on_add = 1;

    if (*bflags & BLIST_SINGLELUN)
        scsi_target(sdev)->single_lun = 1;

    sdev->use_10_for_rw = 1;

    if (*bflags & BLIST_MS_SKIP_PAGE_08)
        sdev->skip_ms_page_8 = 1;

    if (*bflags & BLIST_MS_SKIP_PAGE_3F)
        sdev->skip_ms_page_3f = 1;

    if (*bflags & BLIST_USE_10_BYTE_MS)
        sdev->use_10_for_ms = 1;

    /* some devices don't like REPORT SUPPORTED OPERATION CODES
     * and will simply timeout causing sd_mod init to take a very
     * very long time */
    if (*bflags & BLIST_NO_RSOC)
        sdev->no_report_opcodes = 1;

    /* set the device running here so that slave configure
     * may do I/O */
    ret = scsi_device_set_state(sdev, SDEV_RUNNING); //state
    if (ret) {
        ret = scsi_device_set_state(sdev, SDEV_BLOCK);

        if (ret) {
            sdev_printk(KERN_ERR, sdev,
                    "in wrong state %s to complete scan\n",
                    scsi_device_state_name(sdev->sdev_state));
            return SCSI_SCAN_NO_RESPONSE;
        }
    }

    if (*bflags & BLIST_MS_192_BYTES_FOR_3F)
        sdev->use_192_bytes_for_3f = 1;

    if (*bflags & BLIST_NOT_LOCKABLE)
        sdev->lockable = 0;

    if (*bflags & BLIST_RETRY_HWERROR)
        sdev->retry_hwerror = 1;

    if (*bflags & BLIST_NO_DIF)
        sdev->no_dif = 1;

    sdev->eh_timeout = SCSI_DEFAULT_EH_TIMEOUT;

    if (*bflags & BLIST_TRY_VPD_PAGES)
        sdev->try_vpd_pages = 1;
    else if (*bflags & BLIST_SKIP_VPD_PAGES)
        sdev->skip_vpd_pages = 1;

    transport_configure_device(&sdev->sdev_gendev); //Configuring lun to the scsi transport layer

    if (sdev->host->hostt->slave_configure) {
        ret = sdev->host->hostt->slave_configure(sdev); //The callback set by the host adapter template to perform specific initialization on the SCSI device (Lun)
        if (ret) {
            /*
             * if LLDD reports slave not present, don't clutter
             * console with alloc failure messages
             */
            if (ret != -ENXIO) {
                sdev_printk(KERN_ERR, sdev,
                    "failed to configure device\n");
            }
            return SCSI_SCAN_NO_RESPONSE;
        }
    }

    if (sdev->scsi_level >= SCSI_3)
        scsi_attach_vpd(sdev);

    sdev->max_queue_depth = sdev->queue_depth;  //Set maximum queue depth

    /*
     * Ok, the device is now all set up, we can
     * register it and tell the rest of the kernel
     * about it.
     */ //Add SCSI device (Lun) to sysfs
    if (!async && scsi_sysfs_add_sdev(sdev) != 0)
        return SCSI_SCAN_NO_RESPONSE;

    return SCSI_SCAN_LUN_PRESENT;
}

            <link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-b6c3c6d139.css" rel="stylesheet">
                                            <div class="more-toolbox">
            <div class="left-toolbox">
                <ul class="toolbox-list">
                    
                    <li class="tool-item tool-active is-like "><a href="javascript:;"><svg class="icon" aria-hidden="true">
                        <use xlink:href="#csdnc-thumbsup"></use>
                    < / SVG > < span class = "name" > like</span>
                    <span class="count">9</span>
                    </a></li>
                    <li class="tool-item tool-active is-collection "><a href="javascript:;" data-report-click="{&quot;mod&quot;:&quot;popu_824&quot;}"><svg class="icon" aria-hidden="true">
                        <use xlink:href="#icon-csdnc-Collection-G"></use>
                    < / SVG > < span class = "name" > collection < / span ></a></li>
                    <li class="tool-item tool-active is-share"><a href="javascript:;"><svg class="icon" aria-hidden="true">
                        <use xlink:href="#icon-csdnc-fenxiang"></use>
                    Share</a></li>
                    <! -- reward start -- >
                                            <! -- reward end -- >
                                            <li class="tool-item tool-more">
                        <a>
                        <svg t="1575545411852" class="icon" viewBox="0 0 1024 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="5717" xmlns:xlink="http://www.w3.org/1999/xlink" width="200" height="200"><defs><style type="text/css"></style></defs><path d="M179.176 499.222m-113.245 0a113.245 113.245 0 1 0 226.49 0 113.245 113.245 0 1 0-226.49 0Z" p-id="5718"></path><path d="M509.684 499.222m-113.245 0a113.245 113.245 0 1 0 226.49 0 113.245 113.245 0 1 0-226.49 0Z" p-id="5719"></path><path d="M846.175 499.222m-113.245 0a113.245 113.245 0 1 0 226.49 0 113.245 113.245 0 1 0-226.49 0Z" p-id="5720"></path></svg>
                        </a>
                        <ul class="more-box">
                            < Li class = "item" ></a></li>
                        </ul>
                    </li>
                                        </ul>
            </div>
                        </div>
        <div class="person-messagebox">
            <div class="left-message"><a href="https://blog.csdn.net/haleycomet">
                <img src="https://profile.csdnimg.cn/2/2/C/3_haleycomet" class="avatar_pic" username="haleycomet">
                                        <img src="https://g.csdnimg.cn/static/user-reg-year/1x/4.png" class="user-years">
                                </a></div>
            <div class="middle-message">
                                    <div class="title"><span class="tit"><a href="https://blog.csdn.net/haleycomet" data-report-click="{&quot;mod&quot;:&quot;popu_379&quot;}" target="_blank">haleycomet</a></span>
                                        </div>
                < div class = "text" >
            </div>
                            <div class="right-message">
                                        < a href = "https://im.csdn.net/im/main.html? Username = haleycomet" target = "blank" class = "BTN BTN SM BTN red hole BT button personal letter" > private message
                    </a>
                                                        < a class = "BTN BTN SM attached BT button personal watch" data report Click = "{& quot; mod & quot;: & quot; pop & quot;}" ></a>
                                </div>
                        </div>
                </div>

I'll take my life, laa

Published 12 original articles, praised 0, visited 1958

Private letter follow

Posted by networkguy on Sun, 09 Feb 2020 20:20:54 -0800

Programmer Group