Skip to main content

Using ESXCLI to Troubleshoot Multipathing and PSA-related Issues


This post will look at the ‘esxcli storage‘ namespace, with a focus on troubleshooting

Listing Device and Path Information

The first thing to do when troubleshooting storage using esxcli is to list the attached storage devices:
esxcli storage core device list
This will list all attached storage devices, example output shown below:
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
You can narrow down the output by specifying the storage device:
~ # esxcli storage core device list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
It’s clear from the output that this device is using VMware’s native multipathing plugin:
Multipath Plugin: NMP
To view a host’s paths to the storage device you can use the ‘esxcli storage core path list’ command. Use the ‘-d’ switch and specify the device:
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000002 PortalTag=1

iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C0:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000001 PortalTag=1
The output shows that there are two paths to this storage device and that both are active. This matches what is seen in the vSphere client for this device:
storage-paths
If a path was down, you should see the ‘dead’ state (though I’ve shortened the output here):
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
To investigate this dead path, we can run a command to output more details on this path by referencing the runtime name with the ‘-p’ switch:
~ # esxcli storage core path list -p vmhba33:C1:T0:L0
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
Again, we can see that the path State is ‘dead’. Next steps would be to check other hosts to see if they can see the storage device in order to help determine root cause.

Looking at Storage Related Statistics using ESXCLI

Along with using tools such as esxtop and vscsistats, you can also use ESXCLI to view storage statistics. To do so, use the following command:
~ # esxcli storage core device stats get
This will produce output for each attached storage device such as that shown below
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Successful Commands: 1010
   Blocks Read: 67936
   Blocks Written: 58
   Read Operations: 711
   Write Operations: 58
   Reserve Operations: 15
   Reservation Conflicts: 0
   Failed Commands: 50
   Failed Blocks Read: 0
   Failed Blocks Written: 0
   Failed Read Operations: 0
   Failed Write Operations: 0
   Failed Reserve Operations: 0
Pay attention to any of the ‘Failed’ metrics. Any number other than zero could indicate a problem with the device, or connectivity to it.

Troubleshooting Storage Claimrules

Sometimes when troubleshooting storage issues it will be necessary to modify or add claimrules for a device. I have written about this in detail here.

ESXCLI Storage Command Examples

Operation Command Information
List the SCSI stats for the SCSI Paths in the system for FC/iSCSI LUNs esxcli storage core path stats get Displays the quantity of read and write operations and failures
Lists all the SCSI paths for FC/iSCSI LUNs esxcli storage core path list Displays each path – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
Only get path information for a single device esxcli storage core path list -d deviceID Displays path information for specified device – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
List the adapters in the system esxcli storage core adapter list Displays the HBA, driver, state, UID, and Description of each installed adapter
Turn off a path esxcli storage core path set –state off –path vmhba??:C?:T?:L?  
Activate a path esxcli storage core path set –state active –pathvmhba??:C?:T?:L?  
List devices claimed by Native Multipath Plugin esxcli storage nmp device list Displays Device, Array Type/Config, and Path Selection Policy information
List the available Storage Array Type Plugins esxcli storage nmp satp list Displays Name, Default Path Selection Policy and Description for each SATP
Lists the available Path Selection Policies esxcli storage nmp psp list Displays the name and description of all path selection policies
Lists the paths claimed by the NMP and shows the SATP & PSP information associated with the path esxcli storage nmp path list Displays the NMP claimed path SATP and PSP information

Comments

Post a Comment

Popular posts from this blog

Quick Guide to VCF Automation for VCD Administrators

  Quick Guide to VCF Automation for VCD Administrators VMware Cloud Foundation 9 (VCF 9) has been  released  and with it comes brand new Cloud Management Platform –  VCF Automation (VCFA)  which supercedes both Aria Automation and VMware Cloud Director (VCD). This blog post is intended for those people that know VCD quite well and want to understand how is VCFA similar or different to help them quickly orient in the new direction. It should be emphasized that VCFA is a new solution and not just rebranding of an old one. However it reuses a lot of components from its predecessors. The provider part of VCFA called Tenenat Manager is based on VCD code and the UI and APIs will be familiar to VCD admins, while the tenant part inherist a lot from Aria Automation and especially for VCD end-users will look brand new. Deployment and Architecture VCFA is generaly deployed from VCF Operations Fleet Management (former Aria Suite LCM embeded in VCF Ops. Fleet Management...
Post a Comment
Read more
  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more
  "Cloud zone insights not available yet, please check after some time" message on Aria Automation https://knowledge.broadcom.com/external/article?articleNumber=314894 Products VMware Aria Suite Issue/Introduction Symptoms: The certificate for Aria operations has been replaced since it was initially added to Aria Automation as an integration. When accessing the Insights pane under  Cloud Assembly  ->  Infrastructure  ->  Cloud Zone  ->  Insights  the following message is displayed:   "Cloud zone insights not available yet, please check after some time." The  /var/log/services-logs/prelude/hcmp-service-app/file-logs/hcmp-service-app.log  file contains ssl errors similar to:   2022-08-25T20:06:43.989Z ERROR hcmp-service [host='hcmp-service-app-xxxxxxx-xxxx' thread='Thread-56' user='' org='<org_id>' trace='<trace_id>' parent='<parent_id>' span='<span_id>'] c.v.a.h.a.common.AlertEnu...
Post a Comment
Read more