Skip to main content

Using ESXCLI to Troubleshoot Multipathing and PSA-related Issues


This post will look at the ‘esxcli storage‘ namespace, with a focus on troubleshooting

Listing Device and Path Information

The first thing to do when troubleshooting storage using esxcli is to list the attached storage devices:
esxcli storage core device list
This will list all attached storage devices, example output shown below:
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
You can narrow down the output by specifying the storage device:
~ # esxcli storage core device list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Has Settable Display Name: true
   Size: 5088
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Vendor: OPNFILER
   Model: VIRTUAL-DISK
   Revision: 0
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: unknown
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0100000000316c526565782d6a7a65332d39787735564952545541
It’s clear from the output that this device is using VMware’s native multipathing plugin:
Multipath Plugin: NMP
To view a host’s paths to the storage device you can use the ‘esxcli storage core path list’ command. Use the ‘-d’ switch and specify the device:
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000002 PortalTag=1

iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C0:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 0
   Target: 0
   LUN: 0
   Plugin: NMP
   State: active
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000001,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Transport Details: IQN=iqn.2006-01.com.openfiler:tsn.a75226350df7 Alias= Session=00023d000001 PortalTag=1
The output shows that there are two paths to this storage device and that both are active. This matches what is seen in the vSphere client for this device:
storage-paths
If a path was down, you should see the ‘dead’ state (though I’ve shortened the output here):
~ # esxcli storage core path list -d t10.F405E46494C4540013C625565687D2A6A75633D293877753
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
To investigate this dead path, we can run a command to output more details on this path by referencing the runtime name with the ‘-p’ switch:
~ # esxcli storage core path list -p vmhba33:C1:T0:L0
iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   UID: iqn.1998-01.com.vmware:esxi1-47be1f31-00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1-t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Runtime Name: vmhba33:C1:T0:L0
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device Display Name: OPNFILER iSCSI Disk (t10.F405E46494C4540013C625565687D2A6A75633D293877753)
   Adapter: vmhba33
   Channel: 1
   Target: 0
   LUN: 0
   Plugin: NMP
   State: dead
   Transport: iscsi
   Adapter Identifier: iqn.1998-01.com.vmware:esxi1-47be1f31
   Target Identifier: 00023d000002,iqn.2006-01.com.openfiler:tsn.a75226350df7,t,1
   Adapter Transport Details: Unavailable or path is unclaimed
   Target Transport Details: Unavailable or path is unclaimed
Again, we can see that the path State is ‘dead’. Next steps would be to check other hosts to see if they can see the storage device in order to help determine root cause.

Looking at Storage Related Statistics using ESXCLI

Along with using tools such as esxtop and vscsistats, you can also use ESXCLI to view storage statistics. To do so, use the following command:
~ # esxcli storage core device stats get
This will produce output for each attached storage device such as that shown below
t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Device: t10.F405E46494C4540013C625565687D2A6A75633D293877753
   Successful Commands: 1010
   Blocks Read: 67936
   Blocks Written: 58
   Read Operations: 711
   Write Operations: 58
   Reserve Operations: 15
   Reservation Conflicts: 0
   Failed Commands: 50
   Failed Blocks Read: 0
   Failed Blocks Written: 0
   Failed Read Operations: 0
   Failed Write Operations: 0
   Failed Reserve Operations: 0
Pay attention to any of the ‘Failed’ metrics. Any number other than zero could indicate a problem with the device, or connectivity to it.

Troubleshooting Storage Claimrules

Sometimes when troubleshooting storage issues it will be necessary to modify or add claimrules for a device. I have written about this in detail here.

ESXCLI Storage Command Examples

Operation Command Information
List the SCSI stats for the SCSI Paths in the system for FC/iSCSI LUNs esxcli storage core path stats get Displays the quantity of read and write operations and failures
Lists all the SCSI paths for FC/iSCSI LUNs esxcli storage core path list Displays each path – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
Only get path information for a single device esxcli storage core path list -d deviceID Displays path information for specified device – Includes the CTL, Adapter, Device, transport mode, state, and plugin type/details
List the adapters in the system esxcli storage core adapter list Displays the HBA, driver, state, UID, and Description of each installed adapter
Turn off a path esxcli storage core path set –state off –path vmhba??:C?:T?:L?  
Activate a path esxcli storage core path set –state active –pathvmhba??:C?:T?:L?  
List devices claimed by Native Multipath Plugin esxcli storage nmp device list Displays Device, Array Type/Config, and Path Selection Policy information
List the available Storage Array Type Plugins esxcli storage nmp satp list Displays Name, Default Path Selection Policy and Description for each SATP
Lists the available Path Selection Policies esxcli storage nmp psp list Displays the name and description of all path selection policies
Lists the paths claimed by the NMP and shows the SATP & PSP information associated with the path esxcli storage nmp path list Displays the NMP claimed path SATP and PSP information

Comments

Post a Comment

Popular posts from this blog

  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more

57 Tips Every Admin Should Know

Active Directory 1. To quickly list all the groups in your domain, with members, run this command: dsquery group -limit 0 | dsget group -members –expand 2. To find all users whose accounts are set to have a non-expiring password, run this command: dsquery * domainroot -filter “(&(objectcategory=person)(objectclass=user)(lockoutTime=*))” -limit 0 3. To list all the FSMO role holders in your forest, run this command: netdom query fsmo 4. To refresh group policy settings, run this command: gpupdate 5. To check Active Directory replication on a domain controller, run this command: repadmin /replsummary 6. To force replication from a domain controller without having to go through to Active Directory Sites and Services, run this command: repadmin /syncall 7. To see what server authenticated you (or if you logged on with cached credentials) you can run either of these commands: set l echo %logonserver% 8. To see what account you are logged on as, run this command: ...
Post a Comment
Read more
  The Guardrails of Automation VMware Cloud Foundation (VCF) 9.0 has redefined private cloud automation. With full-stack automation powered by Ansible and orchestrated through vRealize Orchestrator (vRO), and version-controlled deployments driven by GitOps and CI/CD pipelines, teams can build infrastructure faster than ever. But automation without guardrails is a recipe for risk Enter RBAC and policy enforcement. This third and final installment in our automation series focuses on how to secure and govern multi-tenant environments in VCF 9.0 with role-based access control (RBAC) and layered identity management. VCF’s IAM Foundation VCF 9.x integrates tightly with enterprise identity providers, enabling organizations to define and assign roles using existing Active Directory (AD) groups. With its persona-based access model, administrators can enforce strict boundaries across compute, storage, and networking resources: Personas : Global Admin, Tenant Admin, Contributor, Viewer Projec...
Post a Comment
Read more