Skip to main content

 

Troubleshooting VMware Aria Automation 8.x (formerly vRealize Automation 8.x) upgrades


Issue/Introduction

  • This article provides steps to manually collect upgrade logs if the log bundle collection fails when a VMware Aria Automation upgrade fails.
  • When an upgrade fails, the following logs need to be analyzed:
/var/log/vmware/prelude/upgrade-*.logUpgrade reports. Review based
on timestamp

One node environment
/opt/vmware/var/log/vami/*.log

Cluster environments
/opt/log/vmware/var/log/vami/*.log

Packages installation details
/var/log/bootstrap/postupdate.logInitialization scripts details
/var/log/bootstrap/everyboot.logInitialization scripts details
/var/log/vmware/prelude/deploy-*.logServices startup details


Note: Some log files will have timestamps as part of the file name, for others the information is appended. It is important to validate that the information is from the latest upgrade attempt.

Overview

Upgrade Prerequisites

  1. Read the product release notes.
  2. Check Hardware requirements.
  3. Check the Services status.
  4. Ensure backups are available.
  5. Ensure a pre-upgrade snapshot is taken.

Trigger upgrade in Aria Suite Lifecycle

  1. Validate prechecks are successful.  

Upgrade Process Breakdown

  • Manifest is downloaded (upgrade-noop.log)  
  • Searches for the bootstap package and checks if ssh port is open and able to connect (update-datetime.log) 
  • Retrieves product version on all nodes
  • Performs Infrastructure health
  • Shuts down Infrastructure and Application services
  • Saves local restore point for k8s processes node data and activates monitor on all nodes
  • Once upgrade monitor is activated and the cluster nodes are removed successfully, it will proceed for installation.  This takes about 30 minutes. 
  • VAMI upgrade starts
  • Download of all packages happens (vami.log)
  • Once down, installation of packages starts (updatecli.log & postupdate.log)
  • Once completed, Appliance is rebooted and VAMI Upgrade is marked successful (update-datetime.log)
  • Cluster nodes are added back in and restore points are restored
  • Infrastructure and Application Services are started
  • Upgrade cleanup performed



Environment

VMware vRealize Automation 8.x
VMware Aria Automation 8.x

Resolution

Manually collecting upgrade failure diagnostic information

  1. SSH to the VMware Aria Automation node indicated in the Aria Suite Lifecycle error.

  2. Validate that there is available disk space in the root partition (/dev/sda4) running the command vracli disk-mgr

    root@vranode1 [ /tmp ]# vracli disk-mgr
/dev/sda4(/):
        Total size: 47.80GiB
        Free: 33.58GiB(70.2%)
        Available(for non-superusers): 31.13GiB(65.1%)
        SCSI ID: (0:0)
/dev/sdb(/data):
        Total size: 140.68GiB
        Free: 109.54GiB(77.9%)
        Available(for non-superusers): 102.32GiB(72.7%)
        SCSI ID: (0:1)
/dev/sdc(/var/log):
        Total size: 21.48GiB
        Free: 9.09GiB(42.3%)
        Available(for non-superusers): 7.97GiB(37.1%)
        SCSI ID: (0:2)
/dev/sdd(/home):
        Total size: 29.36GiB
        Free: 27.41GiB(93.4%)
        Available(for non-superusers): 25.90GiB(88.2%)
        SCSI ID: (0:3)

  3. Run the following command to collect the directories and logs related to the upgrade:

    mkdir /tmp/upgradelogs && cp -R /var/log/vmware/prelude /tmp/upgradelogs && cp -R /opt/vmware/var/log/vami /tmp/upgradelogs && cp -R /var/log/bootstrap /tmp/upgradelogs && tar -zcvf /tmp/upgradelogs.tar.gz /tmp/upgradelogs

  4. Extract and continue the review with the collected file /tmp/upgradelogs.tar.gz  for the failure code or submit this data to Global Services for additional assistance in troubleshooting the upgrade.

  5. After extracting the file from the appliance, remove the file and directory to save disk space:
cd / tmp
rm upgradelogs.tar.gz
rm -r upgradelogs

Additional Information

Labels:

Comments

Post a Comment

Popular posts from this blog

Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 ðŸ”¹ Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware ⸻ 1️⃣ Memory Ballooning (vmmemctl) Ballooning is the first memory reclamation technique used when ESXi detects memory pressure. ➤ Step-by-Step: How Ballooning Works  1. VMware Tools installs the balloon driver (vmmemctl) inside the guest OS.  2. ESXi detects low free memory on the host.  3. ESXi inflates the balloon in selected VMs.  4. Balloon driver occupies guest memory, making the OS think RAM is full.  5. Guest OS frees idle / unused pages (because it believes memory is needed).  6. ESXi reclaims those freed pages and makes them available to other VMs. Why Ballooning Happens?  • Host free memory is very low.  • ESXi wants the VM to release unused pages before resorting to swapping. Example  • Host memory: 64 GB  • VMs used: 62 GB  • Free: 2 GB → ESXi triggers ballooning  • VM1 (8 GB RAM): Balloon inflates to 2 GB → OS frees 2 GB → ESXi re...
Post a Comment
Read more

ESXi Host Troubleshooting Checklist

  🛠️ ESXi Host Troubleshooting Checklist (With Complete Log Locations) ✅ 1. Host Status & Connectivity Check host state in vCenter (Connected / Not Responding / Disconnected) Review CPU, RAM, and datastore usage Validate HA/DRS recommendations ✅ 2. Hardware Health Monitor hardware sensors (CPU, DIMMs, fans, PSU, RAID) Check RAID controller logs via vendor tools View hardware status in: vCenter → Monitor → Hardware Health DCUI → Hardware Status ✅ 3. Network Validation Verify vSwitches, Port Groups, NIC teaming, VLAN tagging Check vmkernel ports (mgmt, vMotion, iSCSI, vSAN) Test reachability using: Shell vmkping < IP > Show more lines Validate physical NIC status and link speed ✅ 4. Storage & Datastore Checks Confirm datastore accessibility Rescan storage adapters (iSCSI/FC/NFS) Validate multipathing (Round Robin / Fixed / MRU) Check datastore latency with esxtop ✅ **5. Critical Logs & Their Locations (Full List) Here are key ESXi log files that every VMware adm...
Post a Comment
Read more
  vCenter Troubleshooting Tips, Common Issues & Log Locations 🚀 1. General Troubleshooting Approach A. Check service health first For vCenter Server Appliance (VCSA): Shell https : / / < vcenter - FQDN > : 5480 Show more lines Go to Services → Health to verify: vCenter Server ESXi hosts vSphere Client DNS, NTP, DB connections vpxd, vpxd-svcs, vsphere-ui B. DNS & NTP checks vCenter is highly dependent on correct DNS & time sync. Check: Shell nslookup < vcenter - fqdn > nslookup < esxi - host - fqdn > Show more lines Time drift > 5 minutes causes: Login failures vpxd crashes PSC authentication issues C. Check storage/database health Slow DB/storage = UI slow, tasks hung. For VCSA: Shell df - h du - sh / storage / log / * / var / log Show more lines Free up space if partitions hit >80%. D. Restart critical services safely On VCSA: Shell service - control - - status service - control - - stop - - all service - control - - start - - all ...
Post a Comment
Read more