Skip to main content

 

Troubleshooting VMware Aria Automation 8.x (formerly vRealize Automation 8.x) upgrades


Issue/Introduction

  • This article provides steps to manually collect upgrade logs if the log bundle collection fails when a VMware Aria Automation upgrade fails.
  • When an upgrade fails, the following logs need to be analyzed:
/var/log/vmware/prelude/upgrade-*.logUpgrade reports. Review based
on timestamp

One node environment
/opt/vmware/var/log/vami/*.log

Cluster environments
/opt/log/vmware/var/log/vami/*.log

Packages installation details
/var/log/bootstrap/postupdate.logInitialization scripts details
/var/log/bootstrap/everyboot.logInitialization scripts details
/var/log/vmware/prelude/deploy-*.logServices startup details


Note: Some log files will have timestamps as part of the file name, for others the information is appended. It is important to validate that the information is from the latest upgrade attempt.

Overview

Upgrade Prerequisites

  1. Read the product release notes.
  2. Check Hardware requirements.
  3. Check the Services status.
  4. Ensure backups are available.
  5. Ensure a pre-upgrade snapshot is taken.

Trigger upgrade in Aria Suite Lifecycle

  1. Validate prechecks are successful.  

Upgrade Process Breakdown

  • Manifest is downloaded (upgrade-noop.log)  
  • Searches for the bootstap package and checks if ssh port is open and able to connect (update-datetime.log) 
  • Retrieves product version on all nodes
  • Performs Infrastructure health
  • Shuts down Infrastructure and Application services
  • Saves local restore point for k8s processes node data and activates monitor on all nodes
  • Once upgrade monitor is activated and the cluster nodes are removed successfully, it will proceed for installation.  This takes about 30 minutes. 
  • VAMI upgrade starts
  • Download of all packages happens (vami.log)
  • Once down, installation of packages starts (updatecli.log & postupdate.log)
  • Once completed, Appliance is rebooted and VAMI Upgrade is marked successful (update-datetime.log)
  • Cluster nodes are added back in and restore points are restored
  • Infrastructure and Application Services are started
  • Upgrade cleanup performed



Environment

VMware vRealize Automation 8.x
VMware Aria Automation 8.x

Resolution

Manually collecting upgrade failure diagnostic information

  1. SSH to the VMware Aria Automation node indicated in the Aria Suite Lifecycle error.

  2. Validate that there is available disk space in the root partition (/dev/sda4) running the command vracli disk-mgr

    root@vranode1 [ /tmp ]# vracli disk-mgr
/dev/sda4(/):
        Total size: 47.80GiB
        Free: 33.58GiB(70.2%)
        Available(for non-superusers): 31.13GiB(65.1%)
        SCSI ID: (0:0)
/dev/sdb(/data):
        Total size: 140.68GiB
        Free: 109.54GiB(77.9%)
        Available(for non-superusers): 102.32GiB(72.7%)
        SCSI ID: (0:1)
/dev/sdc(/var/log):
        Total size: 21.48GiB
        Free: 9.09GiB(42.3%)
        Available(for non-superusers): 7.97GiB(37.1%)
        SCSI ID: (0:2)
/dev/sdd(/home):
        Total size: 29.36GiB
        Free: 27.41GiB(93.4%)
        Available(for non-superusers): 25.90GiB(88.2%)
        SCSI ID: (0:3)

  3. Run the following command to collect the directories and logs related to the upgrade:

    mkdir /tmp/upgradelogs && cp -R /var/log/vmware/prelude /tmp/upgradelogs && cp -R /opt/vmware/var/log/vami /tmp/upgradelogs && cp -R /var/log/bootstrap /tmp/upgradelogs && tar -zcvf /tmp/upgradelogs.tar.gz /tmp/upgradelogs

  4. Extract and continue the review with the collected file /tmp/upgradelogs.tar.gz  for the failure code or submit this data to Global Services for additional assistance in troubleshooting the upgrade.

  5. After extracting the file from the appliance, remove the file and directory to save disk space:
cd / tmp
rm upgradelogs.tar.gz
rm -r upgradelogs

Additional Information

Labels:

Comments

Post a Comment

Popular posts from this blog

Quick Guide to VCF Automation for VCD Administrators

  Quick Guide to VCF Automation for VCD Administrators VMware Cloud Foundation 9 (VCF 9) has been  released  and with it comes brand new Cloud Management Platform –  VCF Automation (VCFA)  which supercedes both Aria Automation and VMware Cloud Director (VCD). This blog post is intended for those people that know VCD quite well and want to understand how is VCFA similar or different to help them quickly orient in the new direction. It should be emphasized that VCFA is a new solution and not just rebranding of an old one. However it reuses a lot of components from its predecessors. The provider part of VCFA called Tenenat Manager is based on VCD code and the UI and APIs will be familiar to VCD admins, while the tenant part inherist a lot from Aria Automation and especially for VCD end-users will look brand new. Deployment and Architecture VCFA is generaly deployed from VCF Operations Fleet Management (former Aria Suite LCM embeded in VCF Ops. Fleet Management...
Post a Comment
Read more
  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more
  "Cloud zone insights not available yet, please check after some time" message on Aria Automation https://knowledge.broadcom.com/external/article?articleNumber=314894 Products VMware Aria Suite Issue/Introduction Symptoms: The certificate for Aria operations has been replaced since it was initially added to Aria Automation as an integration. When accessing the Insights pane under  Cloud Assembly  ->  Infrastructure  ->  Cloud Zone  ->  Insights  the following message is displayed:   "Cloud zone insights not available yet, please check after some time." The  /var/log/services-logs/prelude/hcmp-service-app/file-logs/hcmp-service-app.log  file contains ssl errors similar to:   2022-08-25T20:06:43.989Z ERROR hcmp-service [host='hcmp-service-app-xxxxxxx-xxxx' thread='Thread-56' user='' org='<org_id>' trace='<trace_id>' parent='<parent_id>' span='<span_id>'] c.v.a.h.a.common.AlertEnu...
Post a Comment
Read more