Tech-Gen

  Scheduled workflows fail post upgrade of VMware Aria Automation/VMware Aria Orchestrator to version 8.18 Issue/Introduction Workflows scheduled in the VMware Aria Orchestrator fail after upgrading to VMware Aria Automation/VMware Aria Orchestrator version 8.18. The same workflows run successfully when executed manually (without scheduling). Environment   It affects both Standalone VMware Aria Orchestrator and embedded VMware Aria Orchestrator. Cause The issue is due to regressions in the VMware vCenter (VC) plugin on VMware Aria Automation/VMware Aria Orchestrator 8.18. Resolution This is a known issue with the VMware Aria Automation/VMware Aria Orchestrator 8.18 and the workaround is to update the VC plugin on VMware Aria Orchestrator with the latest version (attached with the KB). Alternatively, wait for the official fix in VMware Aria Automation/VMware Aria Orchestrator 8.18.1. Workaround :               Note: Take (non-memory)  sna...

  vRA 8 upgrade preparation fails: "Attempt failed to run command: /opt/scripts/upgrade/ssh-noop.sh" Issue/Introduction Symptoms: vRealize Automation 8.x upgrade fails with the error: Error Code: LCMVRAVACONFIG90029 Preparation for upgrade has discovered problems. Review to error report below to correct the problems and try again. The services remained in working order. vRealize Automation Upgrade Errors: [vra_fqdn][Exit Code: 255] Attempt failed to run command: /opt/scripts/upgrade/ssh-noop.sh. Pseudo-terminal will not be allocated because stdin is not a terminal. Welcome to VMware vRealize Automation Appliance 8.9.1 root@vra_fqdn: Permission denied (publickey,password). [vra_fqdn] Remote command failed: /opt/scripts/upgrade/ssh-noop.sh at host: vra_fqdn [vra_fqdn] Remote command failed: /opt/scripts/upgrade/ssh-noop.sh at one or more nodes Environment VMware vRealize Automation 8.x Cause This issue can be caused by a corrupt  /root/home/.ssh/authorized_keys  file. Resol...

  vRealize Automation 8.x upgrade failed when iptables.service did not start Issue/Introduction To be able to successfully upgrade the vRA cluster. Symptoms: The upgrade process appeared to complete successfully on two of three nodes. kubelet  service did not start on one node after rebooting as part of the upgrade. There were no error messages observed in the vRSLCM and task will not fail. In  /var/log/bootstrap/postupdate.log  you see messages similar to: 2023-03-18 15:44:46Z /etc/bootstrap/postupdate.d/01-20-iptables starting... iptables.service is not active, cannot reload. 2023-03-18 15:44:46Z Script /etc/bootstrap/postupdate.d/01-20-iptables failed, error status 1 [INFO][2023-03-18 15:53:31][vra_fqdn] Waiting for VAMI to exit ... [INFO][2023-03-18 15:54:01][vra_fqdn] Verifying VAMI overall upgrade result ... [INFO][2023-03-18 15:54:01][vra_fqdn] VAMI upgrade completed with error. [ERROR][2023-03-18 15:54:01][vra_fqdn] VAMI overall upgrade has failed. VAMI has e...

  Aria Automation upgrade fails with 'Error while deploying services' and 'vco-app' pods failing to come up fully Issue/Introduction Symptoms: Upgrade fails with 'Error while deploying services.' output of 'kubectl -n prelude get pods' shows most pods running but the 'vco-app' pods are only partially running. Checking 'vco'-app' logs we only see below errors: nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.symphony.csp.auth.service.CspClient]: Factory method 'cspClientService' threw exception; nested exception is java.lang.IllegalArgumentException:  expected system property https.proxyPort to be a number but got empty string Environment VMware Aria Automation 8.x Cause The issue occurs when the proxy service has been disabled using command vracli proxy disable. This should not be done unless consulted with VMware. The SOAP plugin requires the http.proxyPort property t...

  New installs of vRealize Orchestrator 8.x fail to install due to a POD STATUS of 'CrashLoopBackOff' Issue/Introduction Symptoms: When deploying a new installation of vRealize Orchestrator 8.x, or vRealize Automation 8.x with embedded vRealize Orchestrator 8.x, you may experience the following: vRealize Suite Lifecycle Manager displays a failure to initialize the VRA cluster with the error  LCMVRAVACONFIG590003 . The Kubernetes pod for  vco-app- <ID>   fails to start with a STATUS of ' CrashLoopBackOff '.  To confirm this, run the following command on the appliance: kubectl -n prelude get pods The status of the vco-app pod is   CrashLoopBackOff Confirm the error by running the following command on the appliance: kubectl -n prelude logs vco-app- <ID from previous command>   -c install-rpms The  vco-app- <ID>  container log contains the following error: You are required to change your password immediately (password expired) g...

  Network isolation causes split-brain scenario in a 3 node cluster: Resetting vPostgres clustering Issue/Introduction Provide instructions on how to monitor and restore a 3 node vPostgres cluster within Kubernetes containers. Symptoms: The command  vracli status  shows multiple primary database nodes. vPostgres is unable to elect a single master node. prelude-noop-intnet-netcheck.log files within pods/kube-system/prelude-noop-intnet-ds-***** directories have entries similar to the below 2019/12/31 08:27:04 Failed ping for 10.244.2.2, packet loss is 100.000000 2019/12/31 08:27:04 Failed ping for 10.244.1.5, packet loss is 100.000000 2019/12/31 08:27:04 Pinging the majority of nodes failed. 3 node vRealize Automation 8.0 / 8.0.1 cluster does not have redundant network pathing Cause 3 node vPostgres clustering can breakdown due to network isolation / connectivity creating a split-brain scenario of 3 running master databases. Resolution Resiliency improvements will be intro...

  Troubleshooting VMware Aria Automation 8.x (formerly vRealize Automation 8.x) upgrades Issue/Introduction This article provides steps to manually collect upgrade logs if the log bundle collection fails when a VMware Aria Automation upgrade fails. When an upgrade fails, the following logs need to be analyzed: /var/log/vmware/prelude/upgrade-*.log Upgrade reports. Review based on timestamp One node environment /opt/vmware/var/log/vami/*.log Cluster environments /opt/log/vmware/var/log/vami/*.log Packages installation details /var/log/bootstrap/postupdate.log Initialization scripts details /var/log/bootstrap/everyboot.log Initialization scripts details /var/log/vmware/prelude/deploy-*.log Services startup details Note:  Some log files will have timestamps as part of the file name, for others the information is appended. It is important to validate that the information is from the latest upgrade attempt. This blog provides an example of the expected output of the logs ...

  Upgrade of Cluster VRA 8.x fails with Split brain scenario Issue/Introduction To successfully upgrade a vRA cluster. Symptoms: 3 node vRA cluster upgrade fails and each node tries to act as a separate cluster upgrade-noop-error  log shows vRA fails to make the upgrade directory : [INFO][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx] Preparing directory for upgrade data on remote node: [INFO][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx] Running remote command: mkdir -p /var/vmware/prelude/upgrade at host: * [ERROR][2022-08-05 19:16:01][xxxx.xxxxx.xxxx.xxxx][Exit Code: 255] Attempt failed to run command: mkdir -p /var/vmware/prelude/upgrade. Environment VMware vRealize Automation 8.x Cause This happens because the appliance fails to make the upgrade folder at " / var/vmware/prelude/upgrade " because the folder already exists from old upgrade attempts. Resolution There is a workaround.  Workaround: The workaround steps are below:  Revert back to the snapshot before the upgr...