Skip to main content

 

Aria Automation upgrade fails with 'Error while deploying services' and 'vco-app' pods failing to come up fully


Issue/Introduction

Symptoms:
  • Upgrade fails with 'Error while deploying services.'
  • output of 'kubectl -n prelude get pods' shows most pods running but the 'vco-app' pods are only partially running.
  • Checking 'vco'-app' logs we only see below errors:
    • nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.symphony.csp.auth.service.CspClient]: Factory method 'cspClientService' threw exception; nested exception is java.lang.IllegalArgumentException: expected system property https.proxyPort to be a number but got empty string

Environment

VMware Aria Automation 8.x

Cause

The issue occurs when the proxy service has been disabled using command vracli proxy disable. This should not be done unless consulted with VMware. The SOAP plugin requires the http.proxyPort property to be non-empty. If you do not wish to leverage the proxy enable it again with the default settings as per the resolution section.

Resolution

To resolve the issue follow the steps outlined below:

1. If using an external HTTP proxy run the following commands to enable the proxy and restart vRO.
# Enable the proxy
vracli proxy enable
# Appy the change
vracli proxy apply

If there is no need of an external proxy, restore the default proxy configuration.
# Delete the current configuration 
vracli proxy delete
# Set the default configuration
vracli proxy set-default
# Apply changes 
vracli proxy apply 


2. To verify that this reset has worked, execute the following command: 

kubectl -n prelude get secret internet.proxy.port -o json | jq -r .data.'"internet.proxy.port"' | base64 -d

If the reset was successful, it should print 3128 


3. Retry the failed LCM upgrade request with the revert to snapshot option set to 'false'

 

Additional Information

If reverted to pre-upgrade snapshots perform steps 1 to 3 and then initiate the upgrade task.

Comments

Popular posts from this blog

Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 ðŸ”¹ Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware ⸻ 1️⃣ Memory Ballooning (vmmemctl) Ballooning is the first memory reclamation technique used when ESXi detects memory pressure. ➤ Step-by-Step: How Ballooning Works  1. VMware Tools installs the balloon driver (vmmemctl) inside the guest OS.  2. ESXi detects low free memory on the host.  3. ESXi inflates the balloon in selected VMs.  4. Balloon driver occupies guest memory, making the OS think RAM is full.  5. Guest OS frees idle / unused pages (because it believes memory is needed).  6. ESXi reclaims those freed pages and makes them available to other VMs. Why Ballooning Happens?  • Host free memory is very low.  • ESXi wants the VM to release unused pages before resorting to swapping. Example  • Host memory: 64 GB  • VMs used: 62 GB  • Free: 2 GB → ESXi triggers ballooning  • VM1 (8 GB RAM): Balloon inflates to 2 GB → OS frees 2 GB → ESXi re...

ESXi Host Troubleshooting Checklist

  🛠️ ESXi Host Troubleshooting Checklist (With Complete Log Locations) ✅ 1. Host Status & Connectivity Check host state in vCenter (Connected / Not Responding / Disconnected) Review CPU, RAM, and datastore usage Validate HA/DRS recommendations ✅ 2. Hardware Health Monitor hardware sensors (CPU, DIMMs, fans, PSU, RAID) Check RAID controller logs via vendor tools View hardware status in: vCenter → Monitor → Hardware Health DCUI → Hardware Status ✅ 3. Network Validation Verify vSwitches, Port Groups, NIC teaming, VLAN tagging Check vmkernel ports (mgmt, vMotion, iSCSI, vSAN) Test reachability using: Shell vmkping < IP > Show more lines Validate physical NIC status and link speed ✅ 4. Storage & Datastore Checks Confirm datastore accessibility Rescan storage adapters (iSCSI/FC/NFS) Validate multipathing (Round Robin / Fixed / MRU) Check datastore latency with esxtop ✅ **5. Critical Logs & Their Locations (Full List) Here are key ESXi log files that every VMware adm...
  vCenter Troubleshooting Tips, Common Issues & Log Locations 🚀 1. General Troubleshooting Approach A. Check service health first For vCenter Server Appliance (VCSA): Shell https : / / < vcenter - FQDN > : 5480 Show more lines Go to Services → Health to verify: vCenter Server ESXi hosts vSphere Client DNS, NTP, DB connections vpxd, vpxd-svcs, vsphere-ui B. DNS & NTP checks vCenter is highly dependent on correct DNS & time sync. Check: Shell nslookup < vcenter - fqdn > nslookup < esxi - host - fqdn > Show more lines Time drift > 5 minutes causes: Login failures vpxd crashes PSC authentication issues C. Check storage/database health Slow DB/storage = UI slow, tasks hung. For VCSA: Shell df - h du - sh / storage / log / * / var / log Show more lines Free up space if partitions hit >80%. D. Restart critical services safely On VCSA: Shell service - control - - status service - control - - stop - - all service - control - - start - - all ...