Skip to main content

 

New installs of vRealize Orchestrator 8.x fail to install due to a POD STATUS of 'CrashLoopBackOff'


Issue/Introduction

Symptoms:

When deploying a new installation of vRealize Orchestrator 8.x, or vRealize Automation 8.x with embedded vRealize Orchestrator 8.x, you may experience the following:

  • vRealize Suite Lifecycle Manager displays a failure to initialize the VRA cluster with the error LCMVRAVACONFIG590003.

  • The Kubernetes pod for vco-app-<ID> fails to start with a STATUS of 'CrashLoopBackOff'.  To confirm this, run the following command on the appliance:

kubectl -n prelude get pods

  • The status of the vco-app pod is CrashLoopBackOff

  • Confirm the error by running the following command on the appliance:

kubectl -n prelude logs vco-app-<ID from previous command> -c install-rpms

  • The vco-app-<ID> container log contains the following error:

You are required to change your password immediately (password expired)
groupadd: PAM: Authentication token is no longer valid; new one required
useradd: group 'vco' does not exist
error: %prein(vco-server-8.1.0.1576591045-15331417.noarch) scriptlet failed, exit status 6
error: vco-server-8.1.0.1576591045-15331417.noarch: install failed

Environment

VMware vRealize Automation 8.x
VMware vRealize Orchestrator 8.x

Cause

The deployed vRealize Orchestrator container may be created with an expired password causing packages to fail and the container to crash.

Resolution

Workaround:
To work around this issue, use the following steps:

  1. Log into the Appliance with SSH.
  2. Navigate to /opt/charts/vco/templates/
  3. Copy/Backup the deployment.yaml file using the command:
cp deployment.yaml /tmp/
NOTE:  Do not copy any backup.yaml files into /opt/charts/vco/ or /opt/charts/vco/templates/.  Doing so will result in 404 errors after successful service restart.
  1. Edit the deployment.yaml file using your preferred editor (vi deployment.yaml).
  2. Locate the string:  init_run.sh
  3. Replace the text as follows.
Search for row that contains this text: init_run.sh:
command:
- "/bin/bash"
- "-c"
- "/init_run.sh"

Edit this row by just adding these two "sed" commands before /init_run.sh script. The row after editing should looks similar to this::
command:

- "/bin/bash"
- "-c"
- "sed -i 's/root:.*/root:x:18135:0:99999:7:::/g' /etc/shadow && sed -i 's/vco:.*/vco:x:18135:0:99999:7:::/g' /etc/shadow && /init_run.sh"
  1. Save the file
If you are deploying outside of vRealize Suite Lifecycle Manager, follow these steps:
  1. Navigate to /opt/scripts/
  2. Execute the script: ./deploy.sh
If you are deploying through vRealize Suite Lifecycle Manager, follow these steps:
  1. Log back into Lifecycle Manager and 'Retry' the deployment that failed.  Installation should now progress as expected.

Comments

Popular posts from this blog

ESXi Host Troubleshooting Checklist

  🛠️ ESXi Host Troubleshooting Checklist (With Complete Log Locations) ✅ 1. Host Status & Connectivity Check host state in vCenter (Connected / Not Responding / Disconnected) Review CPU, RAM, and datastore usage Validate HA/DRS recommendations ✅ 2. Hardware Health Monitor hardware sensors (CPU, DIMMs, fans, PSU, RAID) Check RAID controller logs via vendor tools View hardware status in: vCenter → Monitor → Hardware Health DCUI → Hardware Status ✅ 3. Network Validation Verify vSwitches, Port Groups, NIC teaming, VLAN tagging Check vmkernel ports (mgmt, vMotion, iSCSI, vSAN) Test reachability using: Shell vmkping < IP > Show more lines Validate physical NIC status and link speed ✅ 4. Storage & Datastore Checks Confirm datastore accessibility Rescan storage adapters (iSCSI/FC/NFS) Validate multipathing (Round Robin / Fixed / MRU) Check datastore latency with esxtop ✅ **5. Critical Logs & Their Locations (Full List) Here are key ESXi log files that every VMware adm...

Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 ðŸ”¹ Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware ⸻ 1️⃣ Memory Ballooning (vmmemctl) Ballooning is the first memory reclamation technique used when ESXi detects memory pressure. ➤ Step-by-Step: How Ballooning Works  1. VMware Tools installs the balloon driver (vmmemctl) inside the guest OS.  2. ESXi detects low free memory on the host.  3. ESXi inflates the balloon in selected VMs.  4. Balloon driver occupies guest memory, making the OS think RAM is full.  5. Guest OS frees idle / unused pages (because it believes memory is needed).  6. ESXi reclaims those freed pages and makes them available to other VMs. Why Ballooning Happens?  • Host free memory is very low.  • ESXi wants the VM to release unused pages before resorting to swapping. Example  • Host memory: 64 GB  • VMs used: 62 GB  • Free: 2 GB → ESXi triggers ballooning  • VM1 (8 GB RAM): Balloon inflates to 2 GB → OS frees 2 GB → ESXi re...
  vCenter Troubleshooting Tips, Common Issues & Log Locations 🚀 1. General Troubleshooting Approach A. Check service health first For vCenter Server Appliance (VCSA): Shell https : / / < vcenter - FQDN > : 5480 Show more lines Go to Services → Health to verify: vCenter Server ESXi hosts vSphere Client DNS, NTP, DB connections vpxd, vpxd-svcs, vsphere-ui B. DNS & NTP checks vCenter is highly dependent on correct DNS & time sync. Check: Shell nslookup < vcenter - fqdn > nslookup < esxi - host - fqdn > Show more lines Time drift > 5 minutes causes: Login failures vpxd crashes PSC authentication issues C. Check storage/database health Slow DB/storage = UI slow, tasks hung. For VCSA: Shell df - h du - sh / storage / log / * / var / log Show more lines Free up space if partitions hit >80%. D. Restart critical services safely On VCSA: Shell service - control - - status service - control - - stop - - all service - control - - start - - all ...