Skip to main content

Aria Automation nodes in not ready state and deploy.sh fails

 Aria Automation nodes in not ready state and deploy.sh fails

https://knowledge.broadcom.com/external/article/377044/contact-broadcom-support.html

Products

VMware Aria Suite

Issue/Introduction

  • Unable to shut down Aria Automation by running /opt/scripts/deploy.sh --shutdown
  • deploy.sh script fails with below error: 

Running check eth0-ip

Running check node-name

Running check non-default-hostname

Running check single-aptr

Running check nodes-ready
make: *** [/opt/health/Makefile:56: nodes-ready] Error 1
Running check nodes-count

Running check fips

make: Target 'deploy' not remade because of errors.

 

  • Running kubectl get nodes shows one node in a NotReady state
  • Running kubectl -n prelude get pods -o wide shows the postgres-0 pod in a pending state
  • Running kubectl describe nodes | grep "Name:\|Taints:" shows that the node where postgres-0 is running is tainted


Environment

  • Aria Automation 8.x three node cluster

Cause

  • One of the nodes is tainted and therefore in a NotReady state. This causes the health check scripts to fail when deploy.sh is run

Resolution

  • Work around this issue by removing the taint.
  • Workaround Steps: 
     
    • Run kubectl get nodes to determine which node is a NotReady state
    • Run kubectl -n prelude get pods -o wide to verify that the postgres-0 pod in a pending state
    • Run kubectl describe nodes | grep "Name:\|Taints:" verify that the node where postgres-0 is running is tainted

root@servername01 [ ~ ]# kubectl describe nodes | grep "Name:\|Taints:"
Name:               servername01.example.local
Taints:             node.kubernetes.io/unreachable:NoSchedule
Name:               servername02.example.local
Taints:             <none>
Name:               servername03.example.local
Taints:             <none>

    • Run this to remove the taint (replace the relevant servername in the command with the tainted node in your environment from the above commands):

      kubectl taint nodes servername01.example.local node.kubernetes.io/unreachable:NoSchedule-
    • Run this again to verify that no nodes are tainted:

      kubectl describe nodes | grep "Name:\|Taints:" 

It should now show that there is no taint:

root@servername01 [ ~ ]# kubectl describe nodes | grep "Name:\|Taints:"
Name:               servername01.example.local
Taints:             <none>
Name:               servername02.example.local
Taints:             <none>
Name:               servername03.example.local
Taints:             <none>

Note: In some cases the taint may still show up, rerun the command until the taint no longer shows on the affected node or any of the nodes. 

    • Now run kubectl get nodes
    • After all three nodes show as "Ready", the shutdown command can be run again. 

/opt/scripts/deploy.sh --shutdown

Comments

Popular posts from this blog

ESXi Host Troubleshooting Checklist

  🛠️ ESXi Host Troubleshooting Checklist (With Complete Log Locations) ✅ 1. Host Status & Connectivity Check host state in vCenter (Connected / Not Responding / Disconnected) Review CPU, RAM, and datastore usage Validate HA/DRS recommendations ✅ 2. Hardware Health Monitor hardware sensors (CPU, DIMMs, fans, PSU, RAID) Check RAID controller logs via vendor tools View hardware status in: vCenter → Monitor → Hardware Health DCUI → Hardware Status ✅ 3. Network Validation Verify vSwitches, Port Groups, NIC teaming, VLAN tagging Check vmkernel ports (mgmt, vMotion, iSCSI, vSAN) Test reachability using: Shell vmkping < IP > Show more lines Validate physical NIC status and link speed ✅ 4. Storage & Datastore Checks Confirm datastore accessibility Rescan storage adapters (iSCSI/FC/NFS) Validate multipathing (Round Robin / Fixed / MRU) Check datastore latency with esxtop ✅ **5. Critical Logs & Their Locations (Full List) Here are key ESXi log files that every VMware adm...

Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 🔹 Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware ⸻ 1️⃣ Memory Ballooning (vmmemctl) Ballooning is the first memory reclamation technique used when ESXi detects memory pressure. ➤ Step-by-Step: How Ballooning Works  1. VMware Tools installs the balloon driver (vmmemctl) inside the guest OS.  2. ESXi detects low free memory on the host.  3. ESXi inflates the balloon in selected VMs.  4. Balloon driver occupies guest memory, making the OS think RAM is full.  5. Guest OS frees idle / unused pages (because it believes memory is needed).  6. ESXi reclaims those freed pages and makes them available to other VMs. Why Ballooning Happens?  • Host free memory is very low.  • ESXi wants the VM to release unused pages before resorting to swapping. Example  • Host memory: 64 GB  • VMs used: 62 GB  • Free: 2 GB → ESXi triggers ballooning  • VM1 (8 GB RAM): Balloon inflates to 2 GB → OS frees 2 GB → ESXi re...
  vCenter Troubleshooting Tips, Common Issues & Log Locations 🚀 1. General Troubleshooting Approach A. Check service health first For vCenter Server Appliance (VCSA): Shell https : / / < vcenter - FQDN > : 5480 Show more lines Go to Services → Health to verify: vCenter Server ESXi hosts vSphere Client DNS, NTP, DB connections vpxd, vpxd-svcs, vsphere-ui B. DNS & NTP checks vCenter is highly dependent on correct DNS & time sync. Check: Shell nslookup < vcenter - fqdn > nslookup < esxi - host - fqdn > Show more lines Time drift > 5 minutes causes: Login failures vpxd crashes PSC authentication issues C. Check storage/database health Slow DB/storage = UI slow, tasks hung. For VCSA: Shell df - h du - sh / storage / log / * / var / log Show more lines Free up space if partitions hit >80%. D. Restart critical services safely On VCSA: Shell service - control - - status service - control - - stop - - all service - control - - start - - all ...