Skip to main content

Aria Automation nodes in not ready state and deploy.sh fails

 Aria Automation nodes in not ready state and deploy.sh fails

https://knowledge.broadcom.com/external/article/377044/contact-broadcom-support.html

Products

VMware Aria Suite

Issue/Introduction

  • Unable to shut down Aria Automation by running /opt/scripts/deploy.sh --shutdown
  • deploy.sh script fails with below error: 

Running check eth0-ip

Running check node-name

Running check non-default-hostname

Running check single-aptr

Running check nodes-ready
make: *** [/opt/health/Makefile:56: nodes-ready] Error 1
Running check nodes-count

Running check fips

make: Target 'deploy' not remade because of errors.

 

  • Running kubectl get nodes shows one node in a NotReady state
  • Running kubectl -n prelude get pods -o wide shows the postgres-0 pod in a pending state
  • Running kubectl describe nodes | grep "Name:\|Taints:" shows that the node where postgres-0 is running is tainted


Environment

  • Aria Automation 8.x three node cluster

Cause

  • One of the nodes is tainted and therefore in a NotReady state. This causes the health check scripts to fail when deploy.sh is run

Resolution

  • Work around this issue by removing the taint.
  • Workaround Steps: 
     
    • Run kubectl get nodes to determine which node is a NotReady state
    • Run kubectl -n prelude get pods -o wide to verify that the postgres-0 pod in a pending state
    • Run kubectl describe nodes | grep "Name:\|Taints:" verify that the node where postgres-0 is running is tainted

root@servername01 [ ~ ]# kubectl describe nodes | grep "Name:\|Taints:"
Name:               servername01.example.local
Taints:             node.kubernetes.io/unreachable:NoSchedule
Name:               servername02.example.local
Taints:             <none>
Name:               servername03.example.local
Taints:             <none>

    • Run this to remove the taint (replace the relevant servername in the command with the tainted node in your environment from the above commands):

      kubectl taint nodes servername01.example.local node.kubernetes.io/unreachable:NoSchedule-
    • Run this again to verify that no nodes are tainted:

      kubectl describe nodes | grep "Name:\|Taints:" 

It should now show that there is no taint:

root@servername01 [ ~ ]# kubectl describe nodes | grep "Name:\|Taints:"
Name:               servername01.example.local
Taints:             <none>
Name:               servername02.example.local
Taints:             <none>
Name:               servername03.example.local
Taints:             <none>

Note: In some cases the taint may still show up, rerun the command until the taint no longer shows on the affected node or any of the nodes. 

    • Now run kubectl get nodes
    • After all three nodes show as "Ready", the shutdown command can be run again. 

/opt/scripts/deploy.sh --shutdown

Comments

Post a Comment

Popular posts from this blog

Quick Guide to VCF Automation for VCD Administrators

  Quick Guide to VCF Automation for VCD Administrators VMware Cloud Foundation 9 (VCF 9) has been  released  and with it comes brand new Cloud Management Platform –  VCF Automation (VCFA)  which supercedes both Aria Automation and VMware Cloud Director (VCD). This blog post is intended for those people that know VCD quite well and want to understand how is VCFA similar or different to help them quickly orient in the new direction. It should be emphasized that VCFA is a new solution and not just rebranding of an old one. However it reuses a lot of components from its predecessors. The provider part of VCFA called Tenenat Manager is based on VCD code and the UI and APIs will be familiar to VCD admins, while the tenant part inherist a lot from Aria Automation and especially for VCD end-users will look brand new. Deployment and Architecture VCFA is generaly deployed from VCF Operations Fleet Management (former Aria Suite LCM embeded in VCF Ops. Fleet Management...
Post a Comment
Read more
  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more
  "Cloud zone insights not available yet, please check after some time" message on Aria Automation https://knowledge.broadcom.com/external/article?articleNumber=314894 Products VMware Aria Suite Issue/Introduction Symptoms: The certificate for Aria operations has been replaced since it was initially added to Aria Automation as an integration. When accessing the Insights pane under  Cloud Assembly  ->  Infrastructure  ->  Cloud Zone  ->  Insights  the following message is displayed:   "Cloud zone insights not available yet, please check after some time." The  /var/log/services-logs/prelude/hcmp-service-app/file-logs/hcmp-service-app.log  file contains ssl errors similar to:   2022-08-25T20:06:43.989Z ERROR hcmp-service [host='hcmp-service-app-xxxxxxx-xxxx' thread='Thread-56' user='' org='<org_id>' trace='<trace_id>' parent='<parent_id>' span='<span_id>'] c.v.a.h.a.common.AlertEnu...
Post a Comment
Read more