Skip to main content

Pods do not start on Aria Automation node

 Pods do not start on Aria Automation node

https://knowledge.broadcom.com/external/article?articleNumber=389553

roducts

VMware Aria Suite

Issue/Introduction

Aria Automation UI may be available intermittently or not at all depending on if this is a cluster of three nodes, or a single node. It may also depend on how many nodes are effected if clustered behind a load balancer. 

Some interactions may work, such as deployments via UI, while tools leveraging the API may be reporting connection errors

Environment

Aria Automation 8.x

Cause

Timeouts occur when kubelet and docker client attempt to connect to docker daemon. This can be confirmed with the following logs in `journalctl`:

Feb 25 12:45:47 <node_name> kubelet[1890547]: E0225 12:45:47.593202 1890547 docker_service.go:265] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
Feb 25 12:47:47 <node_name> kubelet[1890547]: F0225 12:47:47.597520 1890547 server.go:269] failed to run Kubelet: failed to get docker info: operation timeout: context deadline exceeded

The following can also be seen in the journal to confirm this issue:

Feb 25 12:47:47 <node_name> kubelet[1890547]: F0225 12:47:47.597520 1890547 server.go:269] failed to run Kubelet: failed to get docker info: operation timeout: context deadline exceeded
Feb 25 12:47:47 <node_name> kubelet[1890547]: goroutine 1 [running]:
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0x1)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0x8a
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x6ee48e0, 0x3, {0x0, 0x0}, 0xc0009ba2a0, {0x5801a35, 0xc0000a2c00}, 0xc000480430, 0x0)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x569
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x5, 0x61eecfd8, {0x0, 0x0}, {0x0, 0x0}, 0xc000bbfcd8, {0xc000480430, 0x1, 0x1})
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:732 +0x191
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).print(...)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:714
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatal(...)

 

Feb 25 12:53:13 <node_name> systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3.
Feb 25 12:53:13 <node_name> systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 25 12:53:13 <node_name> systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Feb 25 12:53:13 <node_name> kubelet[1908513]: ++ uname -n
Feb 25 12:53:13 <node_name> kubelet[1908512]: + node_name=<node_name>

Resolution

  1. Validate the status of the kubelet service with the following command:
    1. systemctl status kubelet
  2. If not running, start with the following command:
    1. systemctl start kubelet
  3. Once started, monitor pods as they should begin to start automatically:
    1. kubectl get pods -n prelude
  4. Once all pods are in either a "Running" or "Completed" state the environment should be usable again

Comments

Post a Comment

Popular posts from this blog

Quick Guide to VCF Automation for VCD Administrators

  Quick Guide to VCF Automation for VCD Administrators VMware Cloud Foundation 9 (VCF 9) has been  released  and with it comes brand new Cloud Management Platform –  VCF Automation (VCFA)  which supercedes both Aria Automation and VMware Cloud Director (VCD). This blog post is intended for those people that know VCD quite well and want to understand how is VCFA similar or different to help them quickly orient in the new direction. It should be emphasized that VCFA is a new solution and not just rebranding of an old one. However it reuses a lot of components from its predecessors. The provider part of VCFA called Tenenat Manager is based on VCD code and the UI and APIs will be familiar to VCD admins, while the tenant part inherist a lot from Aria Automation and especially for VCD end-users will look brand new. Deployment and Architecture VCFA is generaly deployed from VCF Operations Fleet Management (former Aria Suite LCM embeded in VCF Ops. Fleet Management...
Post a Comment
Read more
  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more
  "Cloud zone insights not available yet, please check after some time" message on Aria Automation https://knowledge.broadcom.com/external/article?articleNumber=314894 Products VMware Aria Suite Issue/Introduction Symptoms: The certificate for Aria operations has been replaced since it was initially added to Aria Automation as an integration. When accessing the Insights pane under  Cloud Assembly  ->  Infrastructure  ->  Cloud Zone  ->  Insights  the following message is displayed:   "Cloud zone insights not available yet, please check after some time." The  /var/log/services-logs/prelude/hcmp-service-app/file-logs/hcmp-service-app.log  file contains ssl errors similar to:   2022-08-25T20:06:43.989Z ERROR hcmp-service [host='hcmp-service-app-xxxxxxx-xxxx' thread='Thread-56' user='' org='<org_id>' trace='<trace_id>' parent='<parent_id>' span='<span_id>'] c.v.a.h.a.common.AlertEnu...
Post a Comment
Read more