Skip to main content

Pods do not start on Aria Automation node

 Pods do not start on Aria Automation node

https://knowledge.broadcom.com/external/article?articleNumber=389553

roducts

VMware Aria Suite

Issue/Introduction

Aria Automation UI may be available intermittently or not at all depending on if this is a cluster of three nodes, or a single node. It may also depend on how many nodes are effected if clustered behind a load balancer. 

Some interactions may work, such as deployments via UI, while tools leveraging the API may be reporting connection errors

Environment

Aria Automation 8.x

Cause

Timeouts occur when kubelet and docker client attempt to connect to docker daemon. This can be confirmed with the following logs in `journalctl`:

Feb 25 12:45:47 <node_name> kubelet[1890547]: E0225 12:45:47.593202 1890547 docker_service.go:265] Failed to execute Info() call to the Docker client: operation timeout: context deadline exceeded
Feb 25 12:47:47 <node_name> kubelet[1890547]: F0225 12:47:47.597520 1890547 server.go:269] failed to run Kubelet: failed to get docker info: operation timeout: context deadline exceeded

The following can also be seen in the journal to confirm this issue:

Feb 25 12:47:47 <node_name> kubelet[1890547]: F0225 12:47:47.597520 1890547 server.go:269] failed to run Kubelet: failed to get docker info: operation timeout: context deadline exceeded
Feb 25 12:47:47 <node_name> kubelet[1890547]: goroutine 1 [running]:
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0x1)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0x8a
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x6ee48e0, 0x3, {0x0, 0x0}, 0xc0009ba2a0, {0x5801a35, 0xc0000a2c00}, 0xc000480430, 0x0)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x569
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x5, 0x61eecfd8, {0x0, 0x0}, {0x0, 0x0}, 0xc000bbfcd8, {0xc000480430, 0x1, 0x1})
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:732 +0x191
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).print(...)
Feb 25 12:47:47 <node_name> kubelet[1890547]:         /build/mts/release/bora-19631864/cayman_kubernetes/kubernetes/src/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:714
Feb 25 12:47:47 <node_name> kubelet[1890547]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatal(...)

 

Feb 25 12:53:13 <node_name> systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3.
Feb 25 12:53:13 <node_name> systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Feb 25 12:53:13 <node_name> systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Feb 25 12:53:13 <node_name> kubelet[1908513]: ++ uname -n
Feb 25 12:53:13 <node_name> kubelet[1908512]: + node_name=<node_name>

Resolution

  1. Validate the status of the kubelet service with the following command:
    1. systemctl status kubelet
  2. If not running, start with the following command:
    1. systemctl start kubelet
  3. Once started, monitor pods as they should begin to start automatically:
    1. kubectl get pods -n prelude
  4. Once all pods are in either a "Running" or "Completed" state the environment should be usable again

Comments

Post a Comment

Popular posts from this blog

  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more

57 Tips Every Admin Should Know

Active Directory 1. To quickly list all the groups in your domain, with members, run this command: dsquery group -limit 0 | dsget group -members –expand 2. To find all users whose accounts are set to have a non-expiring password, run this command: dsquery * domainroot -filter “(&(objectcategory=person)(objectclass=user)(lockoutTime=*))” -limit 0 3. To list all the FSMO role holders in your forest, run this command: netdom query fsmo 4. To refresh group policy settings, run this command: gpupdate 5. To check Active Directory replication on a domain controller, run this command: repadmin /replsummary 6. To force replication from a domain controller without having to go through to Active Directory Sites and Services, run this command: repadmin /syncall 7. To see what server authenticated you (or if you logged on with cached credentials) you can run either of these commands: set l echo %logonserver% 8. To see what account you are logged on as, run this command: ...
Post a Comment
Read more
  The Guardrails of Automation VMware Cloud Foundation (VCF) 9.0 has redefined private cloud automation. With full-stack automation powered by Ansible and orchestrated through vRealize Orchestrator (vRO), and version-controlled deployments driven by GitOps and CI/CD pipelines, teams can build infrastructure faster than ever. But automation without guardrails is a recipe for risk Enter RBAC and policy enforcement. This third and final installment in our automation series focuses on how to secure and govern multi-tenant environments in VCF 9.0 with role-based access control (RBAC) and layered identity management. VCF’s IAM Foundation VCF 9.x integrates tightly with enterprise identity providers, enabling organizations to define and assign roles using existing Active Directory (AD) groups. With its persona-based access model, administrators can enforce strict boundaries across compute, storage, and networking resources: Personas : Global Admin, Tenant Admin, Contributor, Viewer Projec...
Post a Comment
Read more