Skip to main content

 https://knowledge.broadcom.com/external/article/344920/vcenter-70x-http-connection-exhaustion.html

vCenter 7.0.x HTTP Connection Exhaustion


Issue/Introduction

This article provides a way to remediate the VC connection limit problem.

Note: This article does NOT apply to local http/https connections happening within the vCenter Appliance such as service to service connections over the local loopback interface. (e.g. vsan-health making calls to VUM)


Symptoms:

In certain circumstances, remote https connections to vCenter may become exhausted.
By default, these connections are set to a maximum of 2048 connections.
 
One possible contributor to the limit being reached:
When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to vCenters Envoy proxy service. When this limit is reached, it may result in failed backups as part of those backup workflows.
 
To confirm that the limit is being reached, please do the following:
 
VDDK logging (file path depends on backup software):


2022-02-09T15:15:47.444+02:00 error -> [297506] [Originator@6876 sub=IO] HandshakeCb; <SSL(<io_obj p:0x00007fb7d806d1b8, h:32, <TCP '##.###.##.##.###:41704'>, <TCP '##.###.##.##.###: 443'>>)>; error: N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.)
 
VC Envoy logging (/var/log/vmware/envoy/envoy.log):


2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] remote https connections exceed > max allowed: 2048
2022-02-09T13:15:47.443Z warning envoy[40146] [Originator@6876 sub=filter] [C15632615] closing connection TCP<##.###.##.##.###:41704, ##.###.##.##.###:443>

 

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

  1. The limit that is more easily reached relates to vCenter Envoy proxy service since vCenter 7.0.x.
  2. When using VDDK 7.0.0 in a backup solution, it increases the amount of connections to the Envoy proxy service, increasing the likelihood that the maxRemoteHttpsConnection limit is reached. When this limit is reached, it may result in failed backups as part of those backup workflows. 

Resolution

None 


Workaround:


To avoid or remediate this scenario, a number of workarounds exist which can be applied in isolation or in parallel as needed:

Upgrade to 7.0 U1 or later VDDK version to avail of code improvements  

Increase the maximum RemoteHttpsConnections limit. See following steps:

 
To change envoy's proxy limit for external inbound HTTPS connections, the following changes are needed in the config options according to vCenter version:

For vCenter Server 7.0.2 and later,  change/add this config option “maxRemoteHttpsConnections” in /etc/vmware-rhttpproxy/config.xml 

<config>
   <envoy>
      <L4Filter>
         <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      </L4Filter> 
   </envoy>
</config>

For vCenter Server 7.0.1(Recommended minimum version) ,   change/add the config options “maxHttpConnections” and “maxHttpsConnections” in /etc/vmware-rhttpproxy/config.xml 

For vCenter Server 7.0.0,  change the config option “max_http_connections” and “max_https_connections” in /etc/vmware-envoy/config.yaml

 

Note vCenter Server 7.x has multiline comments in the /etc/vmware-rhttpproxy/config.xml file so there is a need to uncomment the desired option when change of the value is needed e.g.

Before:


  <L4Filter>
      <!--
      <maxRemoteHttpsConnections>2048</maxRemoteHttpsConnections>


After: 


  <L4Filter>
      <maxRemoteHttpsConnections>3072</maxRemoteHttpsConnections>
      <!--


After the configuration has been updated and the file has been saved, reload the configuration by one of the two ways:
1). Restart rhttpproxy service:
$ service-control --restart rhttpproxy
2). Send SIGHUP signal to rhttpproxy service:
$ kill -1 `pidof rhttpproxy` 
 

NOTE: While no hard limit exists to the maximum configurable limit, it is recommended to increase with caution as an overly extended limit may impact VC performance due to excessive connections and subsequent resource consumption. Increasing to 3072 is verified and suitable for most cases. If it is need to increase to number > 3072, please contact VMware support so that help can be provided to understand and analyze overly excessive connections in the environment.

Comments

Post a Comment

Popular posts from this blog

  Issue with Aria Automation Custom form Multi Value Picker and Data Grid https://knowledge.broadcom.com/external/article?articleNumber=345960 Products VMware Aria Suite Issue/Introduction Symptoms: Getting  error " Expected Type String but was Object ", w hen trying to use Complex Types in MultiValue Picker on the Aria for Automation Custom Form. Environment VMware vRealize Automation 8.x Cause This issue has been identified where the problem appears when a single column Multi Value Picker or Data Grid is used. Resolution This is a known issue. There is a workaround.  Workaround: As a workaround, try adding one empty column in the Multivalue picker without filling the options. So we can add one more column without filling the value which will be hidden(there is a button in the designer page that will hide the column). This way the end user will receive the same view.  
Post a Comment
Read more

57 Tips Every Admin Should Know

Active Directory 1. To quickly list all the groups in your domain, with members, run this command: dsquery group -limit 0 | dsget group -members –expand 2. To find all users whose accounts are set to have a non-expiring password, run this command: dsquery * domainroot -filter “(&(objectcategory=person)(objectclass=user)(lockoutTime=*))” -limit 0 3. To list all the FSMO role holders in your forest, run this command: netdom query fsmo 4. To refresh group policy settings, run this command: gpupdate 5. To check Active Directory replication on a domain controller, run this command: repadmin /replsummary 6. To force replication from a domain controller without having to go through to Active Directory Sites and Services, run this command: repadmin /syncall 7. To see what server authenticated you (or if you logged on with cached credentials) you can run either of these commands: set l echo %logonserver% 8. To see what account you are logged on as, run this command: ...
Post a Comment
Read more
  The Guardrails of Automation VMware Cloud Foundation (VCF) 9.0 has redefined private cloud automation. With full-stack automation powered by Ansible and orchestrated through vRealize Orchestrator (vRO), and version-controlled deployments driven by GitOps and CI/CD pipelines, teams can build infrastructure faster than ever. But automation without guardrails is a recipe for risk Enter RBAC and policy enforcement. This third and final installment in our automation series focuses on how to secure and govern multi-tenant environments in VCF 9.0 with role-based access control (RBAC) and layered identity management. VCF’s IAM Foundation VCF 9.x integrates tightly with enterprise identity providers, enabling organizations to define and assign roles using existing Active Directory (AD) groups. With its persona-based access model, administrators can enforce strict boundaries across compute, storage, and networking resources: Personas : Global Admin, Tenant Admin, Contributor, Viewer Projec...
Post a Comment
Read more