Step-by-Step Explanation of Ballooning, Compression & Swapping in VMware

 

vIDM appliance has no space left on device /db for audit data

Issue/Introduction

This article provides steps to add space in the vIDM appliance adding config to limit audit data.

• vIDM appliance /db file system is almost full, for example usage exceeds about 90% and more.
• You will see these errors:

check /db file system usage by df command as follows
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/db_vg-db 10076 8318 1246 87% /db

• Getting vIDM log-bundle fails with the error: No space left on device.

Environment

VMware Identity Manager 3.3.x

Cause

vIDM appliance uses device /db for Elasticsearch, RabbitMQ and internal DB PostgreSQL.
 /db
   |- data
   |- elasticsearch
   |- log
   |- lost+found
   |- temp
A large number of audit data for Elasticsearch uses almost all space on device /db file system.
As the default settings vIDM appliance keeps audit data forever, the vIDM file system will eventually fill up.

Resolution

To resolve this issue, add a policy to limit audit data in the /usr/local/horizon/conf/runtime-config.properties file:

1. Login to the vIDM appliance.
2. Edit the /usr/local/horizon/conf/runtime-config.properties file.

vi /usr/local/horizon/conf/runtime-config.properties

3. Add these two rows to the file:

analytics.deleteOldData=true -----> The default value is false.
analytics.maxQueryDays=90 -----> The default value is 90, which indicates the number of days to keep the audit date. It is recommended to be 90 days to get 12-week statistics. 

4. Reboot the vIDM appliance to reflect the new setting.
5.  Remove the audit data manually which has already expired more than 90 days using the curl command for cleanup. You can use wildcards to delete the audit data:

Examples:
curl -XDELETE http://localhost:9200/v3_2016*
Explanation : Deletes everything from 2016.
curl -XDELETE http://localhost:9200/v3_2016-12* 
Explanation:Deletes everything from December 2016.

Backup : If the curl command fails, run the following commands:

rabbitmqctl list_queues | grep analytics.null       
Explanation : Lists the RabbitMQ queue

rabbitmqctl purge_queue -.analytics.null           
Explanation : Purges the queue
For versions vIDM 3.3.4 and higher, Workspace ONE Access 20.10 and higher, you will need to do the following workaround as the command above will fail on the new Photon releases. 

wget 'https://raw.githubusercontent.com/rabbitmq/rabbitmq-management/v3.7.20/bin/rabbitmqadmin'
chmod +x rabbitmqadmin
sed -i 's|#!/usr/bin/env python|#!/usr/bin/env python3|' rabbitmqadmin
mv rabbitmqadmin /usr/sbin/
rabbitmqadmin -q list queues | grep analytics
rabbitmqadmin purge queue name=-.analytics.127.0.0.1

Note:

• * in the file name is wildcard.
• Do not use rm commands to remove the audit data unless all nodes are stopped.

Additional Information

There is also a related KB on increasing the size of the /db file system.
* VMware Identity Manager Linux appliance disk containing Postgres database, analytics data, and support log bundle (/db) is full (2082650)
----------------------------------------
rm command is distributed, replicated data store, so removing the files when elasticsearch is running will cause problems. It is recommended to use the “curl” command to perform the removal safely.

https://knowledge.broadcom.com/external/article?articleNumber=367645