Today I was informed that there were some logging problems to vCenter Server (VCSA). I did a short investigation via SSH and check if all required services are running and I found out that vpxd is down. I checked vpxd.log (tail -100 /var/log/vmware/vpxd/vpxd.log) and found that there were some db errors and:
Panic: Unrecoverable database error. Shutting down VC
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_GUEST_DISK (VM_ID, PATH, CAPACITY, FREE_SPACE) VALUES (?, ?, ?, ?)"
2017-06-06T12:55:23.080+02:00 warning vpxd[7F1554F7F700] [Originator@6876 sub=VpxProfiler opID=HB-host-373@511-34d8fdc1] DoHostSync:host-373 [DoHostSyncTime] took 19512 ms
2017-06-06T12:55:23.080+02:00 warning vpxd[7F1554F7F700] [Originator@6876 sub=InvtHostCnx opID=HB-host-373@511-34d8fdc1] Exception occurred during host sync; Got database error; [vim.HostSystem:host-373,esxi3-lab.nihonno.domein], e: N3Vim5Fault13DatabaseError9ExceptionE(vim.fault.DatabaseError)
Panic: Unrecoverable database error. Shutting down VC
I could also found info about VM causing this issue:
Failed to insert guest disk info for VM id = 506554 because of database error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_guest_disk";
I found an old VMware KB about similar issue happened in VMware vCenter Server 2.5 - 4.1.x. I connected to the VCDB by running below command from the vCenter Server Appliance 6.5 bash Shell:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
then I run the following command to determine the virtual machine name which causes the problem:
VCDB=# SELECT NAME FROM VPX_ENTITY WHERE ID = '506554';
name
---------------
RJ-CENTOSNBU8
(1 row)
I contacted an owner of above VM and we decided to do a workaround and rebooted the VM (there weren't any mount points mentioned on below KB). It solved the problem - probably there was a problem with VMware Tools. Finally we decided to remove this "trash VM" to avoid such problems in the future. If you can not power off or remove VM, you have to investigate (probably with VMware Support) the root of the problem.
Conclusion
An old bug like repeating itself in the later versions 🙂 It's annoying but it's not the first time that I have seen such situation.
Useful links:
Interacting with the vCenter Server Appliance 6.5 embedded vPostgres Database (2147285)