Sorting bits into bytes...

SDDC Manager Unable to establish SSH session to host

Today I ran into a problem when trying to replace the self-signed SSL certificates used in a VMware Cloud Foundation 3.9 deployment. Due to security reasons the customer didn’t setup a Microsoft CA. So I had to replace the certificates by hand. This process is well documented on this link. The only thing not documented is what to do in case of trouble… and I hit trouble. The CSR failed only on vRealize Log Insight, for other products the CSR was created. But because of 1 failure, I wasn’t able to download the CSR tar file. So I guess I need to fix the issue before I can continue…

Finding the right log file

I had no clue which log file to tail. So what I did was a tail on all the logfiles. 🙂

SSH into the SDDC Manager with the VCF user.

Change to the log folders “cd /var/log/vmware/vcf

Start the tail and grep for any error messages “tail -f */*.log | grep -i “error”

Output:

vcf@sddcmgr [ /var/log/vmware/vcf ]$ tail -f */*.log | grep -i "error"
2020-02-25T13:25:39.023+0000 ERROR [f21e79acf397be69,c64c] [c.v.e.s.i.s.TaskAssociationServiceImpl,vac-scheduler-1] Cluster not found - 8e56afb4-1cd6-49e3-ba34-8105b6fe9bae
2020-02-25T13:25:39.024+0000 ERROR [f21e79acf397be69,c64c] [c.v.e.s.t.services.TaskServiceImpl,vac-scheduler-1] Inventory update failed
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2020-02-25T13:26:01.751+0000 ERROR [37f413c12e224904,4313] [c.v.v.r.s.i.ResourceDataProviderServiceImpl,http-nio-127.0.0.1-7300-exec-7] Failed to fetch resource info for NSX-T Managers
2020-02-25T13:26:01.913+0000 ERROR [37f413c12e224904,4313] [c.v.v.r.s.i.ResourceDataProviderServiceImpl,http-nio-127.0.0.1-7300-exec-7] Failed to fetch resource info for NSX-T Managers
2020-02-25T13:26:17.726+0000 ERROR [37f413c12e224904,9398] [c.v.e.sddc.common.util.VRMSSHSession,om-exec-5] Unable to establish SSH session to host: vRLI.top.secret.now
2020-02-25T13:26:17.727+0000 ERROR [37f413c12e224904,9398] [c.v.vcf.certmgmt.common.util.CSRUtil,om-exec-5] Unable to generate csr for resource: vRLI.top.secret.now
2020-02-25T13:26:17.736+0000 ERROR [37f413c12e224904,6bc3] [c.v.v.c.s.o.i.CertificateOperationOrchestratorImpl,om-exec-5] Unable to write csr/cert to file for host: vRLI.top.secret.now
2020-02-25T13:26:39.030+0000 DEBUG [15bc05d35e49115f,5f79] [c.v.v.c.r.a.c.ResponseBuilder,http-nio-127.0.0.1-7300-exec-1] {"name":"Certificate Operation: GENERATE_CSR","localizableNamePack":{"component":"operationsmanager","messageKey":"CERTIFICATE_MANAGEMENT_WORKFLOW","arguments":["GENERATE_CSR"]},"startTime":1582637162188,"status":"FAILED","tasksOrder":"vRLI.top.secret.now:vrli","tasks":[{"name":"vRLI.top.secret.now:vrli","localizableNamePack":{"component":"operationsmanager","messageKey":"CERTIFICATE_MANAGEMENT_TASK_NAME","arguments":["vRLI.top.secret.now","vrli"]},"description":"GENERATE_CSR for vRLI.top.secret.now and resource type vrli","localizableDescriptionPack":{"component":"operationsmanager","messageKey":"CERTIFICATE_MANAGER_TASK_DESCRIPTION","arguments":["GENERATE_CSR","vRLI.top.secret.now","vrli"]},"status":"FAILED","errors":[{"message":"Failed to generate CSR for vRLI.top.secret.now due to: SSH: Failed to establish SSH session to vRLI.top.secret.now."}],"createdTime":1582637162188,"updatedTime":1582637177730,"id":"1fc5be9f-507e-4614-ac2e-5b9b42b55147"}],"id":"1fc5be9f-507e-4614-ac2e-5b9b42b55147"}
2020-02-25T13:26:40.027+0000 ERROR [f8cf1aa7f924da69,c125] [c.v.e.s.i.s.TaskAssociationServiceImpl,vac-scheduler-1] Cluster not found - 8e56afb4-1cd6-49e3-ba34-8105b6fe9bae
2020-02-25T13:26:40.027+0000 ERROR [f8cf1aa7f924da69,c125] [c.v.e.s.t.services.TaskServiceImpl,vac-scheduler-1] Inventory update failed
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

 

The error message points towards the SDDC Manager not being able to logon to vRLI. And indeed I also wasn’t able to logon to vRLI with the root account. As the password is stored in a keypass, I guess the account got a lock out due to the high number of failed login attempts. After waiting for about 15min I was able to logon to vRLI. So the problem is password related. The customer did change the root password on the vRLI appliance. But not on the SDDC Manager. Thats the root cause of this. But how to fix? Well following this link and you will find the root vRLI password stored in the SDDC Manager. I changed the password of the vRLI appliance and then I was able to generate the CSR file I was looking for.

8 thoughts on “SDDC Manager Unable to establish SSH session to host

  1. I love your blog.. very nice colors & theme.
    Did you design this website yourself or did you hire someone to do it for you?
    Plz reply as I’m looking to design my own blog and would like to find out where u got
    this from. appreciate it

  2. Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.

Leave a Reply