Cloud Foundation: Deploy of SDDC Manager failed during bring-up

Recently I moved to the latest Cloud Foundation version (v4.4) in a lab environment that I often re-deploy. I noticed that during the Cloud Builder bring-up phase the deploy of the SDDC managed failed ever since, while using the same input parameters in the deployment parameters workbook. So it worked in 4.1.x and 4.2.x, but not in 4.4 (I did not test 4.3). Strange, right?

Symptoms

Cloud Builder was well underway when the error “SDDC Manager VM <vm name> is not yet up” showed up. Cloud Builder tried re-deploying SDDC Manager a couple of times, but eventually stopped and displayed error below in the Cloud Builder UI.

What is the exact issue here? One of the symptoms was that the SDDC Manager Web UI did not load. It only showed an empty page, named “Express HTML”.

So, what did Cloud Builder do right, before it failed?

• SDDC Manager was VM deployed using the correct vApp Options in vCenter
• SDDC Manager IP address was assigned and pingable
• SDDC Manager SSH proces was responding

Troubleshooting

After logging an Service Request (SR) with VMware Support, we worked towards an understanding of the issue. By looking at the Cloud Builder logs, it showed relevant info:

2022-03-02T16:10:19.195+0000 [bringup,9ba7a6999e2f2c57,8d32] ERROR [c.v.e.s.s.DeploySddcManagerOnClusterAction,pool-4-thread-2] Failed while checking SddcMgr VM <SDDC Manager hostname>
com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to <SDDC Manager hostname>

2022-03-02T16:10:19.195+0000 [bringup,9ba7a6999e2f2c57,8d32] ERROR [c.v.e.s.c.u.c.SshCommandExecuter,pool-4-thread-2] Could not connect to the SSH server @ <SDDC Manager hostname> for configuration.
com.jcraft.jsch.JSchException:

This pointed towards an authentication error. After trying to log into the SDDC Manager appliance remotely via SSH and the console, the password for the users “vcf” and “root” (that should be 100% right), did not work.

Solution

Based on the authentication errors shown, the VMware engineer that worked on the SR, remembered seeing something like this in the Cloud Foundation 4.3 version. In that case the password that was used in the deployment parameters workbook for VCF users “root” and “vcf” were too easy.

My use-case for this Cloud Foundation deployment is building a repeatable lab environment, so the passwords were held simple. In my case: VMware1! 😎.

If the authentication to the SDDC Manager failed, it is probable due to the password being weak. Under the hood, Cloud Foundation uses the output of “cracklib-check” to determine if a password is weak. To check it out for yourself, login with SSH to the Cloud Builder VM and perform the command below.

admin@<cloud builder> [ ~ ]# echo "VMware1!" | cracklib-check
VMware1!: it is based on a dictionary word

If the password is based on a dictionary word, the deployment task of SDDC Manager will fail, because no pre-check is done on weak passwords. The VMware engineer that worked on the case, told me that’s password pre-checks are considered in a future update or release of Cloud Foundation.