Cloud Foundation: Update SDDC bring-up parameters

While troubleshooting of a failed SDDC Manager deploy task in Cloud Foundation 4.4 together with VMware support, the engineer showed a way to update the SDDC bring-up parameters. This can be very helpful because it can restart a failed SDDC deployment with updated parameters. This prevents redeploying the physical servers intended for the management domain and reverting Cloud Builder to a previous snapshot.

Solution

The Cloud Builder VM has a so-called Supportability and Serviceability (SoS) utility available, which is located in “/opt/vmware/sddc-support/sos“. One of its features is to update the bring-up parameters. These are the parameters derived from the uploaded deployment parameters workbook. Updated parameters can be needed when the SDDC creation failed because of incorrect parameters.

In my case a more complex passwords for SDDC Manager was needed. Once the SDDC deploy is restarted, the updated settings are used for tasks that still need to be performed, not for finished ones.

Step 1: Update the workbook

As a first step, the deployment parameters workbook should be updated with the correct parameters (for example updated passwords). Afterwards upload the workbook to Cloud Builder VM in the “/home/admin” folder using SCP.

laptop:Downloads user$ scp vcf-ems-deployment-parameter.xlsx admin@<cloud builder IP>:/home/admin

Step 2: Determine SDDC UUID

Open the file “/opt/vmware/bringup/logs/vcf-bringup-debug.log” and search for the last entry with “End of Orchestration with FAILURE for Execution ID“.

2022-03-02T16:10:25.596+0000 [bringup,9ba7a6999e2f2c57,952b] INFO [c.v.e.s.o.core.OrchestratorImpl,pool-4-thread-18] End of Orchestration with FAILURE for Execution ID 6301351f-f1da-46ab-bef7-3ab029ca9963

A second way to determine the SDDC UUID is to open the API directly with curl.

laptop:~ user$ curl -k -u admin https://<cloud cuilder IP>/v1/sddcs
Enter host password for user 'admin':
{"elements":[{"id":"6301351f-f1da-46ab-bef7-3ab029ca9963","name":"Bringup-Ems","status":"COMPLETED_WITH_SUCCESS","creationTimestamp":"2022-03-28T07:50:25.480Z","sddcSubTasks":[{"sddcId":"6301351f-f1da-46ab-bef7-3ab029ca9963"
...

The “Execution ID” in the first way or “sddcId” in second way is the SDDC UUID that is needed in step 4.

Step 3: Generate JSON

Now the new JSON file can be created on the Cloud Builder appliance based on the modified deployment parameters workbook, by the entering the command below.

laptop:~ user$ ssh admin@<cloud cuilder IP>
admin@cb-appliance [ ~ ]$ sudo su -
root@cb-appliance [ ~ ]# cp /home/admin/vcf-ems-deployment-parameter.xlsx /tmp
root@cb-appliance [ ~ ]# /opt/vmware/sddc-support/sos --jsongenerator --jsongenerator-design vcf-ems --jsongenerator-input /tmp/vcf-ems-deployment-parameter.xlsx

The paragraph below shows the relevant output of the sos tool.

Welcome to Supportability and Serviceability(SoS) utility!
...
2022-03-23T21:27:03.227Z [INFO jsongeneratorservice.py::execute::88::MainThread] Input excel filename: vcf-ems-deployment-parameter.xlsx
...
2022-03-23T21:27:57.326Z [INFO JsonGenerator.py::create_file::58::MainThread] Generate output file /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems/vcf-ems.json
	Input JSON spec for management cluster deployment => /opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems/vcf-ems.json
...
2022-03-23T21:27:57.329Z [INFO sos.py::execute::253::MainThread] SoS operation completed in 54.31 seconds.

The new JSON file is by default available in “/opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems/vcf-ems.json” and ready to be used in step 4.

Step 4: Patch the SDDC bring-up parameters

In the last step the new JSON file is used to modify to the SDDC parameters. When executing curl on the Cloud Builder appliance it sends the new JSON file to the API together with the SDDC UUID. When the curl command is done, the SDDC bring-up restarts automatically at the failed task. I my case “Deploy SDDC Manager”.

laptop:~ user$ ssh admin@<cloud builder IP>
admin@cb-appliance [ ~ ]$ sudo su -
root@cb-appliance [ ~ ]# curl -k -u admin -X PATCH https://localhost/v1/sddcs/c000646c-6252-492a-a1df-37191fcd3305 -H "Content-Type: application/json" -d "@/opt/vmware/sddc-support/cloud_admin_tools/Resources/vcf-ems/vcf-ems.json"
Enter host password for user 'admin':

The paragraph below shows the relevant output of the curl command.

{"id":"6301351f-f1da-46ab-bef7-3ab029ca9963","name":"Bringup-Ems","status":"IN_PROGRESS","sddcSubTasks":[{"sddcId":"6301351f-f1da-46ab-bef7-3ab029ca9963"
...
"processingStateDescription":"Deploy SDDC Manager","name":"Generate SDDC Manager Input Data","description":"Deploy SDDC Manager","status":"POSTVALIDATION_COMPLETED_WITH_SUCCESS"
...
"processingStateDescription":"Deploy SDDC Manager","name":"Deploy SDDC Manager","description":"Deploy SDDC Manager","status":"INITIALIZED","creationTimestamp":"2022-03-28T08:57:04.798Z","updateTimestamp":"2022-03-28T12:09:41.262Z","errors":[{"errorCode":"SDDC_MGR_NOT_UP","arguments":["<vcf-appliance-name>"],"message":"SDDC Manager VM <vcf-vm-name> is not yet up","referenceToken":"Q18RLR"}]},{"sddcId":"6301351f-f1da-46ab-bef7-3ab029ca9963"

Succes

After performing the steps above, the Cloud Builder SDDC bring-up restarts and finishes with fully deployed SDDC.