After creating prereq vm, setting all parameters and deploying 4 esxi hosts in the first part of the article
I started the third part by configuring accessibility as more and more elements were needed, more and more elements needed access between environments, I decided to run routing and NAT on prereq VM.
# Enable IP forwarding on the VM
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command 'echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/10-ip_forward.conf' -ShowStandardOutputStream -ShowErrorOutputStream
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command 'sysctl -w net.ipv4.ip_forward=1' -ShowStandardOutputStream -ShowErrorOutputStream
# Configure a service to enable IP forwarding on startup
$ipForward =@"
[Unit]
Description=Enable IP forwarding
[Service]
Type=oneshot
ExecStart=/sbin/sysctl -p /etc/sysctl.d/10-ip_forward.conf
[Install]
WantedBy=multi-user.target
"@
$ipForward = $ipForward -replace "`r`n", "`n"
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "echo '$ipForward' > /etc/systemd/system/enable-ip-forwarding.service"
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "systemctl enable enable-ip-forwarding.service"
# Configure iptables to allow forwarding and save the configuration
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "iptables-legacy -A FORWARD -i eth0 -o eth1 -j ACCEPT" -ShowStandardOutputStream -ShowErrorOutputStream
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "iptables-legacy -t nat -A POSTROUTING -o eth0 -j MASQUERADE" -ShowStandardOutputStream -ShowErrorOutputStream
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "iptables-legacy-save > /etc/systemd/scripts/ip4save" -ShowStandardOutputStream -ShowErrorOutputStreamadditionally, to make it easier for me to get into the environment, I added static routing and hosts entries to solve fqdn names
# Add a route and host entries for ESXi and vCenter on local machine
route add "$globalIPOctet.0" mask "255.255.255.0" "$prereqVmIp"
if ((Get-Content "C:\Windows\System32\drivers\etc\hosts" | Select-String -Pattern "^$globalIPOctet\.10") -ne $null) {
Write-Log "Host entry for hl-esxi10.vworld.domain.lab already exists."
} else {
Write-Log "Adding host entry for hl-esxi10.vworld.domain.lab."
Add-Content -Path "C:\Windows\System32\drivers\etc\hosts" -Value "`n$globalIPOctet.10 hl-esxi10.vworld.domain.lab hl-esxi10"
Start-Sleep -Seconds 2
}after setting up accessibility i started vCenter setups and set up the following one by one
Virtualization extension
$vmNames = @("hl-esxi10","hl-esxi11","hl-esxi12","hl-esxi13")
Connect-VIServer $esxiHost -User $esxiU -Password $esxiP
foreach($vmName in $vmNames)
{
# Get the virtual machine object
$vm = Get-VM -Name $vmName
# Check if the virtual machine is running
if ($vm.PowerState -eq "PoweredOn") {
# If the virtual machine is running, shut it down
Stop-VM -VM $vm -Confirm:$false
}
# Enable virtualization extensions on the virtual machine
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.nestedHVEnabled = $true
$vm.ExtensionData.ReconfigVM($spec)
$vm | Select-Object Name, @{N="HvEnabled";E={$_.ExtensionData.SystemResources.Config.HostFeatureCapability.FeatureName -contains "hv.capable"}}
Start-Sleep -Seconds 10
# Power on the virtual machine
Start-VM -VM $vm
# Wait for the virtual machine to start completely
Start-Sleep -Seconds 180
}Datacenter
$datacenterName = "Datacenter"
$location = "Datacenters"
# Create the new datacenter
New-Datacenter -Name $datacenterName -Location (Get-Folder -Name $location)
Cluster (HA, DRS, vSAN)
# Create the new cluster
$clusterName = "Compute"
$datacenter = Get-Datacenter -Name $datacenterName
New-Cluster -Location $datacenter -Name $clusterName -HAEnabled -DrsEnabled -VsanEnabledAdd host to cluster and exit maintanance mode
#Add Host
$cluster = Get-Cluster -Name $clusterName -Location $datacenter
$hostList = @("hl-esxi10.vworld.domain.lab","hl-esxi11.vworld.domain.lab","hl-esxi12.vworld.domain.lab","hl-esxi13.vworld.domain.lab")
$hostU = "root"
$hostP = "VMware1!"
foreach($esxihost in $hostList)
{
# Add the host to the cluster
Add-VMHost -Name $esxihost -Location $cluster -User $hostU -Password $hostP -Force
Write-Log -ForegroundColor GREEN "Adding ESXi host $esxihost to vCenter"
}
# Get all hosts in maintenance mode
$hosts = Get-VMHost | Where-Object {$_.ConnectionState -eq "Maintenance"}
# Exit maintenance mode for each host
foreach ($node in $hosts) {
Write-Log "Exiting maintenance mode for host $($node.Name)"
Set-VMHost -VMHost $node -State Connected
}
Add Network redundancy and setup MTU, vMotion, vSAN Traffic
#Networking
foreach($esxihost in $hostList)
{
$vSwitch = Get-VirtualSwitch -Name "vSwitch0" -VMHost $esxihost
$networkAdapter = Get-VMHostNetworkAdapter -Physical -Name "vmnic1" -VMHost $esxihost
Add-VirtualSwitchPhysicalNetworkAdapter -VirtualSwitch $vSwitch -VMHostPhysicalNic $networkAdapter -Confirm:$false
}
foreach($esxihost in $hostList)
{
# Get the vSwitch0 and vmkernel network adapter
$vSwitch = Get-VirtualSwitch -Name vSwitch0 -VMHost $esxihost
$vmKernelAdapter = Get-VMHostNetworkAdapter -VMKernel -Name vmk0 -VMHost $esxihost
# Change the MTU size on the vSwitch0 network adapter
Set-VirtualSwitch $vSwitch -Mtu 1600 -Confirm:$false
# Change the MTU size on the vmkernel network adapter
Set-VMHostNetworkAdapter -VirtualNic $vmKernelAdapter -Mtu 1600 -VMotionEnabled $true -VsanTrafficEnabled $true -Confirm:$false
}Setup vSAN
#VSAN
foreach($esxihost in $hostList)
{
$vsanHost = Get-VMHost | Where-Object {$_.Name -eq $esxihost}
$luns = $vsanHost | Get-ScsiLun | Select-Object CanonicalName, CapacityGB
foreach ($lun in $luns) {
if(([int]($lun.CapacityGB)).toString() -eq "75") {
$vsanCacheDisk = $lun.CanonicalName
}
if(([int]($lun.CapacityGB)).toString() -eq "350") {
$vsanCapacityDisk = $lun.CanonicalName
}
}
New-VsanDiskGroup -Server $vcsaName -VMHost $vsanHost -SsdCanonicalName $vsanCacheDisk -DataDiskCanonicalName $vsanCapacityDisk
}
and clear all alarms as I don’t like them in lab 😉
#Clear all alarms
Connect-VIServer $vcsaName -User $vcsaU -Password $vcsaP
$alarmMgr = Get-View AlarmManager
$filter = New-Object VMware.Vim.AlarmFilterSpec
$filter.Status += [VMware.Vim.ManagedEntityStatus]::red
$filter.Status += [VMware.Vim.ManagedEntityStatus]::yellow
$filter.TypeEntity = [VMware.Vim.AlarmFilterSpecAlarmTypeByEntity]::entityTypeAll
$filter.TypeTrigger = [vmware.vim.AlarmFilterSpecAlarmTypeByTrigger]::triggerTypeAll
$alarmMgr.ClearTriggeredAlarms($filter)and this finished the infrastructure preparation for our vRA Management and Compute
As most of you know, vRA deployment is currently done with Easy Installer and the nested solution is not supported and the solution that I will present to you here is even more so. But let’s get to the point
I started the deployment of the environment by setting additional local entries in DNS and in our global DNS on prereq VM.
Then once again using VCC I downloaded the iso with Easy Installer
Write-Log "vRA ISO Downloading!"
if (Test-Path "$localPath\$vraISO") {
Write-Log "File exists!"
} else {
Write-Log "VRA ISO file does not exist."
Write-Log "Downloading a file"
$downloadvRA = "$localPath\vcc.exe download -p vmware_vrealize_suite -s vra -v 8.11.2 -f $vraISO --accepteula --user ""$myUsername"" --pass ""$myPassword"" --output ""$localPath\"""
$output = Invoke-Expression $downloadvRA
}after downloading and mounting as drive I did deployment with ovftool
# Mount the ISO file
Mount-DiskImage -ImagePath $vraISOpath
# Get the drive letter of the mounted ISO
$driveLetter = (Get-DiskImage -ImagePath $vraISOpath | Get-Volume).DriveLetter
Start-Sleep -Seconds 10
C:\'Program Files'\VMware\'VMware OVF Tool'\ovftool.exe --name="hl-lcm" --X:injectOvfEnv --X:logFile=ovftool.log --allowExtraConfig --noSSLVerify --network="$vmPortGroup" --acceptAllEulas --diskMode=thin --powerOn --prop:vami.hostname="hl-lcm.vworld.domain.lab" --prop:varoot-password="VMware1!" --prop:va-ssh-enabled=True --prop:va-firstboot-enabled=True --prop:va-telemetry-enabled=True --prop:va-ntp-servers="$globalIPOctet.2" --prop:vami.gateway.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="$globalIPOctet.2" --prop:vami.domain.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="vworld.domain.lab" --prop:vami.searchpath.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="vworld.domain.lab" --prop:vami.DNS.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="$globalIPOctet.2" --prop:vami.ip0.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="$globalIPOctet.20" --prop:vami.netmask0.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance="255.255.255.0" -ds="$esxiDatastore" "${driveLetter}:\vrlcm\VMware-vLCM-Appliance-8.10.0.6-21471042_OVF10.ova" vi://"$esxiU":"$esxiP"@$esxiHost
$lcmVM = "hl-lcm"After deployment, the VM automatically starts and with the help of the API call we can check if everything is already working
$url = "https://$lcmVM.vworld.domain.lab/lcm/bootstrap/api/status"
...
# Parse response JSON
$data = ConvertFrom-Json $request.Content
# Check deployment status
$deploymentStatus = $data.status
while ($deploymentStatus -ne "SUCCESS") {If the VM reports SUCCESS in the deployment, we can change the so-called first-boot-password, also using the API
$url = "https://$lcmVM.vworld.domain.lab/lcm/authzn/api/firstboot/updatepassword"
$body = @{
username = $lcmU
password = "VMware1!"
} | ConvertTo-Jsonthe next step caused me a bit of trouble because I couldn’t upload ova from vra and idm to lcm via api (too little knowledge of powershell), because something that works with curl is not necessarily easy to achieve here, so this time I decided to use scp from putty
standard check if putty is available, download and install
# Set the path to pscp
$pscpPath = "C:\Program Files\PuTTY\pscp.exe"
# Check if pscp is installed
if (Test-Path $pscpPath) {
Write-Log "pscp is already installed."
} else {
# Set the URL to download the PuTTY installer
$url = "https://the.earth.li/~sgtatham/putty/latest/w64/putty-64bit-0.78-installer.msi"
# Set the path to download the PuTTY installer
$installerPath = "C:\temp\putty-64bit-0.76-installer.msi"
# Download the PuTTY installer
Invoke-WebRequest -Uri $url -OutFile $installerPath
# Install PuTTY
Start-Process msiexec.exe -ArgumentList "/i $installerPath /quiet /qn" -Wait
# Remove the PuTTY installer
Remove-Item $installerPath
Write-Log "pscp has been installed."
}
and copy file
# Copy the file using pscp
& pscp.exe -pw $lcmNewP $vidmfilePath "${lcmR}@${lcmVM}:${remoteVidmFilePath}"
# Copy the file using pscp
& pscp.exe -pw $lcmNewP $vrafilePath "${lcmR}@${lcmVM}:${remoteVraFilePath}"as the files are already on the LCM, they need to be mapped
$url = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/settings/sourcelocationsetting"
$requestBody = @(
@{
"name" = "vidm.ova"
"filePath" = "/data/ova/vidm.ova"
"type" ="install"
},
@{
"name" = "vra.ova"
"filePath" = "/data/ova/vra.ova"
"type" ="install"
}
)Next step is create datacenter
#create datacetner
$url = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/v2/datacenters"
$header = @{
Authorization = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${lcmU}:$lcmNewP"))
}
$payload = @{
"dataCenterName"="Default"
"primaryLocation"="HomeLab;;;;"
} | ConvertTo-JsonCreate vCenter and Default Password
#create vcenter password
$url = "https://$lcmVM.vworld.domain.lab/lcm/locker/api/passwords"
$header = @{
Authorization = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${lcmU}:$lcmNewP"))
}
$payload = @{
"alias"="vcenter"
"password"="VMware1!"
"confirmPassword" = "VMware1!"
"passwordDescription" = ""
"userName" = "Administrator@vsphere.local"
} | ConvertTo-Json
and add vCenter. The process of adding vCenter must be validated, therefore the same payload is called twice
$url = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/v2/datacenters/$datacenterID/vcenters/validate"
$header = @{
Authorization = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${lcmU}:$lcmNewP"))
}
$payload = @{
"vCenterName"="$vcsa"
"vCenterHost"="hl-vcsa.vworld.domain.lab"
"vcUsername" = "Administrator@vsphere.local"
"vcPassword" = "locker:password:${vcenterPasswordID}:vcenter"
"vcUsedAs" = "Administrator@MANAGEMENT_AND_WORKLOAD"
} | ConvertTo-Json
$response = Invoke-RestMethod -Method POST -Uri $url -Body $payload -ContentType "application/json" -Headers $header -SkipCertificateCheck
$requestID = $response.requestId
do{
$url = "https://$lcmVM.vworld.domain.lab/lcm/request/api/v2/requests/$requestID"
$response = Invoke-RestMethod -Method GET -Uri $url -Body $requestBody -ContentType "application/json" -Headers $header -SkipCertificateCheck
$status = $response.state
Write-Log "Validation status is " $status
Start-Sleep -Seconds 15
}while(($status -ne "COMPLETED") -and ($status -ne "FAILED"))
if($status -eq "COMPLETED")
{
$url = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/v2/datacenters/$datacenterID/vcenters"
$header = @{
Authorization = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${lcmU}:$lcmNewP"))
}
$payload = @{
"vCenterName"="$vcsa"
"vCenterHost"="hl-vcsa.vworld.domain.lab"
"vcUsername" = "Administrator@vsphere.local"
"vcPassword" = "locker:password:${vcenterPasswordID}:vcenter"
"vcUsedAs" = "MANAGEMENT_AND_WORKLOAD"
} | ConvertTo-Json
$response = Invoke-RestMethod -Method POST -Uri $url -Body $payload -ContentType "application/json" -Headers $header -SkipCertificateCheck
}just in case, I also started a data collection
$url = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/v2/datacenters/$datacenterID/vcenters/$vcsa/data-collection"I created a certificate
$url = "https://$lcmVM.vworld.domain.lab/lcm/locker/api/v2/certificates"
$header = @{
Authorization = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${lcmU}:$lcmNewP"))
}
$jsonPayload = @{
alias = "vRealize Suite"
cN = "vrealize"
ip = @()
host = @(
"hl-idm.vworld.domain.lab"
"hl-vra.vworld.domain.lab"
)
oU = "vWorld"
size = "2048"
o = "vWorld"
l = "Domain"
sT = "Home Lab"
c = "EU"
} | ConvertTo-Jsonand enabled IDM deployment
$uri = "https://$lcmVM.vworld.domain.lab/lcm/lcops/api/v2/environments"
$jsonPayload = @{
environmentId = "globalenvironment"
environmentName = "globalenvironment"
environmentHealth = $null
environmentDescription = $null
logHistory = $null
environmentStatus = $null
infrastructure = @{
properties = @{
dataCenterVmid = "$datacenterID"
regionName = ""
zoneName = ""
vCenterName = $vcsa
vCenterHost = $vcsaName
vcUsername = "Administrator@vsphere.local"
vcPassword = "locker:password:${vcenterPasswordID}:vcenter"
acceptEULA = "true"
enableTelemetry = "false"
defaultPassword = "locker:password:${defaultPasswordID}:default"
certificate = "locker:certificate:${certificateID}:vRealize Suite"
cluster = "Datacenter#Compute"
storage = "vsanDatastore"
folderName = ""
resourcePool = ""
diskMode = "thin"
network = "VM Network"
masterVidmEnabled = "false"
dns = "$globalIPOctet.2"
domain = "vworld.domain.lab"
gateway = "$globalIPOctet.2"
netmask = "255.255.255.0"
searchpath = "vworld.domain.lab"
timeSyncMode = "host"
ntp = ""
isDhcp = "false"
}
}
products = @(
@{
id = "vidm"
version = "3.3.7"
properties = @{
defaultConfigurationEmail = "admin@vworld.domain.local"
vidmAdminPassword = "locker:password:${defaultPasswordID}:default"
syncGroupMembers = $true
nodeSize = "medium"
defaultConfigurationUsername = "admik"
defaultConfigurationPassword = "locker:password:${defaultPasswordID}:default"
defaultTenantAlias = ""
vidmDomainName = ""
certificate = "locker:certificate:${certificateID}:vRealize Suite"
contentLibraryItemId = ""
fipsMode = "false"
}
clusterVIP = @{
clusterVips = @()
}
nodes = @(
@{
type = "vidm-primary"
properties = @{
vmName = "hl-idm"
hostName = "hl-idm.vworld.domain.lab"
ip = "$globalIPOctet.21"
}
}
)
}
)
} | ConvertTo-Json -Depth 10having such an environment, we can do a vRA deployment which I put in the code and it is enough to uncomment it, however, my home lab is not a speed demon and vRA on nested throws a timeout on the installation of containers using helm, that’s why I decided to create a code fragment that is for people like me, i.e. high IO latency.
The vRA deployment is done with ovftool on our main ESXI host. Ova you have in iso Easy Installer because we copied it from there to LCM
all required parameters for deployment can be taken from ovf which is available after unpacking ova
C:\'Program Files'\VMware\'VMware OVF Tool'\ovftool.exe --name="hl-vra" --X:injectOvfEnv --X:logFile=ovftool.log --allowExtraConfig --noSSLVerify --network="$vmPortGroup" --acceptAllEulas --diskMode=thin --powerOn --prop:vami.hostname="hl-vra.vworld.domain.lab" --prop:varoot-password="VMware1!" --prop:k8s-cluster-cidr="10.244.0.0/22" --prop:k8s-service-cidr="10.244.4.0/22" --prop:ntp-servers="$globalIPOctet.2" --prop:fips-mode="disabled" --prop:features-switch="" --prop:vami.gateway.vRealize_Automation="$globalIPOctet.2" --prop:vami.domain.vRealize_Automation="vworld.domain.lab" --prop:vami.searchpath.vRealize_Automation="vworld.domain.lab" --prop:vami.DNS.vRealize_Automation="$globalIPOctet.2" --prop:vami.ip0.vRealize_Automation="$globalIPOctet.22" --prop:vami.netmask0.vRealize_Automation="255.255.255.0" --prop:vm.vmname="vRealize_Automation" -ds="Data-02" "${driveLetter}:\ova\vra.ova" vi://"$esxiU":"$esxiP"@$esxiHost
$vraVM = "hl-vra"now we can either read the firstboot log in real time or wait for a response from vracli status first-boot. I opted for the second solution
# Define the file and pattern to search for
$file = "/var/log/bootstrap/firstboot.log"
$pattern_true = "First boot complete"
$pattern_false = "error"
# Loop until the pattern is found
while ($true) {
# Get the contents of the file
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "vracli status first-boot"
# Search for the pattern in the file contents
$match_true = $output.Output | Select-String -Pattern $pattern_true
$match_false = $output.Output | Select-String -Pattern $pattern_false
# If the pattern is found, exit the loop
if ($match_true) {
Write-Log "FirstBoot Deployment Finished Successfully"
break
}
if ($match_false) {
Write-Log "FirstBoot Deployment Finished with Error "+$match_false
break
}then we add licenses
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "vracli license add $serialKey"and create a file with the IDM password
#password file
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "echo VMware1! > /tmp/pass"now we need to switch to our IDM because we need SHA256Fingerprint. We can get it with two commands
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "openssl s_client -showcerts -connect localhost:443 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/cert.crt"
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "openssl x509 -noout -fingerprint -sha256 -inform pem -in /tmp/cert.crt"
$sha = $output.Output
$shaFingerpring = $sha.Split("=")[1]we go back to our vRA and we can connect to IDM without additional user intervention
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "vracli vidm set https://hl-idm.vworld.domain.lab admin /tmp/pass admik -f $shaFingerpring" The last element that completes the deployment work is the launch of the deploy.sh script
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "/opt/scripts/deploy.sh"
$session = New-SSHSession -ComputerName $vraVM -Credential $credentialsvraVM -AcceptKey -Force
# Define the file and pattern to search for
$file = "/var/log/deploy.log"
$pattern_true = "Prelude has been deployed successfully"
$pattern_false = "Traceback (most recent call last)"
# Loop until the pattern is found
while ($true) {
# Get the contents of the file
$output = Invoke-SSHCommand -SessionId $session.SessionId -Command "tail $file"
# Search for the pattern in the file contents
$match_true = $output.Output | Select-String -Pattern $pattern_true
$match_false = $output.Output | Select-String -Pattern $pattern_false
# If the pattern is found, exit the loop
if ($match_true) {
Write-Log "Deployment Finished Successfully"
break
}
if ($match_false) {
Write-Log "Deployment Finished with Error "+$match_false
break
}
}Once done, we have a fully functional vRA available. If we hope that it will never break down, we can move it to our nested environment and connect it to LCM as an import, but it’s up to you.
I achieved what I wanted full script on deployment 2458 lines of code, many sleepless nights, but full of joy. All code is available on my git
https://github.com/vWorldLukasz/vmware/blob/main/HomeLab-codes/HomeLab-p1.ps1
Plans for the future, I would definitely like to add HA for vRA plus additional vRO deployment, but these are minor elements. I still think that the code could be refined and some elements that are static could be changed to variables, but I don’t know if it will be in the near future. Now more challenges ahead of me and I hope more interesting articles for you
I hope that at least one of my readers will test the whole thing and enjoy it as much as I do