Recently, I managed to replace my worn-out Lab with something that has a little more power, I changed the memory from 92GB to 1024 ?.
So, due to the greater computing power, I decided to set myself a solution that has been tempting me for a long time and I dealt with its part through my career. I’m talking about the VCF here. The choice fell on VMware Cloud Foundation version 4.5.
This solution includes the following products
As you can see, we have a lot of fun ahead of us, but some of it is already known to you / us, so I will focus here on the VCF
The new version does not yet have an official release date, so I focused on the latest available. However, before I devoted time to deployment, I decided to explore the topic a bit. However, if you want to set up a nested environment because I will put it in this article, you will not find too much material. Most of the articles that have been written refer to the VLC solution, which by the way is a super-prepared tool, we really have to specify a few variables and the whole environment is ready to work. However, I wanted to play/learn, so I decided to bet everything myself without automation.
The element that I configured before I started any fun with VCF is a physical ESXI managed by vCenter (so that you can create a template, but it is not required at all)
On the host we create two portgroups or on vCenter we add VDS. I left Management on a regular vSwitch, and for the whole VCF integration I created a separate VDS that has two portgroups.
The first is the portgroup for Cloud Builder, unfortunately, with the OVF deployment, we do not have the option to set the VLAN, so I created Portgrupe vLAN, which adds the Management VLAN ID (1611) to the Cloud Builder frames. The second portgroup is already a trunk portgroup with the range 0-4094, i.e. allowing all VLANs. In addition, I increased the MTU to 9000 and changed all security options to Accept
The first element I prepared were ESXi hosts. I created a clean virtual machine template with the following parameters
The drives are Thin and I will mention network adapters later. An additional configuration that I made on the template is to add 3 parameters in Advanced Settings. As for the processors, at a later stage when you do a vra deployment in the Medium version, the machine does not start because they need 12 processors and we only have 8. However, this is a limitation of my physical equipment and we can’t help here.
featMask.vm.cpuid.PDPE1GB = Val:1
scsi1:0.virtualSSD = 1
scsi1:1.virtualSSD = 1
The first parameter concerns the support of hugepage by processors, my machine does not support it, so it was necessary to add such a value so that NSX-T Edge could be run in the nested environment. The next two parameters concern the marking of drives connected to esx as SSD (Flash)
There is one more element that I have not mentioned, more precisely in the processor, we must remember to select Expose hardware assisted virtualization to the guest OS. From such a template I built 4 machines that will work as my Management Domain
More specifically in the documentation
A cluster of physical hosts that contains the virtual machines of management components, such as vCenter Server, NSX-T Data Center, SDDC Manager, and so on.
After configuring 4 machines with ESX, i.e. the system installed on the smallest disk, I set
- IP address from my subnet
- I set the VLAN to 1611 because that’s what it uses (see diagram)
- I have disabled IPv6
- set DNS and hostname in the form of FQDN
- removed Custom DNS Suffix
After restarting the host (required if disabling IPv6) I generated a new certificate with the command
Here is the link to the documentation https://docs.vmware.com/en/VMware-Cloud-Foundation/4.5/vcf-deploy/GUID-20A4FD73-EB40-403A-99FF-DAD9E8F9E456.html
After this configuration you should login to GUI and on Portgroup VM Network also set VLAN 1611
The last element that needs to be configured here is NTP
- NTP must be configured to start with the host as well as point to a specific NTP host (in my environment it is pfSense, which I will describe in a moment)
pfSense is a virtual router that allows us to configure VLAN, Firewall, NAT, BGP and many other services that we can use in our environment. I used it for BGP and VLAN which it uses in the entire virtual infrastructure
The installation of pfSense from ISO itself is not complicated, so I will not write here how to do it. The machine I made for this router has the following parameters
Network Adapter 1 -> VM Network our WAN
Network Adapter 2 -> PG-allVLAN our LAN
After installing the machine, we have access to the console where we set the IP address
And then we can configure everything in a very friendly GUI
a) The first element that will actually be used at the very end but I “turned it on” at the beginning is to install the frr package which will later allow us to configure BGP
Go to the system tab and then package manager in available packages, look for frr and click install
When will the package be installed. We leave it alone until we get to the BGP configuration
b) The second thing I configured is the VLANs in the whole clean and simple VCF deployment we need 7 of them. We all create the same way.
Go to the Interfaces -> Assignments tab
And then to the VLANs tab
In the lower right corner, press Add and add additional VLANs as follows
In my environment, I have configured the VLANs as follows
With VLANs configured in this way, go to the first tab -> Interface Assignments and click Add to add all vlans as a new interface.
The interfaces added in this way must be configured, i.e. set the IP as Gateway and specify the MTU. All ports have Gateway set to 1. The first octet is 172. And 2 and 3 are the VLAN ID. For example, the first interface that will be my management VLAN
Having configured the interfaces in this way, I went to configure the Firewall, and more precisely, I passed all traffic through it as Allow, because it is a LAB, I do not need to wonder where my traffic is blocked.
c) We set the firewall in the Firewall Rules tab
Each port on the router has its own firewall, so on each port I added a rule using the Add button for any protocol, any source and any destination on any port Allow and on pfSense it looks like this
d) After configuring the firewall, I went to configure the DHC for our NSXs that will be deployed as part of the VCF
DHCP is configured in the Services -> DHCP Server tab
In my case, I fired DHCP on the OPT4 interface and it is VLAN 1614 for nsx-overlay. There is nothing complicated about the DHCP setting. We only need to do enable and specify the range
e) The element that I set to pfSense and is required for basic VCF deployment is NTP
In the services tab, go to NTP
What we need to do here is to enable NTP, choose which interface it will work on (I chose all available) and specify the Time Servers from which our Server will download the time
Such configured pfSense will allow us to deploy VCF from Cloud Builder. However, I want to have an environment where I will create various games from the vRA level, so I will show you how to configure two elements that will be useful in the future.
f) During the deployment of the EDGE cluster, we must choose the routing protocol, either we will create static routes or we will use the dynamic routing protocol, which is BGP (I will not tell you anything more about this because network matters are not my thing)
To configure BGP we need to have the package I showed you earlier installed. As we have the package installed, we go to the FRR Global Zebra tab available in the Services panel
Check Enable next to Enable FRR, set the Default router ID, I chose an ID that points to the 250th host in the subnet for which I will actually be routing.
As you can see, for more information on this topic, you can jump to Wikipedia
The next element is the master password which is mandatory
The last element that I configured here is Next hop Tracking where I set the IPv4 value for Resolve via default route
Nothing else for the Global tab so we can save and go to Route Maps. Here I added one RouteMap where I set it to permit and sequence value to 100
I didn’t change any other value from default. After pressing save and clicking on BGP we have a new bookmarks panel. This is where we configure BGP itself
We need to enable this routing protocol and set the local AS in my case it is 65001
At the bottom of this page in Network Distribution I set everything to be redistributed and the network and all possible networks by setting the subnet to 0.0.0.0/0 and selecting the previously created RouteMap
Once again, press Save and move on.
In the Neighbor tab, we add our neighbors, in my case, these are 4 interfaces that will be attached to EDGE when deploying Edge Cluster
When adding, in General options, we enter the Address and password with the appropriate option, which we will later provide when configuring Edge
Below, in Basic Options, we set Remote AS, which is something that will be on Edge in my case, it is 65003 and I selected the option Address Family -> Allow neighbor to advertise and receive routes for both IPv4 and IPv6
The last element that I have configured here is Peer Filtering. In the Route Map Filter tab for inbound and outbound traffic, I selected our previously created route map
Once configured, we can press save. Even though our neighbors haven’t been made yet. By going to the Services status tab, we can check if our BGP has started. If the daemon does not work, it means that we have something broken in the configuration
At this stage, we could once again say thank you, but I personally configured one more element, namely NAT, which in my network is required for xRegion which will be afforded by vRA, LCM, etc. Solutions.
g) Here is a simple matter. In the Firewall -> NAT tab
Go to Outbound and change the mode from Automatic (i.e. all vlans are automatically natated, good job pfsense, I don’t have to click it myself) to Hybrid which will allow us to add our own mappings
I added my xRegion subnet here so that I can access the Internet through all my routers
And now with a pure heart I can say that we have prepared and configured a router for VCF
- Cloud Builder.
Cloud Builder is a tool that does automagically ? deployment of the entire environment based on Excel which we fill with data. From what I’ve noticed, this version of Excel 4.5 is quite truncated.
However, before we move on to the Excel deployment, our Cloud Builder must be placed in our environment, but it is a regular deployment with OVF, so most of you can do it with your eyes closed. After launching the browser and entering the url of the cloud builder, of course logging in, the wizard appears
Regardless of whether we are doing a nested or physical environment, everything here works the same way.
We click Next
We accept that we have met all the requirements that are given
Of course, let’s not kid ourselves that there is a Validator because the first time we always forget something ?
We click next
Now we have the option to download excel or if we have it already prepared, we can upload it. According to the instructions, we can also use JSON, but I’ve never seen it, so I’ll focus on excel here
There are 3 tabs in the file in which we need to complete the data
Outside of Credentials, this is what my excel looks like. It’s almost default
After uploading the file to the wizard, Cloud Builder performs an environment check
Here we have, for example, checking if all gateways are reachable, if the hosts are configured correctly, if the thumbprints match, if the times are correct, etc. etc.
After fully validation
As you can see, I didn’t run it only once ? When we have everything green, we go to the installation process, which is cool that the validation went green, the installation process can fail.
In my case, however, this was not the case and after a few hours my deployment was over
And after pressing the finish button, a message appeared to my eyes
And here a new world appears to us ? where the fun has no end.
However, before we continue to play, I have configured the network. Therefore, we enter the tab on the left. Inventory -> Workload Domains
And click on our domain that Cloud Builder created
Here we have information about hosts, services and NSX access
In the Actions menu, select Add Edge Cluster
we provide all the parameters and Cloud builder on NSX creates an environment for T0 and T1 Routers
In these routers, the BGP that we configured earlier is used. Parameters we need to fill in
In the routing settings for T0, we create the ASN that we set on the pfSense side
And here is information about BGP for T0
The whole process of adding EDGA takes a while and has many tasks, so you can easily make a cup of coffee here
It remains to configure AVN
AVN or Application Virtual Network (Segments on NSX) in this case we create for the vRealize solution
We need to select our cluster and the router to which the Segments will be connected
I created an AVN with these parameters
Of course, if you make a mistake, AVN can be removed from the database with two commands by logging into SDDC Manager, but I will not reveal the details here and VMware provides us with a tool to remove the Edge Cluster https://kb.vmware.com/s/article/78635
For my part, that’s it, I hope this article will make it easier for you to have fun in your own labs and I’m going to use this environment for Aria Automation and in the future TANZU. So, to read in new articles and I hope that on a new server under a new domain, so stay tuned, changes are coming 😀
P.S. This is my IP configuration
This is what my DNS looks like
And if I haven’t forgotten anything, here’s a diagram of the whole solution