Nested VCF 4.5 - Step by Step

Recently, I managed to replace my worn-out Lab with something that has a little more power, I changed the memory from 92GB to 1024 ?.

So, due to the greater computing power, I decided to set myself a solution that has been tempting me for a long time and I dealt with its part through my career. I’m talking about the VCF here. The choice fell on VMware Cloud Foundation version 4.5.

This solution includes the following products

As you can see, we have a lot of fun ahead of us, but some of it is already known to you / us, so I will focus here on the VCF

The new version does not yet have an official release date, so I focused on the latest available. However, before I devoted time to deployment, I decided to explore the topic a bit. However, if you want to set up a nested environment because I will put it in this article, you will not find too much material. Most of the articles that have been written refer to the VLC solution, which by the way is a super-prepared tool, we really have to specify a few variables and the whole environment is ready to work. However, I wanted to play/learn, so I decided to bet everything myself without automation.

Host/vCenter

The element that I configured before I started any fun with VCF is a physical ESXI managed by vCenter (so that you can create a template, but it is not required at all)

On the host we create two portgroups or on vCenter we add VDS. I left Management on a regular vSwitch, and for the whole VCF integration I created a separate VDS that has two portgroups.

The first is the portgroup for Cloud Builder, unfortunately, with the OVF deployment, we do not have the option to set the VLAN, so I created Portgrupe vLAN, which adds the Management VLAN ID (1611) to the Cloud Builder frames. The second portgroup is already a trunk portgroup with the range 0-4094, i.e. allowing all VLANs. In addition, I increased the MTU to 9000 and changed all security options to Accept

Hosts

The first element I prepared were ESXi hosts. I created a clean virtual machine template with the following parameters

The drives are Thin and I will mention network adapters later. An additional configuration that I made on the template is to add 3 parameters in Advanced Settings. As for the processors, at a later stage when you do a vra deployment in the Medium version, the machine does not start because they need 12 processors and we only have 8. However, this is a limitation of my physical equipment and we can’t help here.

featMask.vm.cpuid.PDPE1GB = Val:1
scsi1:0.virtualSSD = 1
scsi1:1.virtualSSD = 1

The first parameter concerns the support of hugepage by processors, my machine does not support it, so it was necessary to add such a value so that NSX-T Edge could be run in the nested environment. The next two parameters concern the marking of drives connected to esx as SSD (Flash)

There is one more element that I have not mentioned, more precisely in the processor, we must remember to select Expose hardware assisted virtualization to the guest OS. From such a template I built 4 machines that will work as my Management Domain

More specifically in the documentation

Management Domain
A cluster of physical hosts that contains the virtual machines of management components, such as vCenter Server, NSX-T Data Center, SDDC Manager, and so on.

After configuring 4 machines with ESX, i.e. the system installed on the smallest disk, I set

IP address from my subnet
I set the VLAN to 1611 because that’s what it uses (see diagram)
I have disabled IPv6
set DNS and hostname in the form of FQDN
removed Custom DNS Suffix

After restarting the host (required if disabling IPv6) I generated a new certificate with the command

/sbin/generate-certificates

Here is the link to the documentation https://docs.vmware.com/en/VMware-Cloud-Foundation/4.5/vcf-deploy/GUID-20A4FD73-EB40-403A-99FF-DAD9E8F9E456.html

After this configuration you should login to GUI and on Portgroup VM Network also set VLAN 1611

The last element that needs to be configured here is NTP

NTP must be configured to start with the host as well as point to a specific NTP host (in my environment it is pfSense, which I will describe in a moment)

pfSense

pfSense is a virtual router that allows us to configure VLAN, Firewall, NAT, BGP and many other services that we can use in our environment. I used it for BGP and VLAN which it uses in the entire virtual infrastructure

The installation of pfSense from ISO itself is not complicated, so I will not write here how to do it. The machine I made for this router has the following parameters

Network Adapter 1 -> VM Network our WAN
Network Adapter 2 -> PG-allVLAN our LAN

After installing the machine, we have access to the console where we set the IP address

And then we can configure everything in a very friendly GUI

a) The first element that will actually be used at the very end but I “turned it on” at the beginning is to install the frr package which will later allow us to configure BGP

Go to the system tab and then package manager in available packages, look for frr and click install

When will the package be installed. We leave it alone until we get to the BGP configuration

b) The second thing I configured is the VLANs in the whole clean and simple VCF deployment we need 7 of them. We all create the same way.

Go to the Interfaces -> Assignments tab

And then to the VLANs tab

In the lower right corner, press Add and add additional VLANs as follows

In my environment, I have configured the VLANs as follows

With VLANs configured in this way, go to the first tab -> Interface Assignments and click Add to add all vlans as a new interface.

The interfaces added in this way must be configured, i.e. set the IP as Gateway and specify the MTU. All ports have Gateway set to 1. The first octet is 172. And 2 and 3 are the VLAN ID. For example, the first interface that will be my management VLAN

Having configured the interfaces in this way, I went to configure the Firewall, and more precisely, I passed all traffic through it as Allow, because it is a LAB, I do not need to wonder where my traffic is blocked.

c) We set the firewall in the Firewall Rules tab

Each port on the router has its own firewall, so on each port I added a rule using the Add button for any protocol, any source and any destination on any port Allow and on pfSense it looks like this

d) After configuring the firewall, I went to configure the DHC for our NSXs that will be deployed as part of the VCF

DHCP is configured in the Services -> DHCP Server tab

In my case, I fired DHCP on the OPT4 interface and it is VLAN 1614 for nsx-overlay. There is nothing complicated about the DHCP setting. We only need to do enable and specify the range

e) The element that I set to pfSense and is required for basic VCF deployment is NTP

In the services tab, go to NTP

What we need to do here is to enable NTP, choose which interface it will work on (I chose all available) and specify the Time Servers from which our Server will download the time

Such configured pfSense will allow us to deploy VCF from Cloud Builder. However, I want to have an environment where I will create various games from the vRA level, so I will show you how to configure two elements that will be useful in the future.

f) During the deployment of the EDGE cluster, we must choose the routing protocol, either we will create static routes or we will use the dynamic routing protocol, which is BGP (I will not tell you anything more about this because network matters are not my thing)

To configure BGP we need to have the package I showed you earlier installed. As we have the package installed, we go to the FRR Global Zebra tab available in the Services panel

Check Enable next to Enable FRR, set the Default router ID, I chose an ID that points to the 250th host in the subnet for which I will actually be routing.

As you can see, for more information on this topic, you can jump to Wikipedia

The next element is the master password which is mandatory

The last element that I configured here is Next hop Tracking where I set the IPv4 value for Resolve via default route

Nothing else for the Global tab so we can save and go to Route Maps. Here I added one RouteMap where I set it to permit and sequence value to 100

I didn’t change any other value from default. After pressing save and clicking on BGP we have a new bookmarks panel. This is where we configure BGP itself

We need to enable this routing protocol and set the local AS in my case it is 65001

At the bottom of this page in Network Distribution I set everything to be redistributed and the network and all possible networks by setting the subnet to 0.0.0.0/0 and selecting the previously created RouteMap

Once again, press Save and move on.

In the Neighbor tab, we add our neighbors, in my case, these are 4 interfaces that will be attached to EDGE when deploying Edge Cluster

When adding, in General options, we enter the Address and password with the appropriate option, which we will later provide when configuring Edge

Below, in Basic Options, we set Remote AS, which is something that will be on Edge in my case, it is 65003 and I selected the option Address Family -> Allow neighbor to advertise and receive routes for both IPv4 and IPv6

The last element that I have configured here is Peer Filtering. In the Route Map Filter tab for inbound and outbound traffic, I selected our previously created route map

Once configured, we can press save. Even though our neighbors haven’t been made yet. By going to the Services status tab, we can check if our BGP has started. If the daemon does not work, it means that we have something broken in the configuration

At this stage, we could once again say thank you, but I personally configured one more element, namely NAT, which in my network is required for xRegion which will be afforded by vRA, LCM, etc. Solutions.

g) Here is a simple matter. In the Firewall -> NAT tab

Go to Outbound and change the mode from Automatic (i.e. all vlans are automatically natated, good job pfsense, I don’t have to click it myself) to Hybrid which will allow us to add our own mappings

I added my xRegion subnet here so that I can access the Internet through all my routers

And now with a pure heart I can say that we have prepared and configured a router for VCF

Cloud Builder.

Cloud Builder is a tool that does automagically ? deployment of the entire environment based on Excel which we fill with data. From what I’ve noticed, this version of Excel 4.5 is quite truncated.

However, before we move on to the Excel deployment, our Cloud Builder must be placed in our environment, but it is a regular deployment with OVF, so most of you can do it with your eyes closed. After launching the browser and entering the url of the cloud builder, of course logging in, the wizard appears

Regardless of whether we are doing a nested or physical environment, everything here works the same way.

We click Next

We accept that we have met all the requirements that are given

Of course, let’s not kid ourselves that there is a Validator because the first time we always forget something ?

We click next

Now we have the option to download excel or if we have it already prepared, we can upload it. According to the instructions, we can also use JSON, but I’ve never seen it, so I’ll focus on excel here

There are 3 tabs in the file in which we need to complete the data

Outside of Credentials, this is what my excel looks like. It’s almost default

After uploading the file to the wizard, Cloud Builder performs an environment check

Here we have, for example, checking if all gateways are reachable, if the hosts are configured correctly, if the thumbprints match, if the times are correct, etc. etc.

After fully validation

As you can see, I didn’t run it only once ? When we have everything green, we go to the installation process, which is cool that the validation went green, the installation process can fail.

In my case, however, this was not the case and after a few hours my deployment was over

And after pressing the finish button, a message appeared to my eyes

And here a new world appears to us ? where the fun has no end.

However, before we continue to play, I have configured the network. Therefore, we enter the tab on the left. Inventory -> Workload Domains

And click on our domain that Cloud Builder created

Here we have information about hosts, services and NSX access

In the Actions menu, select Add Edge Cluster

we provide all the parameters and Cloud builder on NSX creates an environment for T0 and T1 Routers

In these routers, the BGP that we configured earlier is used. Parameters we need to fill in

In the routing settings for T0, we create the ASN that we set on the pfSense side

And here is information about BGP for T0

The whole process of adding EDGA takes a while and has many tasks, so you can easily make a cup of coffee here

It remains to configure AVN
AVN or Application Virtual Network (Segments on NSX) in this case we create for the vRealize solution

We need to select our cluster and the router to which the Segments will be connected

I created an AVN with these parameters

Of course, if you make a mistake, AVN can be removed from the database with two commands by logging into SDDC Manager, but I will not reveal the details here and VMware provides us with a tool to remove the Edge Cluster https://kb.vmware.com/s/article/78635

For my part, that’s it, I hope this article will make it easier for you to have fun in your own labs and I’m going to use this environment for Aria Automation and in the future TANZU. So, to read in new articles and I hope that on a new server under a new domain, so stay tuned, changes are coming 😀

P.S. This is my IP configuration