NSX Load Balancer AutoScale Automation Tool

The NSX LB AutoScale Automation Tool is a PowerNSX script developed by Dimitri Desmidt, NSX Senior Technical Product Manager @ VMware.
The tool is opensource and you can find it on VMware PowerNSX Github: https://github.com/vmware/powernsx/tree/master/Examples
It’s super-easy to configure but very powerful.
In an environment based on vCenter and NSX-v, you can easily leverage this tool to auto-scale (growing and shrinking aka “Elastic LB”) your load balanced pools of VMs based on their load.

What the script does?

  • It checks the load (CPU and RAM) on all LB Pool members;
  • If the load is above a threshold:
    • vCenter Server deploys a new VM (from a template configured like the LB Pool members);
    • NSX Manager adds this new VM to the LB Pool as a new member;
  • When the load is below a threshold:
    • NSX Manager removes a VM from the pool;
    • vCenter Server deletes the VM.

To test the tool I want to use a 3-tier application made by 2 load balanced Web Servers, an Application Server and a DB Server.
I’ll artificially create load on one of the Web Servers to trigger a condition where the AutoScale Tool will deploy a new VM and add it the LB Pool.
As a second step, I’ll reset CPU load to a normal condition to trigger the deletion of the new VM.

This is the high-level architecture of my infrastructure for the 3-tier app:

To leverage the AutoScale Tool, it is important to note the relevant objects in the vCenter inventory.
We will use these elements to configure the PowerNSX script later.

My 3-tier App is hosted in a vSphere Cluster named “Compute Cluster A“.
In the following screenshot you can see the VMs that make the App: 2 Web Servers, 1 Application Server and 1 Database Server.
Highlighted you can see the 2 Web Servers, web-01a and web-02a, members of the Load Balancer Pool we’ll monitor and extend when needed using the AutoScale Tool.

A template VM, configured like the Web Servers already members of the Load Balancer Pool, has been prepared and will be used as a baseline for all new LB Pool members when needed.

The Edge Gateway providing the Load Balancing Services for my 3-tier App is named “OneArm-LoadBalancer-01“.

I’ve configured a Load Balancing Pool named Web-Tier-Pool-01. The 2 Web Servers of my application, web-01a (172.16.10.11) and web-02a (172.16.10.12), are configured as members of the Pool.
In my scenario, the Load Balancing algorithm has been set to Round Robin, but this doesn’t affect the behavior of the Tool.

The Web-Tier-Pool-01 LB Pool is configured as the default Pool for the Virtual Server Web-Tier-VIP-01. The configured Virtual IP is 172.16.10.10.
My application is registered in the DNS as webapp.corp.local, resolved in the Virtual IP address of the LB Virtual Server.

Access to the application shows me that the Web Server web-02a is serving my request.

To test the AutoScale Tool, I artificially create a 100% CPU Usage on web-01a. The expected behavior is that the AutoScale Script will detect the anomaly and will trigger the deploy of a new Web Server that will be added to the Pool.

The Script can be easily scheduled to run as frequently as needed (for example every 1 minute), check the Load Balancer pool members performance and trigger the needed action.

The AutoScale Script is structured with a custom section where you can configure all the needed variables to connect to your environment and define the expected behavior.
The expected settings are very well explained, you can see the details in the following screenshot where I’m showing my configuration.
Note the name of the target Compute Cluster, the name of the Template VM, the name of the Edge Gateway, LB Pool and LB Virtual Server are the one I’ve highlighted in my vCenter and NSX inventory screenshots.

Let’s wait for the next scheduled run.
The Tool detects an 87% average load on web-01a CPU and trigger the creation of a new VM named Web_Clone1.

Here you can see the vCenter task that is creating the Web_Clone1 VM deploying it from the template named Web-Template.

The Script ends with the addition of the new VM to the Load Balancing Pool for my 3-tier application.
I’ve configured the script for a maximum of 3 Pool members and a minimum of 2, this means that starting from the next run it will monitor for the need to shrink the Pool if load will decrease, but it will not add any additional member even if I’ll have additional load. This configuration can be easily change editing the $VM_Min and $VM_Max variables.

Here you can see the additional member (Web_Clone1) added to the LB Pool.

Access to the application shows me that Web_Clone1 is actively serving content for my App.

On the next run of the Tool, web-01a CPU usage is still high. No actions are triggered because my $VM_Max variable is set to 3.

At this point I stop the process causing CPU load on web-01a and wait for the next run of the Tool.
As soon as CPU load is below the threshold, the removal of Web_Clone1 from the LB Pool and its deletion from vCenter are triggered.

My 3-tier App is now back in normal state with 2 Load Balanced Web Servers.

Conclusion: if you’re looking for a simple and very powerful way to manage the Auto-Scaling of your load balanced workloads in a vCenter+NSXv environment, you definitely have to try the NSX Load Balancer AutoScale Automation Tool.

VMware Cloud Provider Program (formerly vCloud Air Network)

Today, August 28th 2017, VMware has announced the new VMware Cloud Provider Program (formerly vCloud Air Network).

You can read the announcement here in the VMware News:
https://www.vmware.com/company/news/releases/vmw-newsfeed.VMware-Helps-Enterprises-Succeed-in-the-Multi-Cloud-Era.2184714.html

And also here in the VMware Blog:
https://blogs.vmware.com/vcloud/2017/08/announcing-vmware-cloud-provider-program.html

vRealize Operations Manager Tenant App for vCloud Director

With the release of the vRealize Operations Manager Tenant App for vCloud Director, VMware Cloud Provider Program Service Providers can now easily offer vRealize Operations Manager as a Service to their vCloud Director Tenants, to grant them self-service monitoring and capacity planning for their vCloud Director Organizations, Organization Virtual Datacenters and vApps.
The new Virtual Appliance comes with the release of vRealize Operations Management Pack for vCloud Director version 4.5.
The vRealize Operations Manager Tenant App is provided as a Virtual Appliance based on Photon OS.
The Tenant App Web Portal is an HTML5 Portal based on the Clarity Design System.
While the Management Pack alone works with vRealize Operations Manager 6.5 and above, to use the Tenant Appliance you need to upgrade your vROps installation to version 6.6.
Supported versions of vCloud Director are: 8.10 and 8.20 (supported and tested), 5.x and above (below 8.10) (supported not tested).
Supported versions of vCenter Server are 5.5 and 6.0.

In this blog post I want to show you all the steps needed to configure this new service and the features it offers to Providers and Tenants.
The first step is to access the VMware Solution Exchange website and download both the Management Pack and the Virtual Appliance.
You can point to the following URL in your Web Browser:
https://marketplace.vmware.com/vsx/solutions/management-pack-for-vcloud-director
The Management Pack is in the PAK format while the vApp is in the OVA format.

To download the Management Pack you must Login hitting the green “Login to Try” button.

After you have downloaded the files, the next step is to access the vRealize Operations Manager Admin Portal in order to install the version 4.5 of the vCloud Director Management Pack.

The next step is to select the Software Update page and click on “INSTALL A SOFTWARE UPDATE…” button.

Click the Browse button the select the right PAK file.

In this release of the vCD Management Pack, the file is named vmware-MPforVCD-4.5-5700881.pak; wait for the check to complete (green flag), then click the “UPLOAD” button.

Once the PAK is uploaded and the green flag “The PAK file signature is valid” appears, you can click on the “NEXT” button.

You must read and accept the EULA to proceed clicking on the “NEXT” button.

The next section, update and release information, is empty. No additional information to read so you can click on the “NEXT” button.

Finally, you must click on the “INSTALL” button to start the installation process.

When the installation completes, the system shows the PAK as installed and the status as completed. This concludes the Management Pack installation. The next step is to configure the vCloud Director adapter provided with the Management Pack to feed vROps with vCloud Director data.

To configure the vCD adapter, you need to enter the vRealize Operations Manager UI.

The next step is to access the Administration section.

In the Administration section you can find the Solutions page, where you find all the installed Management Packs.
You need to configure both the vCloud Director adapter to point to the instances you want to monitor and the vCenter adapter to point to the vCenters that hosts all the Clusters used by vCloud Director as Provider VDCs.
The vCenter Server Management Pack is installed by default with vRealize Operations Manager.
I proceed with the vCloud Director adapter configuration highlighting the solution and clicking the “Gear” button.

You need to provide a friendly name for the instance, the FQDN or IP address of the vCloud Director instance, set auto-discovery to true or false and specify the Organization to monitor (specifying “System”, all the Organizations will be monitored). You can optionally filter discovered data by Provider VDC list and/or by Organization List. You then need to add the credentials to be used to connect, you do this by clicking the green “Plus” sign near the Credentials field.

In the Manage Credentials window, you must specify the credential kind (vCloud Credentials to monitor the instance as Provider System Administrator, Tenant Credentials to optionally monitor a single Organization), the credentials friendly name, the administrative Username and Password.

After you’ve added the credentials, you can test the connection clicking the “TEST CONNECTION” button, then close the Info windows that inform you about the connection test result clicking the “OK” button.

The last action for this task is to click the “SAVE SETTING” button, then the “OK” button to close the Info window and finally the “CLOSE” button. This concludes the Management Pack installation and configuration.

The next step is to deploy the Tenant vApp. Browse to the location where you’ve saved the file vROps-Tenant-App-for-vCD-1.0.0.0-6413280_OVF10.ova you’ve downloaded from the VMware Solution Exchange site and click the “next” button.

Review the Appliance details and click the “Next” button.

Read and accept the EULA, then click the “Next” button.

Select the Name to assign to the vApp and the containing Folder, then click the “Next” button.

Select the target Datastore and the Virtual disk format, then click the “Next” button.

Choose the Network (Port Group) where you want to VM to be attached and click the “Next” button.

Configure all the required properties (Hostname, IP address, DNS etc.) and click the “Next” button.

Review the configured settings, optionally set the flag to Power on the vApp after deployment and click the “Finish” button. If you don’t set the “Power on after deployment” flag, you’ll have to manually power on the VM after deployment.

After the deployment, the vApp Console instructs you regarding the URL to point to access the Portal.

You can now open a Web Browser and point to the Tenant App URL.
As a System Administrator, you must use vRealize Operations Manager administrative credentials to access the Portal. You can subsequently create specific users for each Tenant from inside the Portal.

Once accessed the Portal as vROps admin, you’re presented with the Provider Dashboard.
The System (Provider) Administrator can immediately see the summary of configured Organizations, Virtual Datacenters, vApps and Virtual Machines.
You can also find a Capacity Overview dashboard for the configured Provider VDCs and the list of all Critical alerts, Immediate issues and Warnings.

Scrolling down the Provider Dashboard you have an overview of all the configured Organizations.
You can click the “User” icon for a specific Organization to create a new User.

To create a new User, you then need to click on the “+ USER” button.

Provide the Username, Password, Name, Surname, email address, then click the “Save” button followed by the “Done” button.

The new User is shown in the Organizations Overview.
Note: the User is actually created in vRealize Operations Manager with a limited “Tenant Admin” role. This role has not the privilege to interactively logon into the vROps UI granted to it, and has a limited read-only view of the subset of objects provided by the vCloud Director adapter corresponding to its vCloud Organization.

After a User has been created for an Organization (ACME in my example), the ACME Tenant User can access the Portal. The User is presented with its Organization Dashboard.
The Dashboard shows the summary of all Virtual Datacenters, vApps, VMs and Powered On VMs.
There’s the view of all the Allocated Resources to all Organization Virtual Datacenters. The default view is the combined view of all the VDCs.
In the System Status section the Tenant can see all the Critical alerts, Immediate issues and Warnings for its Organization objects.

The alternative view for the Organization VDC Capacity Overview is the split view, that shows the details about each Organization VDC resource consumption.

Scrolling down the Organization Dashboard you can find the Summary of the Top-5 consumers VDCs and vApps for Compute, Memory, Storage.

On the left “Troubleshooting” Menu, the Tenant can choose the Organization VDCs section to see the list of all the Organization VDCs available in its Organization.
Clicking on any of the VDCs provides the Summary View of the VDC.

In the Organization Virtual Datacenter Summary, the Tenant has a detailed view of the VDC configuration (CPU Limit, Memory Limit, Storage Limit, vApp Count, VM Count and IP addresses used).
The Capacity Overview for the VDC provides details on CPU, Memory, Storage used and available.
The lower section provides a list of all the vApps instantiated in the VDC. You can click on a vApp to obtain the detailed view of its configuration.

The next available section in the Troubleshooting Menu is the “vApps” section. You can click on a vApp to open its detailed summary.

The VM Summary provides all the resources configuration details, the status of the VM, the snapshot size if present (remember that vCloud Director support only  single snapshot per vApp/VM), the list of all the VMs contained in the vApp and some performance counters.

The available CPU Performance counters are CPU Usage (MHz) and CPU Usage (%). You can pinch the chart to zoom in for a more detailed and granular view.

The available Memory Performance counters are Memory Usage (MB) and Memory Usage (%).
The available Storage Performance counters are Average Read Request and Average Write Request.

The available Network Performance counters are Packets Dropped (Count) and Network Rate (KBps).

The VMs section in the Troubleshooting Menu provides the list of all the VMs available in the Organization.

The Alerts section in the Troubleshooting Menu provides the list of all Critical, Immediate and Warning alerts for the Organization.
In the screenshot below you can see an example of an extremely important warning for a Tenant. The Critical Alert provided tells the Tenant that the available CPU resources for the Virtual Datacenter ACME-AP is nearing capacity.

The last available section for the Tenant is the Metric Selector. This is an extremely powerful tool that empowers the Tenant with the capability to obtain the performance graph for any available Metric provided by vRealize Operations for the configured vCloud Director instances. You can filter on Resource Type, Resource Name, Start and End Date to obtain a detailed and granular view of the performance for the selected Metric.

Back to the Provider Dashboard, I want to highlight the main differences in the Menu and available sections, related to the Tenant view. The Provider have the capability to see all the Organization and related objects.
In the Troubleshooting Menu, the Provider has two additional sections: Resource Pools and Provider VDCs. The reason why there are specific sections for Resource Pools and Provider VDCs is that each Provider VDC can be backed by more than one vSphere Resource Pool (usually a Cluster), so it makes sense to have a separate view of these objects.

The Organization Summary gives the Provider a list of all the Organizations and the option to create Users for a Tenant.
Like all the other lists provided by the Portal, you can export it as a comma separated file.

The Resource Pool section provides the list of all vSphere Resource Pools and the related Provider VDCs. As said, a Provider VDC can be made by more than one vSphere Resource Pool. The percentage of utilized resources is provided for each Resource Pool.

The Provider VDCs section provides the list of the available Provider VDCs, with the related vCloud Director instance and the details about resources (CPU, Memory, Storage) limit and percentage of over-allocation.

Clicking on a Provider VDC opens the VDC Summary. A Capacity Overview is provided for CPU, Memory and Storage.

The Provider VDC summary also provides the list of all the contained Organization VDCs and the list of all the Resource Pool that backs the selected Provider VDC.

This concludes this overview of the installation and configuration process for this new tool.
In my opinion this new Tenant App fills an important gap and empowers both Service Providers (to offer) and Tenants (to consume) an IaaS Public Cloud with a powerful monitoring and capacity planning functionality, strictly integrated with the tools and the platform already in place.
This also confirm the big commitment of VMware to the VMware Cloud Provider Program (formerly vCloud Air Network).

VMware Integrated Containers Networking with NSX

Recently I had the chance to work on a PoC on VMware Integrated Containers (VIC).
VIC enables you to work with Docker Containers leveraging the full feature set of vSphere (HA, DRS, vMotion etc).
The logic used in VIC is to map every single Container to a micro-VM. Having a single Container in a VM provides the capability to leverage NSX Micro-Segmentation to secure Applications, and to leverage all the NSX fetures like Edge Gateways, Logical Routers, Logical Switches.
The official VIC documentation can be found at the following URL: https://vmware.github.io/vic-product/assets/file/html/1.1/§
In the official documentation you can find an excellent starting point to understand the VIC logic and mapping to Docker constructs.

In VIC, Containers are created into a Virtual Container Host (VCH), that maps to a vSphere vApp.
The VCH vApp contains an Endpoint VM that provides the Management and Networking functions.
All the Container micro-VMs are instantiated in the scope of a VCH vApp.

This is the Networking logic in VIC. As of version 1.1, each VCH can have a maximum of 3 Network Interfaces.

Public Network: The network that container VMs use to connect to the internet. Ports that containers expose with docker create -p when connected to the default bridge network are made available on the public interface of the VCH endpoint VM via network address translation (NAT), so that containers can publish network services.

Bridge Network: The network or networks that container VMs use to communicate with each other. Each VCH requires a unique bridge network. The bridge network is a port group on a distributed virtual switch.

Container Network: Container networks allow the vSphere administrator to make vSphere networks directly available to containers. This is done during deployment of a VCH by providing a mapping of the vSphere network name to an alias that is used inside the VCH endpoint VM. You can share one network alias between multiple containers.

For the scope of this PoC, I’ve designed the following Architecture:

The main goals I want to achieve are the following:

  • Leverage NSX Logical Switches for Containers Networking (this opens a scenario of easy integration between Containers and “classic” VMs leveraging the NSX Logical Router, for example to Connect an Application Server to a DB Server);
    • A Logical Switch for the Bridge Network;
    • A Logical Switch for the External (Containers) Network;
  • Leverage NSX DFW for Micro-Segmentation between Containers;
  • Leverage NSX Edge Gateway to protect Containers instantiated to an External Network and public facing. The Edge Gateway provides the following Services:
    • Firewall for North/South traffic;
    • NAT;
    • DHCP.

In my scenario, I have the Public Network accessed internally by Developers and the External Network accessed by Consumers.
The Public Network is not protected by an Edge Gateway and leverages the native Docker networking: Containers attached to the Bridge Network are NAT’d to the Public Network by the Virtual Container Host Endpoint VM.
The External Network, where Containers can be directly attached bypassing the VCH network stack, is protected by an Edge Gateway.

Before starting the installation, I’ve created the required PortGroups on my Distributed Switch, shown in the following screenshot.
You can see two “standard” PortGroups backed by VLANs, Docker-Public and External-Consumer, and two PortGroups corresponding to two NSX Logical Switches backed by VXLANs. As of version 1.1.1 there’s not yet native integration with NSX, you need to use the vSphere PortGroup name instead of the NSX native Logical Switch name to instruct VIC to use Logical Switches.

You start the installation of VIC by deploying a Virtual Appliance, provided in the OVA format.
You can see that I’ve created two Resource Pools in my Cluster, the first used for Management workloads, the second used to Host Containers. With this configuration I’m showing that Container VMs and “standard” vSphere VMs can coexist, this is a totally supported configuration. You can leverage different Resource Pools to create different Virtual Container Host. Each VCH can then be managed by different develeper teams with specific Resources, Projects, Registries, Users.

I’ve deployed the VIC vApp, in my case named VIC 1.1.1, in my Management Resource Pool because the vApp is used to manage the overall VIC installation.
The vApp is based on VMware Photon OS and provides the VIC Engine, the Container Management Portal (aka Admiral), the Registry Management Portal (aka Harbor).
From the VIC Management Appliance command line, I’ve used the vic-machine-Linux command with the create parameter to create a Linux VCH.
The command used to create a VCH accepts all the parameters to configure the Bridge, Public, Management, Client and Container Networks to be used for this specific VCH. The Bridge Network must be dedicated to this specific VCH.

After all the checks the deployment and configuration of the new VCH is automatically made base on the provided command line parameters.
I’ve chosen to attach my VCH endpoint to the Public Network with the IP address 192.168.110.180/24.
The Bridge Interface IP address is automatically assigned and defaults to 172.16.0.1/16. An internal IPAM manage the IP address assignment to all Containers connected to the Bridge Network, with a DHCP Scope on the network 172.16.0.0/16.
There’s a check made during the VCH installation process regarding needed ports to be open for communication between the VCH and the ESXi Hosts in the Cluster. You can use the “vic-machine-<OS> update” command to open the required ports on all ESXi Hosts. See here for details instructions: https://vmware.github.io/vic-product/assets/files/html/1.1/vic_vsphere_admin/open_ports_on_hosts.html
The output of the installation process provides all the information you need to interact with the VCH: Admin Portal URL, published ports and the value to be set as environment variables to manage your VCH with the Docker command line.
You must run the export command with the provided information, then I the validity of the environment variables can be checked with the “docker info” command.

You can use the command “docker network ls” to list the available networks for Containers.

In the case you need to delete a specific VCH, you can use the command “vic-machine-<OS> delete” with the appropriate parameters.

After the creation of the first VCH, it can be added to the VIC Management Portal. From the Portal home page, choose “ADD A HOST“.

You need to provide the URL to reach the VCH, the Host type choosing between VCH (VIC based) and Docker (standard Docker), and the credentials to connect to the Host.

After the parameters validation, you can choose “ADD” to add the VCH to the Management Portal.

After you add the VCH to the Admiral Portal, you can see it in the Management/Hosts section. An overview of the VCH status is provided in the dashboard.

With at least one VCH created, you can start to provision Containers. Enter the Containers section in the Portal and choose “CREATE CONTAINER“.

A default Registry is available, pointing to the public Docker Hub which hosts standard Docker images available to be fetched and deployed. From the available images, my choice is to deploy a Nginx Web Server.

In the Network section of the provisioning configuration, my first scenario uses Bridge as the choice for Network Mode, configuring a binding from Port 80 (http) of the VCH Endpoint to Port 80 of the Nginx Web Server. With this configuration, developers will be able to point to the VCH Endpoint IP address 192.168.110.180 to reach the Nginx Web Server. The VCH will automatically provide the needed NAT to the Container IP on the Bridge Network.

Container provisioning can be started with the “PROVISION” button.
On the right side of the Management Portal you can open the “REQUESTS” tab to look at the progress of the deployment process.

The “FINISHED” message informs you about the Container creation completed.

The newly created Container is now available in the Container section of the Portal, with connection details provided in the dashboard.

You have the capability to manage the Container with four specific buttons: Details, Stop, Remove, Scale.

Entering the details of the Container, you have all the details about CPU usage, Memory usage and the Properties.
In the Network Address row, you can see that Bridge in the choosen Network Mode and 172.16.0.2 is the IP address automatically assigned to the Container VM.

Accessing the address of the VCH via http on Port 80 on the Public Network shows that the configuration is correct, the Nginx Home Page is shown as expected.

I want now to deploy a second Nginx Container, this time attached to the Container Network instead of the Bridge Network.

I could do this using the Command Line, but I prefer to follow the UI way accessing the list of available templates, choosing the official Nginx and selecting the arrow on the “PROVISION” button to access the “Enter additional info” section.

From here, I choose to save the Nginx template to create a customized version that can subsequently be automatically deployed on the Container Network.

You can edit the new template using the “Edit” button. This brings you to the “Edit Container Definition” page.

Once in the “Edit Container Definition”, you must enter the “Network” section. In this section you have the chance to add specific Networks that can be leveraged to directly attach Containers to them, bypassing the VCH network stack. You can add a new network by choosing “Add Network” in the “Network” parameter.

Here you can choose an existing network, in my case the NSX Logical Switch I named as Container Network.

I change the template name to make it unique and I save it.

Back in the “Edit Template” section you can graphically see that Containers instantiated from this template will be attached to the VIC-Container-Network NSX Logical Switch.

In the Templates view in the default Registry you can now find the customized Nginx and provision a new Container from it using the “PROVISION” button.

At the end of the provisioning process, the new Nginx Container can be found in the Containers section beside the previously deployed Nginx. The difference between the two Containers is that the first has a standard Bridge Network connection, the second is attached to an external Network, as highlighted in the following screenshot.

Looking at the vSphere Web Client, you can see three VMs deployed in the vch1 vApp (the Virtual Container Host): vch1 is the Container Endpoint, the other two VMs are the two deployed Containers with Nginx.
The highlighted Custom_nginx-mcm430… is the Container VM attached to the NSX Logical Switch VIC-Container-Network. The IP address 10.10.10.10 has been assigned by the Edge Gateway providing the DHCP Service for the Container Network.

Based on the expected Architecture shown at the beginning of the article, I’ve already configured the Edge Gateway with the appropriate NAT and Firewall configuration to publish Services delivered by Containers.
The External, consumer facing interface of the Edge Gateway is configured with the 192.168.140.152 IP Address and has a DNAT configured to expose the Nginx Web Server deployed on the Container Network.
Accessing http://192.168.140.152 correctly expose the Nginx Home Page.

 

Some useful commands you may need to use:
vic-machine-<OS> ls” list all the VMs deployed in the VCH, giving you the VM ID you need as input for additional commands.

An example of command that needs the Virtual Machine ID is the “vic-machine-OS debug”.
This command can be used to enable ssh access to a Container VM and to set the root password on it.

 

Edge Gateway Configuration.

A simple DNAT configuration. The first rule provides DNAT for the first created Custom_Nginx (the one attached to the Container Network).
The second role is pre-provisioned for the next Container I’ll deploy, with a default configuration on a different Port (8080) of the same Edge Gateway external IP address that DNAT to the next allocated internal IP address on Port 80 on the Container Network.

I’ve deployed a second Custom Nginx on the Container Network, reachable pointing to Port 8080 of the Edge Gateway as per DNAT configuration.

Containers Micro-Segmentation:

The importance of NSX Distributed Firewall “Applied To” setting

By default, Distributed Firewall (DFW) rules configured on NSX Manager are applied to all vNICs of all VMs in the vCenter inventory. The only exclusion is for VM added to a specific Exclusion List (see my previous post for this topic: NSX Distributed Firewall Exclusion List)

The DFW configuration provides a section called “Applied To” that enable the Administrator to limit the scope of applicability of a specific rule, providing a so called Point of Enforcement. You can limit the scope of a rule filtering on all objects that NSX Manager recognize for DFW rules configuration (Clusters, Logical Switches etc.)

Leveraging “Applied To” is fundamental in at least two scenarios:

  1. Avoid processing of unnecessary rules on a vNIC in an environment with hundreds or thousand of configured rules;
  2. Provide the capability to manage Virtual Machines with the same IP Address (Multi Tenancy scenario), avoiding the application of wrong rules to objects.

I want to use a specific example to show “Applied To” in action.

I have a 3-Tier Application composed by the following Virtual Machines:

  • web-01a and web-02a, attached to a Logical Switch called Web_Tier_01;
  • app-01a attached to a Logical Switch called App_Tier_01;
  • db-01a attached to a Logical Switch called DB_Tier_01.

Note: Routing and Load Balancing services have been configured appropriately to publish the application but this configuration is out of scope for the purpose of this article.

To avoid unnecessary traffic flows, I create specific Micro-Segmentation rules to grant communication between Application Tiers (and also infra-tier) only on the needed Ports and Protocols.
I want to obtain the following configuration:

  • No communication between VMs attached to the Web-Tier_01 Logical Switch;
  • Only PING and https traffic allowed from any source to the Load Balancer IP Address (172.16.10.10) and to all VMs attached to the App-Tier-01 Logical Switch;
  • Only PING and https (port 8443, Tomcat) from all the VMs attached to the Web-Tier-01 Logical Switch to all the VMs attached to the App-Tier-01 Logical Switch;
  • Only PING and TCP Port 3306 (MySQL) from all the VMs attached to the App-Tier-01 Logical Switch to all the VMs attached to the DB-Tier-01 Logical Switch.

The configuration of the NSX DFW is the following:

By default, “Distributed Firewall” is the value configured for the “Applied To” field. This means that all the configured rules will be applied to all the vNICs of all the VMs.
If we focus on the DataBase Virtual Machine, db-01a (attached to the DB-Tier-01 Logical Switch), you can see the unneeded rules (these rules have nothing to do with the DB machine) highlighted in red and the only needed rule highlighted in green. Let’s keep note of the IDs of unneeded rules: 1006, 1007, 1008.
Note: Using Logical Switches instead of the VM name (or the legacy way, the IP Address) enables you to have a dynamic environment, all the VMs you’ll attach to a specific Logical Switch will implicitly inherit the needed rules.

I want to leverage the NSX Command Line to show the different impact you may have leaving the default “Applied To” instead of (recommended) change “Applied To” to a specific scope.
I will look at the specific vNIC DFW configuration applied at ESXi Host level.

First step, I connect to NSX Manager to leverage the centralized CLI. The first command I run is “show cluster all” to obtain the list of all clusters under the related vCenter management:

The Web, App and DB Virtual Machines are hosted on my Compute Cluster named “Compute Cluster A”. To work on objects using the CLI I need to use their IDs, in this case domain-c33 is the ID of my Compute Cluster.
I run the command “show cluster domain-c33” to obtain the list of all the Hosts that are members of this Cluster:

I decide to look at the configuration of the Host esx-01a.corp.local, running the command “show host host-28“, where host-28 is the ID of my Host:

The output of the “show host” command is the list of all VMs hosted on this specific ESXi Host, with their ID. Our focus is on the DataBase machine, db-01a with ID vm-218.
I run the command “show dfw vm vm-218” to obtain the details of all the vNIC configured in this Virtual Machine. Details are vNIC Name, vNIC ID and Filters.

Filters output shows which filter is applied to the vNIC. In this case the VMware DFW in the Slot 2 of the IOChain. In this slot all the DFW rules are stored and enforced.
I use the obtained Filter to get the list of all rules applied to the only vNIC configured on db-01a.
The command I need to run is “show dfw host host-28 filter nic-39230-eth0-vmware-sfw.2“:

From the output of the command you can see that rules with ID 1006, 1007, 1008 are enforced on the db-01a vNIC.
These rules are not needed for the DB Server and only cause overhead in the overall processing of DFW rules.

To maximize the benefits of DFW, I leverage the “Applied To” section to apply only the needed rules to each specific vNIC.
To do this, I change the “Applied To” field to the value of the relevant objects, this is the outcome:

Web Tier Micro-Segmentation rule will only be applied to the vNICs of VMs attached to the Web-Tier-01 Logical Switch.
The Rule that govern traffic flows from any source to the Web Tier will only be applied to the vNICs of VMs attached to the Web-Tier-01 Logical Switch.
Rules that govern traffic flows from the Web Tier to the App Tier will only be applied to Web-Tier-01 and App-Tier-01 Logical Switches.
Rules that govern traffic flows from the App Tier to the DB Tier will only be applied to App-Tier-01 and DB-Tier-01 Logical Switches.

To check the rules enforced at the vNIC level to db-01a, I run the command “show dfw host host-28 filter nic-39230-eth0-vmware-sfw.2“:

Not needed rules are not applied to the DB Virtual Machine vCNIC, avoiding unnecessary overhead in the Firewall rules processing.
This shows one of the most important benefits of the “Applied To” section of the NSX Distributed Firewall.