How to build a 250 node VMware vSphere/ ESXi lab environment in AWS for testing

This blog was written with guidance from Scott Lowe around best practices of VMware data center design and automation.

As most of you are aware, we recently announced the public beta of a new feature that allows users to run VMware ESXi on AWS or Google cloud. Essentially, we have implemented Intel VT/ AMD-V functionality in software in our hypervisor, HVX. That makes the underlying cloud look like real x86 hardware - complete with silicon extensions required to run modern hypervisors like ESXi and KVM. In this blog, I am going to illustrate how to set up a large scale, 250-node VMware ESXi data center in AWS for less than $250/hr. We believe that this could be extremely useful for enterprises for upgrade testing their vSphere environment or for new product and feature testing.

Prerequisites

1. If you do not already have an account in Ravello, please first open a trial account.
2. A VMWare ESXi VM image in Ravello Library (described in another document).
3. A VMWare vCenter server VM image in Ravello Library (described in another document).

The first ESXi host

Adding the first ESXi

1. Create an empty application (do not use any blueprint) in Ravello and give it a name.
2. Add the vCenter server VM from the library and publish the application (note - it takes a few minutes to publish and few more minutes for the vCenter and all its services to go up). While this is being done, please continue to the next item (adding ESXi machine).
3. Add one ESXi VMs from the library and update the application (note - it takes a few minutes to this kind of update and few more minutes for the ESXi machine and all its services to go up) and name it "firstesxi"( You will need to change the hostname (only in the Ravello web - defined in "General" tab->"Hostnames") to "firstesxi")). While this is being done, please continue to the next item (adding NFS machine).
4. Use the VM named "NFS" from Ravello library to add an NFS server to your application and update your application in order to publish the NFS machine (you will need to select a key pair for that). This NFS machine contains some images to later on create virtual machines in your cluster. Note - You can use your own private NFS image (there is another blog describes how to create such an NFS image). If you choose your own private NFS image please adjust later steps accordingly.
5. After all machines are published and finished booting, login to the vCenter server web (HTTPS on port 9443). The credentials are the same as when you saved your original vCenter into the Ravello library.
6. Add a new datacenter.
7. Add the ESXi host to the datacenter using its Ravello hostname "firstesxi".
   
   The credentials are the same as when you saved your original ESXi into the Ravello library. Note - you will see a yellow warning regarding your host because it does not have any datastore. Later on in this document, When you will add NFS datastore, this warning will disappear.

Configuring ESXi host to use NFS

1. Select the ESXi machine.
2. Browse to "Related Objects" tab.
3. Browse to "Datastores" tab (see screenshot).
   
4. Click the "Create a new datastore" button.
5. When asked to select either VMFS or NFS datastore type, select NFS and click Next.
6. In the "Server" edit-box insert the hostname of the NFS server (should be "lio1").
7. In the "Folder" edit-box insert "/nfs" (see screenshot) and click "Next".
   
8. Continue and finish the wizard.
9. You can now see that the yellow warning sign regarding no local datastore has disappeared.

Optional - Setting virtual distributed switch

In order to configure virtual distributed switch, follow the instructions on this article. Note - each ESXi in your datacenter has 2 NICs:

1. The first one is used for management.
2. The second NIC is used for data.

It is recommended putting the 2nd interface on a virtual distributed switch.

Note - You can decide whatever your networking setting are preferred. Just please notice that some network configurations are not supported. If you prefer another networking configuration than described above, when it comes to Host Profiles and specifically Host Profile Remediation - it might fail/make your ESXi unreachable.

Extract host profile

1. Right click on the ESXi host.
2. "All vCenter Actions->"Host Profiles->"Extract Host Profile..."
   
3. Give the profile a name (for example "myprofile") and click "Finish".

Creating a VM to run on your ESXi

Get an OVA image to deploy to your local machine

In order to deploy VMs in vCenter, the most efficient way is to deploy an OVA template. You can download such templates of free OS (such as Ubuntu, Fedora, etc) from the internet.
In addition, in the NFS machine, we have uploaded a few OVA machines, for example Ubuntu 14). To Download from the NFS machine you will need to:

1. Using vCenter web interface, browse to the datastore.
2. Select "Manage" tab.
3. Select "Files" tab.
   
4. Right click on the OVA file and click "Download from Datastore" to save the OVA file. Note - you might need to install a plugin for your browser for that. Pay attention to popups/blocked windows during the plugin installation.

Deploy the VM

1. Right click on the ESXi host.
2. Select "Deploy OVF Template..."
3. Select the OVA file from your local machine
   
4. Complete the wizard and wait few minutes until VM is deployed.
5. Power on the VM.
6. Using vCenter web interface- open the VM’s console and login with user and password associated with the OVA template (can be found in the "Notes" section in the "Summary" tab). Note - you might need to install a plugin for your browser for that. Pay attention to popups/blocked windows during the plugin installation.

Configure the network of the VM

Static IPs

1. Need to configure the default gateway to Ravello’s default gateway on the network of the application (usually 10.0.0.2).
2. Set the netmask/network in such a way it will be exactly the same as the Ravello’s network (For example 255.255.0.0).
3. Set the static IP of the machine to a "high unique ip" that it will not conflict with other Ravello VMs in the same application (For example 10.0.100.1).
4. Set the DNS to known public DNS (like 8.8.8.8).

To test the networking - Open the console and ping from your vm some address in the internet, like www.google.com.

DHCP

Not supported for now

Install VMWare tools

If VMWare tools is not installed on the VM, it is important you will first install VMWare tools.

Create a cluster with many hosts and VMs

This is the most important part in this document. We will describe here how to build a big datacenter using scripts, automation and multi-selection actions in the GUI.

You can repeat the following steps few times and create several big clusters. Each cluster can contain tens of ESXi hosts and hundreds of VMs. Here are some nice screenshot of a datacenter with 4 clusters, 64 hosts per cluster and around 500 VMs in total.

Create a new Cluster

1. Please give it a simple name. For example "ClusterA".
2. Turn on the cluster’s DRS (by default - Fully automated).

Add several ESXi hosts

1. First you need to know your application ID and your ESXi template vm id.
   1. Your application ID can be found in the URL of your Canvas view (for example) of your Ravello application:
      https://cloud.ravellosystems.com/#/apps/YOUR_APP_ID/canvas?isNew=true
   2. Your ESXi template ID can be found when also in the URL when browsing to "Library->VMs"
      https://cloud.ravellosystems.com/#/library/vms/?vmIds=;YOUR_TEMPLATE_ID
2. Then, download publish_vms_from_template_into_existing_application.py script from Ravello’s GitHub (https://github.com/ravello/vmware-automation).
3. Then, run the script to deploy several ESXi hosts. In this example I have created 16 ESXi machines named "esxiA1".."esxiA16".
   
   python publish_vms_from_template_into_existing_application.py -t 56232441 -u "ohad@ravellosystems.com" -a 56558267 -b esxiA -n 16
4. Script should print "Success" within few few seconds. Then it takes ~5 minutes for ESXi machines to be deployed.

Add ESXi hosts to the cluster - Using PowerCLI

Install PowerCLI

Installation instructions here.

Connect to your vCenter

Connect-VIServer -Server vcenter -Protocol https -User root -Password vmware

(you can ignore the yellow warnings if such appear)

Whereas:

1. vcenter - indicates the hostname of your vCenter machine.
2. root and vmware are the credentials for your vCenter machine.

Add ESXi hosts to the cluster

1..16| Foreach { Add-VMHost esxiA$_ -Location (Get-Cluster -Name "ClusterA")[0] -User root -Password esxpassword -Force -RunAsync}

Whereas:

1. 1..16 - indicates that we want to add 16 hosts.
2. esxiA$_ - indicates that that hosts names to add are "esxiA1".."esxiA16".
3. "ClusterA" is the name of the cluster in which hosts will be added.
4. root and esxpassword are the credentials for your ESXi machine.

Apply host profile to the new cluster

Right click on the cluster - "Attach Host Profile..."

Select all hosts in the cluster

Again, select all hosts in the cluster. Right click and "All vCenter Actions->" "Host profiles->" "Remediate..." and then continue the wizard (might take a while for "Validation")

Clone VM several times - Using PowerCLI

Now, that you have a nice cluster, it is time to deploy some VMs on it. In this document we will describe how to fastly clone a VM into the cluster several times (using a "linked clone").

Install PowerCLI

Installation instructions here.

Connect to your vCenter

Connect-VIServer -Server vcenter -Protocol https -User root -Password vmware

(you can ignore the yellow warnings if such appear)

Whereas:

1. vcenter - indicates the hostname of your vCenter machine.
2. root and vmware are the credentials for your vCenter machine.

Clone VMs using a linked clone

$sOriginVM="Ubuntu 14.04"
$sOriginVMSnapshotName="mastervm_linkedclone_snap"
$oVCenterFolder=(Get-VM $sOriginVM).Folder
$oSnapShot=New-Snapshot -VM $sOriginVM -Name $sOriginVMSnapshotName -Description "Snapshot for linked clones" -Memory -Quiesce
$oESXDatastore=Get-Datastore -Name "Datastore"
$oResourcePool=(Get-ResourcePool -Location (Get-Cluster "ClusterA"))
1..25| Foreach {New-VM -Name UbuntuCloned$_ -VM $sOriginVM -Location $oVCenterFolder -Datastore $oESXDatastore -ResourcePool $oResourcePool -LinkedClone -ReferenceSnapshot $oSnapShot}

Whereas:

1. 1..25 - indicates that we want to clone 25 vms.
2. Ubuntu 14.04 - indicates that the original vm to clone is named "Ubuntu 14.04".
3. UbuntuCloned$_ - indicates the name of the target cloned VMs, "UbuntuCloned1".."UbuntuCloned25" .
4. Datastore - indicates that the NFS datastore is named "Datastore".
5. ClusterA - indicates that the target location is in a cluster named "ClusterA"

Then, Using vCenter web interface, select all vms and power on.

Set the VM order accordingly in Ravello GUI

Storage VM and vCenter VM need to start first (before all ESXi machines) and shut down last (after all ESXi machines).

Optional - Monitoring crucial vms

It is recommended that you will install a monitoring tool to monitor crucial machines in your datacenter, such as NFS and vCenter server. It helps a lot understanding how the datacenter behaves and how you can stretch its limits.

I will demonstrate graphs with New Relic.

Saving application to blueprint

How to create a blueprint

Known issues/limitations

1. The ESXi network interfaces in Ravello are having some problems working with VMXNet3. Please use only E1000 for the VMs network interfaces and for the ESXi network interfaces.
2. There are some limitations running some guests (such as Ubuntu 14) on GCE. Please use Amazon EC2 cloud when publishing your application in Ravello.

The post How to build a 250 node VMware vSphere/ ESXi lab environment in AWS for testing appeared first on The Ravello Blog.