vmware – The Blog of Nitin J Mutkawoa, AKA @TheTunnelix

Tag: vmware

Upgrading firmware of HP ProLiant Gen9 on VMware infrastructure

05/09/2019 Nitin J Mutkawoa Add Comment

Some days back, I was having a conversation with a friend about the recent CVEs that impact the firmware of several physical machines. HP usually will release several vulnerability alerts and it is very important to patch your firmware. You might also notice that the mitigation happens at CPU levels such as AMD or Intel. In this blog post, I will focus on one of the very basic ways to update your firmware. Prior before upgrading, its important to make a checklist. My HP Proliant Gen9 is actually an ESXi on VMware infrastructure. You can view the step by step actions below and pause the video at any time. I have blurred some information for security purpose such as the name of servers, IPs, Logins, etc..

A basic checklist can be considered as follows:

Monitoring consideration.
How many and size of the VMs on production.
The consistency of the Firmware provided by HP.
Logins and Passwords for the HP ILO, vCenter, Virtual Machines running, etc..
Java or Dot Net framework for accessing the ILO.
Where is the storage node of the virtual machines?
Load on the cluster or the ESX itself.
The expected amount of time during migration.
The output of the update (Correction of bugs, New feature, etc..).

1. Prior before upgrading the firmware, you need to make a survey about the oversized VMs. Consider performing a manual migration before activating the maintenance mode. Then, enter maintenance mode, all the virtual machines in the cluster should migrate to other physical machines in the cluster. We assume that the datastore of each machine is not on the physical machine itself which is not recommended.

2. Once all virtual machines migrated to other ESXi hosts, connect to the HP ILO onboard administrator interface. Consider checking the health status of other ESXi hosts on your chassis.

3. Also consider, verifying the system information of your ESXi host (HP physical machine).

4. On the ‘information’ tab, click on ‘system information’, you will notice the field ‘Integrated remote console’. You can choose any framework whether Java or .Net to open the console.

5. Once connected, you should be able to see the following screen.

6. From vCenter, upload the image file which constitutes of the patch for the new firmware.

7. On vCenter, right click on the physical machine, then ‘reboot’. Consider checking the grey bar that is now blinking on the left just below the ‘<F2> tag ‘at the bottom on the ESXi console.

ESXI-Console-blinking — ESXi-Console-blinking

8. By the time, you should also notice that you have been logout on vCenter.

9. Normally, after a few minutes, the server will reboot showing the HP Enterprise logo followed by other system information, then you will notice a screen that with four key options below: F9 (System Utilities), F10 (Intelligent Provisioning), F11 (Boot Menu), and F12 (Network Boot). Hit the F9 button to enter ‘System Utilities’.

10. Choose the ‘one-time boot menu’ option. Then go to the USB virtual disk that you have mounted at step 6 and hit ‘Enter’.

11. By now you should notice the installation of the firmware in progress. This might take a considerate amount of time. Monitor the installation.

12. After extracting the iso file, it will go through three steps: Inventory, Review, and Deployment.

13. You can also monitor for the ‘blink’ message on the HP Onboard Administrator interface which means that the upgrade has not completed yet.

14. Once, the UID state is off, you can remove the server from maintenance. On vCenter, right click on the server, and click on ‘Exit maintenance mode’.

15. Several machines will now join the ESXi host which has been added back to the cluster through an election process.

If you are interested more on the election process and how High Availability works, please check the article ‘VMware vSphere High Availability‘ which I published several months back.

All steps from 1 to 15 have been described in the video below. If you like the article please click on the like button and share.

[yotuwp type=”videos” id=”P6lh6pyTBfs” ]

Virtualisation CloudComputing, ESXi, HP, vmware, vSphere

VMware vSphere High Availability Basics

07/22/2017 Nitin J Mutkawoa Add Comment

VMware vSphere HA is one of the core feature in a cluster. So let’s bring some more precision about it. High Availability – HA enables a cluster of ESXi hosts to work together so that they can provide high levels of High Availability for virtual machines rather than just an ESXi host by itself. In brief, the High Availability feature is provided by pooling virtual machines and the ESXi hosts in the cluster for protection. Some examples could be host failures, host isolations and application crashes. The requirements for HA is a minimum of two hosts, vCenter Server and Shared Storage.

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

One ESXi goes down

By default, HA uses management network (Service Console/Management Network VMkernel connections). Let’s take a scenario where there are three ESXi hosts in a cluster. In the event where a physical server (ESXi hosts) goes down, the VM machines will be restarted on the other ESXi hosts. We can also set up applications to be started on the other physical server. From the three physical servers in the cluster one is going to be elected as master. The master server is going to keep track of other ESXi hosts through the heartbeat of other servers. This is done at the management network level. The master server will always expect to have heartbeat responses from other ESXi hosts.

Only the management network went down

If at any moment, the master server detects that a host is down, it will report that to the vCenter server and all servers will be powered on the other ESXi hosts. What is more interesting is that if only the management network goes down, and other network such the datastore network is still working, that would be referred as an Isolation incident. In that case, the vSphere will communicate to the master server and will claim that the ESXi host is still active is through the datastore heartbeat. In that case, the VMs will not be powered onto other ESXi host because it is an Isolation incident.

Only the Datastore network went down

Now, what if only the Datastore network went down and not the Management network? The master server will still receive heartbeat messages from other ESXi hosts, but no data communication is being sent to the datastore. Another element that is included in HA is VMCP – VM Component Protection which is a component that detects that if a VM is having access to the datastore. In the event of failure messages from the datastore heartbeat, the VMs will be powered onto other ESXi hosts where the datastore is sending alive heartbeat messages.

In all three scenarios, HA implies downtime as servers will be restarted in other ESXi hosts, but same is usually done within minutes. Another point to keep in mind is that HA applies only to physical host. For example, if a particular VM encounter a BSOD or Kernel Panic, HA will not know about it because the Physical server (ESXi host) is still communicating with the master server.

How the election process takes place to become the master?

When HA gets activated in the vSphere, the election process takes around 10-15 seconds. In that process (Enabling HA) an agent gets installed to activate HA which is called FDM – Fault Domain manager. Logs can be checked at /var/log/fdm.log. The election process is defined by an algorithm with two rules. For the first, the host with access to the greatest number of datastores wins.

Now, what if all ESXi hosts see the same number of datastores ? There will be a clash. This is where the second rule kicks in i.e; the host with the lexically-highest Managed Object ID (MOID) is chosen. Note that in vCenter Server each object will have a MOID. For example, objects are ESXI servers, folders, VMs etc.. So the lexical analyzer is a first component where it takes a character stream as input, outputs a token which goes into a syntax analyzer and the lexical analysis is performed. Care must be taken when attempting to rig this election because lexically here means, for example, that host-99 is in fact higher than host-100.

What IF …. ?

So what if vCenter Server goes down after setting up HA?

The answer is HA will still work as it now the capacity to power on the vCenter Server. FDMs are self sufficient to carry on the election process as well as to start the vCenter Server. FDMs are inside the VMs but not inside the vCenter Server.

Enable and Configure vSphere HA

I will be using the free labs provided by VMware to set up HA.

1.The first action is to choose the Cluster then click on ‘Actions‘ then ‘Settings‘.

2. Choose ‘vSphere Availability‘ on the left -> then click on ‘Edit‘.

3. Click on ‘Turn ON vSphere HA’.

4. Choose ‘Failures and Responses‘ option and click on -> and enable ‘VM and Application monitoring‘.

5. On the ‘Admission control‘ -> check the ‘Cluster resource percentage‘ option.

[google_ad data_ad_slot=” data_ad_format=’rectangle’]

6. Click on ‘Heartbeat Datastores’ and select ‘Automatically select datastores accessible form the host‘.

7. From the ‘Summary’ tab click on ‘vSphere Availability‘, it should mentioned vSphere HA: Protected.

REFERENCES and CREDITS:

1.VMware Tech Plus:

VMware official Youtube educational channel

2.VMware White paper:

VMware vSphere HA WhitePaper

3.VMware Labs:

Hands-On-Lab by VMware

4.Other Links:

(i) kb.vmware.com-Error during HA installation

(ii) Lexical analysis – IIT Bangalore

Virtualisation CloudComputing, vmware, vSphere

ESXi installation on my Dell Laptop and hands on VMware Labs

07/22/2017 Nitin J Mutkawoa Add Comment

If you are thinking why i should install a bare metal hypervisor on a laptop, i assure you its just for educational and testing purpose only. I noticed that it was quite difficult for me to get this done. However, after some research it looks that my Dell Inspiron n5110 motherboard will not authorised me to install ESXi 6.x. Probably, it looks like there are some drivers missing or the motherboard does not support it.

Here is what my processors looks like from the configuration menu on VMware vSphere Center

Anyway, i have been able to inject some network drivers – VIB files into the ESXi5.0 which allowed me to install the ESXi 5.0 on the laptop. You can follow the instructions at the link how to make your unsupported NIC work with ESXi. Once installed, VMware will provide you with a two months free trial before you purchase the license.

Another way of messing around VMware Vsphere is to deploy a lab from labs.hol.vmware.com That’s so easy to deploy labs and access the VMware vSphere web client. All credentials will be available on the readme.txt file found on the desktop. Also a lab manual will be shown alongside whilst working on the environement labs.

I am sure this would help anyone to get into hands on lab quickly and it would be a nice start for beginners.

Virtualisation vmware

Protecting Data and Applications with Zerto DRaaS

03/23/2017 Nitin J Mutkawoa Add Comment

Zerto can be used to protect Data, Sites, Applications and Files. The best-in-class replication, orchestration and automation that ZVR provides is made by VPGs – Virtual Protection Groups. With Zerto, virtual machines are protected in VPGs, which are a consistency grouping of VMs you want to protect and recover together. For example, a VPG for an application like Microsoft Exchange might include the VMs for the software, database and web server.

What is a VPG ?

Is complete application protection and recovery
VM and VMDK level consistency groups
Protects across servers and storage locations
Fully supports vMotion, Storage vMotion, HA and vApp
Journal-based point-in-time protection.
Works with Group Policy Protection
Has VSS Support

To create a VPG click on the VPG button which will list the VPGs from both the local and peer sites and provides summary details of each VPG. To create a new VPG, you have to click on the “New VPG” button.

This will bring you to the Create VPG wizard. Once you put a name for the new VPG click on next. Here is an example of the VPG wizard. On the VM’s tab, we are going to see a list of unprotected machine which is in the production site. Once you choose the VM, you click on the arrow in between and can also choose the boot order of each VMs. Then click next to move to the Replication tab.

The Replication tab covers details of the recovery/target site as well as the default value to use for the replication. If you have multiple site from the ZVM it can be selected in the Recovery Site option. The journal is a powerful feature. The journal is a dynamic record of every checkpoints in the history according to the values specify here. By default the value is one day and a journal is always on and stored on recovery site. If production is down, failovers are still possible by not writing changes locally first. The WAN traffic compression allows data to be compressed first before data is transmitted to the recovery site.

The ZVR journal is dynamic record of every checkpoint in a VM’s history. Checkpoints are inserted algorithmically and allow you to rewind to seconds, minutes or hours in the past to a max of 30 days. A journal is always on and only stored on the recovery site. If production is down, failovers are still possible and you also save storage space by not writing changes locally first.

The storage tab list the VM in the protection group and are associated with volumes details. If a VM is on thin provision during creation, ZVR uses that same thin provision or dynamic disk set up here by default. The “Edit Selected” button allows you to change this and to replicate swap disk if needed. Storage can also be moved from one datastore to another.

The recovery tab is for specifying defaults network and any post recovery scripts. You can specify the default recovery network and default recovery folder. For example:

The NICs tab allows you to specify specific network and IP settings for each of the recovery VMs. By default ZVR keeps the same MAC and IP addresses as the production site. So each VM need to be edited if a different IP is needed.

The backup tab is turn off by default whereas the summary tab gives an overview of all the settings chosen.

You also create a repository for offsite backup. The repository is created by clicking on the Setup tab then on New repository.

As from here, you can click back on the backup tab from VPG and toggle the backup option ON.

Backup can also be run manually from the offsite backup menu. The offsite menu will appear once you activate it on the VPG backup menu.

Virtualisation CloudComputing, vmware, Zerto

Initial Configuration of ZVR – Zerto Virtual Replication

03/21/2017 Nitin J Mutkawoa Add Comment

In the last article we have an overview of the basics of ZVR and its installation. We will now see how to install VRA – Virtual Replication Appliances as the first step of initial configuration. VRA sits on each ESXi host and performs always on block level replication. Zerto can also replicate on different hypervisors. During installation of ZVM or VRA, there is absolutely no downtime. Here is an example of a Production and Recovery site showing where the ZVM and VRA is located.

A VRA is simply a virtual machine that is installed alongside the virtual machines on the host. It is a custom 64-bit Debian version of Linux and is designed for security and a small footprint in terms of disks, memory and CPU. Exact requirement for VRA installation varies depending on the hypervisor. Here is the requirement:

The following information are needed before installation of VRA.

The network to be used for replication
The IP address, Subnet Mask, Default Gateway (When using static and not DHCP)
The network settings to access the peer site

Whilst installing VRA, SSH will be enable on the host and will be disabled automatically after installation. Once you connect to the ZVM main dashboard with the IP address on port 443, click on set up to begin the installation process.

Each time a VRA is installed, you need to click on New VRA. The idea is to choose the host and installed the VRA on each HOST. It is also recommended to use static IP for the VRA Network details. Details such as the datastore, network, V Ram (The amount of RAM determines the maximum buffer size for the VRA when buffering IOs ) should be entered before the installation as follows:

Once installation process started, the VRA Status will be on mode “installing”. The alert notification on bottom left can be clicked. An alert that could be prompted could be “Host xxx.xxx.xx.xx has no VRA installed”. Zerto API can also be used to automate the process for installation of VRAs. Once the VRA’s installed on each host, its important to paired with the Recovery site by repeating the same process on the production site.

There is two way to pair with a disaster recovery site. One when login to the ZVR console, it will prompt a licence to pair with which should that of the recovery site or a manual pairing can be done when already connected by clicking on Sites -> Pair and click on the Pair button

After pairing from source to recovery site, it is to be noted that no replication is happening yet. By setting up Virtual Protection Groups (VPGs) we can start tracking block-level changes and replication across the two sites.

Virtualisation CloudComputing, vmware, Zerto