vCloud Director, Virtualization, VMware

Enabling nested 64-bit virtual hosts on vCloud Director 1.5 using MSSQL

After a crash of the database inside of my lab, I was forced to setup my vCloud Director environment once more. Before you ask, yes, I did have a backup of my database. But as Murphy would have it, it wasn’t usable for a restore.

Anyway, this allowed me to actually re-create my environment, which wasn’t a bad thing. My idea was to create an easy to use nested 64-bit vSphere environment, where I could actually quickly deploy a vSphere lab to work/test/play with.

First off, I had to enable my hosts to allow nested 64-bit vSphere guests to be installed. A way to set this up can be found here.

In summary, you can either manually add the following line:
vhv.allow = "TRUE"
to the file /etc/vmware/config on your ESXi host, or you can SSH to your ESXi host, and use the following esxcli command to set the flag (which only works if the vCloud agent has already been installed, as @lamw correctly pointed out on Twitter here):
esxcli vcloud esxvm enable64bitnested

So much for step one. 😉

But now comes the fun part, enabling this in vCloud Director. Basic instructions on how to do that can be found here, and I can only confirm the warning given there:


This is not a supported configuration by VMware and this can disappear at any time, use at your own risk!


Since the instructions found on virtuallyGhetto are a bit more targeted towards Oracle, I thought I’d might as well share the instructions for a Microsoft SQL server, since these are slightly different.

For starters, go to the SQL server that is running your vCloud Director database, open the Object Explorer, and run a query against the dbo.config table that will allow nested 64-bit systems to run inside of vCloud Director. That query should look like this:

USE ReplaceWithYourDatabaseName;
SELECT config_it, cat, name, value, sortorder
FROM dbo.config
WHERE (name = 'extension.esxvm.enabled');

From there you can simply edit the value from “false” to “true”

Next up, you need to create an additional guest operating system type. However, by default you don’t have any permissions to add values to the table, so on SQL2008, you need to first change the “IDENTITY_INSERT” setting for the table, add the new family type, and finally set the Identity Insert value back to it’s original value, which goes like this:

USE ReplaceWithYourDatabaseName;
SET IDENTITY_INSERT dbo.guest_osfamily ON;
INSERT
INTO dbo.guest_osfamily (family_id,family)
VALUES (6,'VMware ESX/ESXi');
SET IDENTITY_INSERT dbo.guest_osfamily OFF;

Next up, we need to insert the operating systems for the entry we just created. We do this once for ESXi 4.1:

USE ReplaceWithYourDatabaseName;
SET IDENTITY_INSERT dbo.guest_os_type ON;
INSERT INTO dbo.guest_os_type
(guestos_id, display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version,
supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto,
is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version)
VALUES (81, 'ESXi 4.x', 'vmkernelGuest', 6, 1, 1, 8, 3072, 7, 1, 1, 4, 8, 0, 0, 0, 0, 107, 40);
SET IDENTITY_INSERT dbo.guest_os_type OFF;

And once more for ESXi 5:

USE vmvblvcd15;
SET IDENTITY_INSERT dbo.guest_os_type ON;
INSERT INTO dbo.guest_os_type
(guestos_id,display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version)
VALUES (82, 'ESXi 5.x', 'vmkernel5Guest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 50);
SET IDENTITY_INSERT dbo.guest_os_type OFF;

Should the query analyzer give an error on the 81 or 82 values, you can increase these, because that just means that these values were already in use in the table. Just increase the numbers until the query analyzer doesn’t give you an error anymore.

And that’s it. You should now be able to see the new options when you create a new virtual machine for your vApp.

There are some additional steps to follow if you actually want to use the newly created options though. You need to restart the vCloud Director daemon on your vCloud cells, and re-prepare your hosts. Also, make sure to set promiscuous mode for the portgroups backing your vCloud network infrastructure, and you can check the post virtuallyGhetto for the details on that.

Virtualization, VMware, vSphere

Changing a forgotten ESXi 5 root password

It shouldn’t happen, but most folks I’ve spoken to have run in to this at some point in time. You are trying to log on to your ESXi host, and for some reason your root password isn’t working anymore.

The official stance that VMware has taken on this can be found in knowledge base article 1317898, and at the time of writing, it states the following:

ESXi 3.5, ESXi 4.x, and ESXi 5.0

Reinstalling the ESXi host is the only supported way to reset a password on ESXi. Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode do not apply.


So, after searching a bit and combining infos from several folks, I’ve found a way to reset the password, but you should note that this is not officially supported by VMware!


First off, I would recommend you empty your host of any running virtual machines, and put it in to maintenance mode. Next up, inside your vSphere Client, go to the “Home” screen, and select “Host Profiles”, or just press “Ctrl + Shift + P”. Once you are there, create a new profile from an existing host, and select the host that has the unknown password, and give it a name that you can remember.

Next up, edit the newly created profile and open up the “Security Configuration” section. From there, select the “Administrator Password” option, and in the right hand drop down menu, select “Configure a fixed administrator password”.

Now, you can set a new password, but please be careful about one thing. You need to set the password with a certain complexity level. For the exact details, have a look at VMware knowledge base entry 1012033, which states that the default password complexity policy that is set with PAM has the following default:

password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=8,8,8,7,6which actually means the following:

  • retry=3: A user is allowed 3 attempts to enter a sufficient password.
  • N0=12: Passwords containing characters from one character class must be at least twelve characters long.
    example: chars1234567
  • N1=10: Passwords containing characters from two character classes must be at least ten characters long.
    example: CHars12345
  • N2=8: Passphrases must contain words that are each at least eight characters long.
    example: software
  • N3=8: Passwords containing characters from all four character classes must be at least eight characters long.
    example: CHars12!
  • N4=7: Passwords containing characters from all four character classes must be at least seven characters long.
    example: CHars1!
  • Example: password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min= 12,10,8,8,7

If you don’t actually follow these rules, and try to apply the profile, you will get a pretty cryptic error message that in my case just stated the following:
Authentication token manipulation errorWhich isn’t that helpful.

Once you have set the new password, you just need to select the profile you have created, and you need to attach your host to this profile. Once that is done, go to the “Hosts and Clusters” view (again, click “Ctrl + Shift + H” to jump there immediately), and right click your host. Select “Host Profile” from the menu, and from there click “Apply profile”.

Now, if SSH was disabled by the applied profile, enable it again by going to the “Configuration” tab, selecting “Security Profile” and going to the properties of the “Services” part. You can start the SSH service from there. Now you can log on using the newly assigned password and that was that.

But…! There’s always a but, isn’t there? This change only works as long as you keep the host profile, or as long as the host stays withing your vCenter. So, what can you do to make the change permanent? Simple, you log on via SSH, change the password with the “passwd” command and then run the auto-backup.sh script from /sbin.

Also, if you would like to work around the password complexity policy, you can modify the following file:
/etc/pam.d/passwd to reflect your own policy. If you want to do this, create a backup of the file, modify it to reflect your own policy. After that, change the password and run the auto-backup.sh script.

Again, these last steps are not recommended by me or VMware, and this will impact the security of your system, so be extremely cautious of your changes! I’m just trying to document the steps so you might have it a bit easier should this situation occur.

EMC, Storage, VAAI, VMware, VNX, vSphere

“My VAAI is Better Than Yours” – The file side of things

EMC VNXI have to admit it. I stole, or rather “borrowed”, part of this title from a blog post of a colleague of mine, Erik Zandboer. He just now published a post on the mindset behind VAAI, and what the actual effect is on the array itself, and on your vSphere infrastructure.

VAAI was already available in vSphere 4.1, and with the switch to vSphere 5 some new features were introduced, which means that as of this release, we now have the following situation:

Block: File:
HW accelerated Zeroing NFS – Full Copy
HW accelerated Copy NFS – Extended Statistics
HW accelerated Locking NFS – Space Reservation

Some folks will say that I left out Thin Provision Stun, which is true. And while it does help to resolve some issues, I left it out because I don’t really view it as a hardware offload, which is what I’m trying to focus on.

I took the hardware in our lab, – a EMC VNX 5300 -, for a spin in our vSphere 5 setup to show the same thing as Erik showed in his blog, but instead showing off some of the File / NFS accelerations.

To get the VNX to actually support NAS VAAI offloading and get the result you expected, you need to meet the following prerequisites:

  • vSphere 5 – You need vSphere 5 installed with an Enterprise or Enterprise Plus license
  • VNX OE for File 7.0.35.x – You need your VNX Operating Environment for File to be at least at version VNX OE 7.0.35.x or newer
  • NFSv3 – The offloads only work on NFSv3-based datastores
  • The vSphere NFS VAAI offload plugin which is referenced here

If all those prerequisites are met, you should normally be able to go in to your vSphere Client and see Hardware Acceleration as Supported:

You could also enable SSH for your ESXi host, – do this by going to the individual host, click on the “Configuration” tab, select “Security Profile” and start the SSH service -, and check the support from the command line. For block devices you could enter the following command:

esxcli storage core device vaai status getand get back a result that shows you the NAAID, the VAAI plugin name, and the primitives with their support state. By using the following command:
esxcli storage core device list you get a similar output, but again this only works for block devices, and won’t really help you when checking the support for NFS. I haven’t found any way so far to get a reliable statement back via SSH, but I’ll try to continue looking and update this post if I find something.

In case of the VNX, we can actually check on the array itself to see if we are using the primitives, so I’m actually showing you the output from the array itself, using the following command on the VNX:

server_stats server_2 -monitor nfs.v3.vstorage -type accu -i 1
First off, I went back in to my ESXi host and went in to the NFSv3 datastore that was hosting my virtual machine. In this case, a Windows 2008 server, running an SAP Enterprise Portal, and I used the vmkfstools to create a clone:

vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdkand I set off a snap using a similar command:
vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdk. All the while, I had the VNX command that I posted before running in a different window. The output from the VNX was showing that we are actually using the VAAI NFS offloading functions:

server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Timestamp Op Max Average
uSecs uSec/Op
09:07:14
09:07:15
09:07:16
09:07:17
09:07:18 vaaiFastClone 1 0 0 0
vaaiVxAttrs 3 0 1 0
vaaiRegister 5 0 0 0
09:07:19
.......
09:08:27 vaaiOffloadStatus 1 0 0 0
vaaiVxAttrs 7 1 1 0
vaaiRegister 10 0 0 0
09:08:28
09:08:29
09:08:30
09:08:31
09:08:32 vaaiOffloadStatus 2 0 0 0
server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Summary Op Max Average
uSecs uSec/Op
Minimum vaaiFullClone 0 0 83308 -
vaaiFastClone 0 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 0 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 0 0 0 0
Average vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 3 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 5 0 0 0
Maximum vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 2 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 7 1 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 10 0 0 0
(sorry for the formatting, I couldn’t get it to show the way it should).

Once the files are created, use a:
vmkfstools --extendedstat GI-C-SAP-EPBW.vmdk on the source file, or on the snap or clone to actually display the extended statistics. The “Capacity bytes” show the allocated space for the virtual disk, the “Used bytes” displays the blocks used for the virtual disk which in case of our snapshot is the fast clone and it’s parent. The “Unshared bytes” displays the usage of the actual fast clone itself without the parent.

I should point out that the offload did speed up my full clone operation, but it was “only” in the range of 20%. That isn’t a great deal, but using both esxtop and the vSphere Client performance graphs showed that the ESXi server was busy doing what it is supposed to do: virtualizing my resources! And that’s the most important thing, isn’t it?

Virtualization, VMware, vSphere

vSphere 5 introduction – Links

Hey folks,

since there is a lot going on surrounding the VMware launch of vSphere 5 today, I thought I’d just start a little page with links to the various blog posts and press announcements around this event.

So, here goes: