Category: vSphere

Storage, Virtualization, VMware, VMworld, vSphere

VMworld 2013 – Link collection

Bas Raayman

As most of you will know, VMworld is going on right now, and they kicked off this morning with the general Keynote. There were some new announcements, like for example the introduction of NSX, the public beta of VSAN, and the vCloud Suite 5.5.

As always, you’ll be flooded with blog posts and articles, so like the last couple of years, I’ll be trying to give you an overview with links. If you feel like something is missing, please leave a note in the comments, or send me a direct message on Twitter and I’ll try to get it added pronto.

So, here goes:

Clustering, GestaltIT, Nutanix, Storage, VMware, vSphere

Nutanix – What do you mean: “You are not a storage company”…?

Bas Raayman

Image copyright of the Davis Museum
Image copyright of the Davis Museum
“You are a black guy, you must be great at dancing and basketball”. “You’re a blonde? Let me explain that joke to you once more”.

Stereotypes. We all know them, we all apply them in some form or the other. We put things in boxes after a quick look, and every drawer has a different label and content to separate the stereotypes. But what if it doesn’t work that way?

Since I joined Nutanix, I’ve been in several customer and partner meetings. Some of the people I’ve get got the idea right away. We are doing something new. Others put us in to a respective box or drawer. “You are a storage company” is one of the classic pieces of feedback. Or, “So you do virtual desktop infrastructure?”.

But there’s more to it. We offer a combination of commodity hardware, combined with a piece of software, and sell that as a solution. And while the use case of virtual desktops is a great one, we can also run things like Splunk, Hadoop and classic server virtualization workloads.

And while we combine the benefits of a shared storage approach to run workloads, we’re not a storage company. We utilize features offered by shared storage to make your life easier. Each node performs its operations on the local storage, but I can use the “Nutanix Distributed File System” or NDFS to create an abstracted layer that offers many of the shared storage benefits. An example would be a shared container for my virtual machines, that are accessible to all of the hosts, enabling features like live migration between hosts.

While that works out really well with our customers, and it gives you the idea you have a SAN or NAS underneath the hoods, Nutanix’s main point is not to replace your SAN or NAS. We want to offer you a “Virtual Computing Platform”, a way to make your life easier when installing, configuring and deploying virtualized workloads and solutions.

That works great, and we’ve received great feedback. There seems to be a slight disconnect though. That begins when people start asking questions like:

What do you mean: “You are not a storage company”…?

A fair question by all means, but the simple answer is: No, we are not.

A simple example that seems to come up as of late is the following. How do I share disk space from your file system directly in to a virtual machine? While there is a way to export the storage directly in to a VM (for example via NFS), this bypasses some of the concepts we utilize. By default, we mount a datastore using an NFS IP address of 192.168.5.1, which runs over a virtual switch that has no uplinks. Since we are talking about traffic that stays within the same vSwitch, we can work at blazing speeds that are not limited by the speed of the physical NIC.

If I were to mount the NFS share from a virtual machine (or a different host), we could use the external IP of the Controller VM. The problem here, is that since the external IPs are different between controller VMs, if you were to migrate your NFS client VM to a different host, everything would go over the regular network. Also, if the controller VM that you connect to as an NFS Server would be offline, your NFS share is not accessible.

The thing is, the Nutanix block is designed to work this way. It offers great flexibility when it comes to running virtualized workloads, but it is not a 100% distributed storage system. We didn’t intend on being a storage system.

It then boils down to design. Is there a way around this? Certainly.

If you want to create a distributed CIFS file share, take a look at solutions like DFS from Microsoft. You can run multiple VMs inside of a container/datastore, and just pass the disk space of the VM through. If you need more space, just add more VMs on a different node, and add capacity, and off you go. And if you run out of space on your cluster? Just add another Nutanix node, get a VM up and running, and follow the same procedure.

That way, you are actually utilizing the distributed nature of our virtual compute platform, and running your storage services in a distributed manner. Gluster FS could be a possible solution to achieve the same thing with NFS on Linux.

And like I said, if this sounds like we are not a storage company? You are absolutely right, we are not. So you might want to categorize us under a different label, put us in a different box, or create an entirely new stereotype. 😉

Nutanix, NX-2400, Virtualization, VMware, vSphere

Upgrading your Nutanix NX-2400 block from ESXi 5.0 to ESXi 5.1 using a USB thumb drive.

Bas Raayman

At the moment, I’m lucky enough to have a Nutanix block at home that I use for demos (it’s coming along to Switzerland with me tomorrow). It’s not the model with the highest specs, but it helps in giving customers a chance to actually see the kit, and give partners some hands-on time. In case you are wondering, I’m actually working with a NX-2400, or a 4-node NX-2000 cluster, hence 2400.

Thing is though, that it was running an older version of the Nutanix Operating System (NOS), which I upgraded to the latest version (NOS 3.1) without a hitch, and it was running ESXi 5.0. And to play with some of the latest features, I actually decided to upgrade to ESXi 5.1, and I figured I might as well share how that worked out for me.

The steps are relatively simple, but I figured I’ll document them here anyway. One word of caution though:

    This was done with the latest info from the Nutanix knowledge base. Be sure to check if there are updated instructions available prior to upgrading your own block.

So, step one is to actually get the installation media for ESXi 5.1. In case of the NX-3000, you can use the standard ESXi 5.1 image. For the NX-2000 systems, you need to use an image that is customized by Nutanix. Contact myself or your local systems engineer to get the download location.

Next, create a bootable USB stick using the image. Easiest way I found is to actually format the stick with FAT32 as the filesystem. I recommend using a Windows system, or a Windows VM, since no matter what I tried, I couldn’t get it to boot using a Mac. Once the drive is formatted, I used UNetbootin:

UNetbootin ESXi 5.1 Nutanix

Click on “Disc Image” and select the ISO file. Make sure “USB Drive” is selected, and point it to the correct drive. Then click on “OK”, and watch it go to work. If it gives you a message stating that menu.c32 is already present, click on the “Yes” button.

We’ll also need to edit the NFS heartbeat timeout settings. To do that, log on to vCenter, select the node and go to “Software” -> “Advanced settings”. There go to the NFS entries, and modify the “NFS.HeartbeatTimeout” setting to 30 seconds. Do that for each host.

Next, we need to make sure the multiextent module is loaded. Add the following lines to /etc/rc.local.d/local.sh on each host (if not already there):
#added to support multiextent
localcli system module load -m multiextent
#end of adding
Then restart the host.

Once you are done, it is time to start the upgrade. Go in to the BIOS (using the Delete key) on the node you want to upgrade, and change the boot order so that you actually start off of your USB stick. Once you save the config and restart, you will be given a menu where you select the second option:

Unetbootin - Nutanix ESXi 5.1 upgrade menu

After that you should be able to see the trusted ESXi boot sequence:
ESXi 5.1 boot screen

At the installation screen, just hit the “Enter” key to continue with the installation. Read the license agreement, and continue with F11. Next, you are asked where the installation should reside. Normally you should see the Intel SSD already have a VMFS partition, indicated by the small asterisk in front of the disk. Select that disk and press “Enter” to continue:
ESXi 5.1 upgrade VMFS

Next, a prompt should show up asking if you want to upgrade. Select that option, and press “enter” once more:
Upgrade VMFS ESXi 5.1

The final step is to confirm your upgrade by pressing the “F11” key. Once the upgrade is done, remove the USB thumb drive, and reboot the server by again hitting the “Enter” key. Let the node reboot, change the boot order to the original sequence, and, tadaaaaaa:
Nutanix - ESXi 5.1 upgrade complete

Now, obviously this would be easier using the vSphere Update Manager, but this was the solution I used, since I only installed the vCenter virtual appliance. Not pretty but it works.

One key thing left to do, is to re-register the controller VMs on your ESXi host. You can do this via the vSphere client going directly to the ESXi host. Just right-click the VM and select “remove from inventory”. Then browse the datastore, go to the folder saying “ServiceVM-1.24_Ubuntu” and add the VM to the inventory using the VMX file. You can now start your VM after you confirmed that you moved it. 🙂

The other alternative to re-register your VM using vim-cmd via an SSH session on to your ESXi host. Just check which VMs you have running:
vim-cmd vmsvc/getallvms

Vmid Name File Guest OS Version Annotation
190 NTNX-TRAIN2-S11317022510746-A-CVM--2- [NTNX-local-ds-S11317022510746-A] ServiceVM/ServiceVM.vmx ubuntu64Guest vmx-07Remember the VMID and de-register the VM:
vim-cmd vmsvc/unregister [vmid of controller VM]Now simply re-register the VM:
vim-cmd solo/register [/full/file/path/to/the/controller_vm_name.vmx]You might want to rename the controller VM once you have registered it.

Should you have any issues starting the VM, make sure that there is no line saying:
sched.mem.prealloc = "TRUE"in the .vmx file of you VM. If this line is present, remove it, and re-register your VM.

VAAI, VMware, vSphere

vBrownbag – VAAI on NFS session during VMworld

Bas Raayman

Well, after being in Barcelona for a week for VMworld Europe, and after some other things that I had going on, I wanted to take some time to throw out a quick blog post on somethign that I have been getting positive feedback on.

If you aren’t familiar already with the vBrownbag initiative, make sure to check it out at http://professionalvmware.com/brownbags/. To quote from the site:

The ProfessionalVMware #vBrownBag is a series of online webinars held using GotoMeeting and covering various Virtualization & VMware Certification topics.

While VMworld was going on, some of the vBrownbag crew were visiting, and set up short 10 minute sessions in which presenters could come by and discuss various topics. Topics ranging from VCDX certification, “unsupported” sessions which showed off some neat unsupported tricks, and other topics.

Fellow blogger Julian Wood, actually put up a great blog post over on wooditwork.com that directly shows you all of the recorded sessions, including my own which is titled: “VAAI tips, specifics, common pitfalls and caveats on NFS”.

It’s good that it adds video and audio commentary, but I thought I’d also add the slides, which you can find here:

http://app.sliderocket.com:80/app/fullplayer.aspx?id=34d7923b-017c-435b-8ea7-043ab0a895da

I hope that it’s of use to you, and look forward to your feedback.

Virtualization, VMware, VMworld, vSphere

VMware releases vSphere 5.1

Bas Raayman

Today, at VMworld in San Francisco, VMware released a new version of their virtualization platform, namely vSphere 5.1.

To anyone who has been working with vSphere for some time, the version number won’t be that big of a surprise. Also, just before the weekend, the new version number actually showed up in the VMware Compatibility Guide (and was taken offline again over the weekend). But, as little surprise as the version number was, there was one quite big surprise that went along with all of the sparkly new features: A change in the licensing model.

Rumors were already circulating a week before the convention, and this change certainly wasn’t an easy decision for VMware. I was in an early partner briefing, and while we were getting the briefing, there were still mails going around inside of VMware, and a change in the licensing policy was actually communicated via an internal mail during the briefing. Since most people didn’t really like the change in licensing that came with vSphere 5, VMware made a subsequent change in its new licensing policy just a month after releasing vSphere 5.

So, what changed in the licensing department?

It all become much easier to deal with, since VMware dropped the VRAM licensing model. Yes, you read that right, VMware is no longer charging the virtualized RAM. The short version is this:

vRAM licensing is no longer used, the licensing is now per CPU socket.

There are other changes like the VMware vCloud Suite, but I will cover this in a different post.

And what is new in the technology department?

Well, the usual upgrade in terms of bumped maximums:

  • Up to 64 virtual CPUs
  • Up to 1TB of vRAM
  • Up to 32 hosts can now access a linked clone
  • 16Gb FC HBA support

And more things that will put vSphere and Hyper-V on par from a maximums standpoint. But I don’t really think that those limits are interesting. To find the latest maximums we can have a look at the configuration maximums guide. So, what else has changed?

  • Virtual Shared Graphics Acceleration (vSGA)
    “vSGA expands upon existing non‐hardware accelerated graphics capabilities for basic 3D workloads, by supporting accelerating VDI workloads using physical GPU resources. With this new capability, it is now possible to virtualize physical GPU resources, sharing them across virtual desktops. This functionality supports an array of graphically rich and intense applications such as full motion video, rich media services, and more demanding 3D graphics”To actually use this feature, you currently need an NVIDIA GPU that is based on the GF100 architecture (Fermi) such as the Quadro 4000, Quadro 5000 or Quadro 6000  series. For people who like to dig around, this is also what the X server is for in the installation media.
  • No more reboots to upgrade the VMware tools
    You read that right. You can now perform an online upgrade of the VMware tools for any VM that is running Windows Vista, or a later Windows release.
  • Network Health Check
    “Assures proper physical and virtual operation by providing health monitoring for physical network setups including VLAN, MTU or Teaming. Today, vSphere administrators have no tools to verify whether the physical setup is capable of deploying virtual machines correctly. Even simple physical setups can be frustrating to debug end to end. Network health check eases these issues by giving you a window into the operation of your physical and virtual network”
  • Single Sign On (SSO)
    “The vSphere 5.1 SSO feature simplifies the login process for the Cloud Infrastructure Suite. You can log into the management infrastructure a single time through the vSphere Web Client or the API and be able to perform operations across all components of the Cloud Infrastructure Suite without having to log into the components separately SSO operates across all Cloud Infrastructure Suite components that support this feature.”This is a great addition. You can now use your vCenter installation as a SSO source, or you can integrate directly in to existing OpenLDAP and Active Directory sources. Scheme support is present for LDAP, LDAPS and NIS.
  • vMotion
    You can now simultaneously vMotion memory and storage. I hear you thinking that “you could do that for a while now”, and you are correct. But with vSphere 5.1, you can finally do it online. Additionally, there is no need for shared storage to perform a vMotion. This means that you can use local disks, inside of your hosts, and online migrate your virtual machines between hosts without having centralised storage, using NBD/NFC in the background. In my book, this is a great feature when working with a home lab.

Those are some pretty neat things, and there are even more out there, but there is one major change that I wanted to save. Previously, VMware announced the vSphere Web Client about a year ago (David Davis has done a nice writeup of it here), and set the tone for the future interface. Now, in vSphere 5, they made it very clear:

To use new things like the newer VM hardware version, the shared nothing vMotion, or any of the other new features, you have to use the new vSphere Web Client.

And that’s ok. The Web Client works like a charm. It did have some smaller bugs during testing, but to me proved to be quite reliable and easy to use. Plus, it allows you to search for objects from any location, adds features like custom tags that you can add to resources, and has modifications that make life easier. An example of that last point, is when you add a new datastore to a host. If you re-use the name, the Web Client will detect this, and will ask you if you would like to use the same settings, saves a bit of time.

There is one problem with this strategy though. You won’t be able to completely switch to the Web Client. For four tasks, you will still need the classic vSphere Client:

  • Import and export host profiles. You cannot import or export host profiles with the vSphere Web Client.
  • vSphere Update Manager. vSphere Update Manager isn’t available in the vSphere Web Client.
  • Datastore Browser inflate thin disk option. The Datastore Browser in the vSphere Client has an option to inflate a thin disk to a thick disk. The vSphere Web Client does not have this option. You cannot inflate a thin disk using the vSphere Web Client.
  • vSphere Authentication Proxy Server.

That might change once the final version is available for download though. Also, with the Web Client, the way that vCenter plugins work, has changed. This will mean that if you rely on any plugins for your daily operation, now is the time to contact your software/hardware vendor, and ask them if they are planning on the release of a new plugin that will work in the Web Client.

All in all?

All in all, I would say that with vSphere 5.1, the maximum configurations were aligned with what other hypervisors offer, and we again see some nice additions in functionality. A lot of folks will welcome the change in licensing policy, and all of those Mac users will welcome the fact that they can now perform their daily administration, without having to install a VM or connect to a remote desktop.

Some things aren’t entirely logical, like the fact that not all of the functionality was ported to the Web Client (yet), but I think it’s safe to say that there is more good than bad in this release. If you want to learn more about the technical side, or the rest of the VMware vCloud Suite, make sure to check every now and then, since I’ll be posting follow ups with exactly that info. In terms of the software being released, we are still waiting for an official release date, but I’ll update this posting once the date has been announced.

Update – August 31st:

The Dutch VMware Twitter account (VMware_NL) just gave me an expected release date for the vCloud Suite, and for vSphere 5.1: September 11th 2012. Keep in mind that this may change though.

Update – September 18th:
You can now actually download the release. It went live during my holiday, so I didn’t update the post. Download it from the VMware website. Also, the configuration maximums guide got released. Download it at: http://www.vmware.com/pdf/vsphere5/r51/vsphere-51-configuration-maximums.pdf.

Virtualization, VMware, VMworld, vSphere

vSphere 5.1 launch – Link Collection

Bas Raayman

There’s a lot of articles going online at the moment, due to the launch of vSphere 5.1, and due to the fact that everyone, – myself included – has their share to say.

Since it will be hard to keep track of everything that is put online, I’d thought I’d help us all out by creating a page with links to the various blog posts and articles. So here goes:

The official VMware links:

Obviously there are also a lot of other posts going online, so here are the…

other links:

If you feel like I missed a link, please let me know in the comments, or send me a tweet.

GestaltIT, vCloud Director, Virtualization, VMware, VMworld, vSphere

VMworld 2012 – Call for voting and a jiffy?!

Bas Raayman

vote! by smallcaps, on FlickrThe Twitter world has been slightly abuzz. The reason? Well, a couple of weeks ago people were allowed to submit session proposals on VMworld.com. Basically, the call for papers is a way for folks to say “Hey, this is a cool idea for a session I have. This is what I would like to talk about.”. You submitted that on the site, and a first selection was made of the submissions, before they were now put online.

What do you need to do now? Well, you need to vote! If you go to VMworld.com you can click on the “Call for Papers Public Voting” link, and then cast a vote for the sessions you would like to see at VMworld. The only thing you need is a registered account at VMworld.com, and if you don’t have an account, you can create one here.

Once your are on the site, just browse through the sessions, and click on the thumb symbol in front of the session to cast your vote. It’s as easy as that, and you can vote for all the sessions that seem interesting to you (and others).

And while you are browsing, why not also take a quick look at session number 1665? This was submitted by my colleague Jonas Rosland and myself, and is titled:

Automagically Set-up Your Private Cloud Lab Environment: From Empty Box to Infrastructure as a Service in a Jiffy!

After casting your vote, it should look like this:

In the session, we will cover setting up a fully automated vCloud Director deployment in your lab environment. Starting off with an empty server and teaching you how to automate the installation of a full Cloud Infrastructure with ESXi, vCenter, vCloud Director and vApps, combined with the power of vCenter Orchestrator. And with all of this combined, you’ll be done in a jiffy!

If you think it would be interesting, we are both thankful for your vote! 🙂

Clustering, EMC, Storage, Virtualization, VMware, VPLEX, vSphere

VMware HA demo using vMSC with EMC VPLEX Metro

Bas Raayman

That’s a mouth full of abbreviations for a title, isn’t it?

So, let me give you some background info. VMware introduced something called the vSphere Metro Storage Cluster, and Duncan Epping talks about this feature here.

What the vMSC allows us to do, is to create a regular stretched vSphere cluster, but now also stretch out the storage between the two clusters. This can be done in two ways (to quote from Duncan’s article):

I want to briefly explain the concept of a metro / stretched cluster, which can be carved up in to two different type of solutions. The first solution is where a synchronous copy of your datastore is available on the other site, this mirror copy will be read-only. In other words there is a read-write copy in Datacenter-A and a read-only copy in Datacenter-B. This means that your VMs in Datacenter-B located on this datastore will do I/O on Datacenter-A since the read-write copy of the datastore is in Datacenter-A. The second solution is which EMC calls “write anywhere”. In this case VMs always write locally. The key point here is that each of the LUNs / datastores has a “preferred site” defined, this is also sometimes referred to as “site bias”. In other words, if anything happens to the link in between then the storage system on the preferred site for a given datastore will be the only one left who can read-write access it.

The last scenario described here is something that obviously can cause some issues. EMC tried to address this by introducing the “independent 3rd party”, in form of the VPLEX Witness. Some documentation states that this witness should run in a 3rd site, but I would recommend to run this in a separate failure domain.

In essence, we have created the following setup:

© VMware

Awesome stuff, because we can do new things that weren’t quite possible before. Since VPLEX is one of the key storage virtualization solutions from EMC that allow us to perform an active/active disk access, we can perform a vMotion between the two sites, and due to the nature of VPLEX, we also perform a sort of storage vMotion on the underlying disks. That, without you having to shut down the VM to do both things at the same time. Pretty neat!

Now, as Chad describes here, a new disk connectivity state was introduced with vSphere 5, called “Permanent Device Loss” or PDL. This was a great feature to communicate to your infrastructure that a target was intentionally removed. You could unmount the disk, and remove the paths to your target in a proper way.

It was also useful to indicate an unexpected loss of your target, indicating that your cluster is in a partitioned state. The problem here was that a PDL state and VMware HA didn’t work so well together. When you had an APD notification, HA didn’t “kill” your VM, and your virtual machine would usually continue to respond to pings, but that was about it.

Then along came vSphere 5 Update 1, which allows us to set a flag on each of the hosts inside our cluster, and set a different flag for our HA cluster. Now, we can actually use HA and see terminate the VMs and have it restart the virtual machines on the hosts in our cluster that still have access to their datastores in their respective preferred sites.

I’ve created a short (ok, 8 minutes) video that show exactly this scenario. You’ll get a quick view of the VPLEX setup. You’ll see the Brocade switches that will change from a config with the normal full zoneset, being switched to a zoneset that will disable the inter-switch links between both VPLEX clusters. And you’ll see the settings inside of my vSphere lab setup, with the behavior of the hosts and virtual machines.

Since I’m quite new to creating videos like this, I hope the output is acceptable, and the video is clear enough. If you have any questions, feedback or would like to see more, please leave me a comment and I’ll see what I can do. 🙂

Just a quick modification to my post, since it wasn’t actually VM-HA (or VM monitoring) responding to the PDL event, but HA terminating the VM when running in to the PDL state, as Duncan pointed out to me on Twitter. Sorry for any confusion I may have caused!

Virtualization, VMware, vSphere

Changing a forgotten ESXi 5 root password

Bas Raayman

It shouldn’t happen, but most folks I’ve spoken to have run in to this at some point in time. You are trying to log on to your ESXi host, and for some reason your root password isn’t working anymore.

The official stance that VMware has taken on this can be found in knowledge base article 1317898, and at the time of writing, it states the following:

ESXi 3.5, ESXi 4.x, and ESXi 5.0

Reinstalling the ESXi host is the only supported way to reset a password on ESXi. Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode do not apply.

So, after searching a bit and combining infos from several folks, I’ve found a way to reset the password, but you should note that this is not officially supported by VMware!

First off, I would recommend you empty your host of any running virtual machines, and put it in to maintenance mode. Next up, inside your vSphere Client, go to the “Home” screen, and select “Host Profiles”, or just press “Ctrl + Shift + P”. Once you are there, create a new profile from an existing host, and select the host that has the unknown password, and give it a name that you can remember.

Next up, edit the newly created profile and open up the “Security Configuration” section. From there, select the “Administrator Password” option, and in the right hand drop down menu, select “Configure a fixed administrator password”.

Now, you can set a new password, but please be careful about one thing. You need to set the password with a certain complexity level. For the exact details, have a look at VMware knowledge base entry 1012033, which states that the default password complexity policy that is set with PAM has the following default:

password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=8,8,8,7,6which actually means the following:

  • retry=3: A user is allowed 3 attempts to enter a sufficient password.
  • N0=12: Passwords containing characters from one character class must be at least twelve characters long.
    example: chars1234567
  • N1=10: Passwords containing characters from two character classes must be at least ten characters long.
    example: CHars12345
  • N2=8: Passphrases must contain words that are each at least eight characters long.
    example: software
  • N3=8: Passwords containing characters from all four character classes must be at least eight characters long.
    example: CHars12!
  • N4=7: Passwords containing characters from all four character classes must be at least seven characters long.
    example: CHars1!
  • Example: password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min= 12,10,8,8,7

If you don’t actually follow these rules, and try to apply the profile, you will get a pretty cryptic error message that in my case just stated the following:
Authentication token manipulation errorWhich isn’t that helpful.

Once you have set the new password, you just need to select the profile you have created, and you need to attach your host to this profile. Once that is done, go to the “Hosts and Clusters” view (again, click “Ctrl + Shift + H” to jump there immediately), and right click your host. Select “Host Profile” from the menu, and from there click “Apply profile”.

Now, if SSH was disabled by the applied profile, enable it again by going to the “Configuration” tab, selecting “Security Profile” and going to the properties of the “Services” part. You can start the SSH service from there. Now you can log on using the newly assigned password and that was that.

But…! There’s always a but, isn’t there? This change only works as long as you keep the host profile, or as long as the host stays withing your vCenter. So, what can you do to make the change permanent? Simple, you log on via SSH, change the password with the “passwd” command and then run the auto-backup.sh script from /sbin.

Also, if you would like to work around the password complexity policy, you can modify the following file:
/etc/pam.d/passwd to reflect your own policy. If you want to do this, create a backup of the file, modify it to reflect your own policy. After that, change the password and run the auto-backup.sh script.

Again, these last steps are not recommended by me or VMware, and this will impact the security of your system, so be extremely cautious of your changes! I’m just trying to document the steps so you might have it a bit easier should this situation occur.

EMC, Storage, VAAI, VMware, VNX, vSphere

“My VAAI is Better Than Yours” – The file side of things

Bas Raayman

EMC VNXI have to admit it. I stole, or rather “borrowed”, part of this title from a blog post of a colleague of mine, Erik Zandboer. He just now published a post on the mindset behind VAAI, and what the actual effect is on the array itself, and on your vSphere infrastructure.

VAAI was already available in vSphere 4.1, and with the switch to vSphere 5 some new features were introduced, which means that as of this release, we now have the following situation:

Block: File:
HW accelerated Zeroing NFS – Full Copy
HW accelerated Copy NFS – Extended Statistics
HW accelerated Locking NFS – Space Reservation

Some folks will say that I left out Thin Provision Stun, which is true. And while it does help to resolve some issues, I left it out because I don’t really view it as a hardware offload, which is what I’m trying to focus on.

I took the hardware in our lab, – a EMC VNX 5300 -, for a spin in our vSphere 5 setup to show the same thing as Erik showed in his blog, but instead showing off some of the File / NFS accelerations.

To get the VNX to actually support NAS VAAI offloading and get the result you expected, you need to meet the following prerequisites:

  • vSphere 5 – You need vSphere 5 installed with an Enterprise or Enterprise Plus license
  • VNX OE for File 7.0.35.x – You need your VNX Operating Environment for File to be at least at version VNX OE 7.0.35.x or newer
  • NFSv3 – The offloads only work on NFSv3-based datastores
  • The vSphere NFS VAAI offload plugin which is referenced here

If all those prerequisites are met, you should normally be able to go in to your vSphere Client and see Hardware Acceleration as Supported:

You could also enable SSH for your ESXi host, – do this by going to the individual host, click on the “Configuration” tab, select “Security Profile” and start the SSH service -, and check the support from the command line. For block devices you could enter the following command:

esxcli storage core device vaai status getand get back a result that shows you the NAAID, the VAAI plugin name, and the primitives with their support state. By using the following command:
esxcli storage core device list you get a similar output, but again this only works for block devices, and won’t really help you when checking the support for NFS. I haven’t found any way so far to get a reliable statement back via SSH, but I’ll try to continue looking and update this post if I find something.

In case of the VNX, we can actually check on the array itself to see if we are using the primitives, so I’m actually showing you the output from the array itself, using the following command on the VNX:

server_stats server_2 -monitor nfs.v3.vstorage -type accu -i 1
First off, I went back in to my ESXi host and went in to the NFSv3 datastore that was hosting my virtual machine. In this case, a Windows 2008 server, running an SAP Enterprise Portal, and I used the vmkfstools to create a clone:

vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdkand I set off a snap using a similar command:
vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdk. All the while, I had the VNX command that I posted before running in a different window. The output from the VNX was showing that we are actually using the VAAI NFS offloading functions:

server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Timestamp Op Max Average
uSecs uSec/Op
09:07:14
09:07:15
09:07:16
09:07:17
09:07:18 vaaiFastClone 1 0 0 0
vaaiVxAttrs 3 0 1 0
vaaiRegister 5 0 0 0
09:07:19
.......
09:08:27 vaaiOffloadStatus 1 0 0 0
vaaiVxAttrs 7 1 1 0
vaaiRegister 10 0 0 0
09:08:28
09:08:29
09:08:30
09:08:31
09:08:32 vaaiOffloadStatus 2 0 0 0
server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Summary Op Max Average
uSecs uSec/Op
Minimum vaaiFullClone 0 0 83308 -
vaaiFastClone 0 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 0 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 0 0 0 0
Average vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 3 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 5 0 0 0
Maximum vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 2 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 7 1 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 10 0 0 0
(sorry for the formatting, I couldn’t get it to show the way it should).

Once the files are created, use a:
vmkfstools --extendedstat GI-C-SAP-EPBW.vmdk on the source file, or on the snap or clone to actually display the extended statistics. The “Capacity bytes” show the allocated space for the virtual disk, the “Used bytes” displays the blocks used for the virtual disk which in case of our snapshot is the fast clone and it’s parent. The “Unshared bytes” displays the usage of the actual fast clone itself without the parent.

I should point out that the offload did speed up my full clone operation, but it was “only” in the range of 20%. That isn’t a great deal, but using both esxtop and the vSphere Client performance graphs showed that the ESXi server was busy doing what it is supposed to do: virtualizing my resources! And that’s the most important thing, isn’t it?