NFS – BasRaayman's Technical Take

Clustering, GestaltIT, Nutanix, Storage, VMware, vSphere

Nutanix – What do you mean: “You are not a storage company”…?

August 9, 2013 Bas Raayman

“You are a black guy, you must be great at dancing and basketball”. “You’re a blonde? Let me explain that joke to you once more”.

Stereotypes. We all know them, we all apply them in some form or the other. We put things in boxes after a quick look, and every drawer has a different label and content to separate the stereotypes. But what if it doesn’t work that way?

Since I joined Nutanix, I’ve been in several customer and partner meetings. Some of the people I’ve get got the idea right away. We are doing something new. Others put us in to a respective box or drawer. “You are a storage company” is one of the classic pieces of feedback. Or, “So you do virtual desktop infrastructure?”.

But there’s more to it. We offer a combination of commodity hardware, combined with a piece of software, and sell that as a solution. And while the use case of virtual desktops is a great one, we can also run things like Splunk, Hadoop and classic server virtualization workloads.

And while we combine the benefits of a shared storage approach to run workloads, we’re not a storage company. We utilize features offered by shared storage to make your life easier. Each node performs its operations on the local storage, but I can use the “Nutanix Distributed File System” or NDFS to create an abstracted layer that offers many of the shared storage benefits. An example would be a shared container for my virtual machines, that are accessible to all of the hosts, enabling features like live migration between hosts.

While that works out really well with our customers, and it gives you the idea you have a SAN or NAS underneath the hoods, Nutanix’s main point is not to replace your SAN or NAS. We want to offer you a “Virtual Computing Platform”, a way to make your life easier when installing, configuring and deploying virtualized workloads and solutions.

That works great, and we’ve received great feedback. There seems to be a slight disconnect though. That begins when people start asking questions like:

What do you mean: “You are not a storage company”…?

A fair question by all means, but the simple answer is: No, we are not.

A simple example that seems to come up as of late is the following. How do I share disk space from your file system directly in to a virtual machine? While there is a way to export the storage directly in to a VM (for example via NFS), this bypasses some of the concepts we utilize. By default, we mount a datastore using an NFS IP address of 192.168.5.1, which runs over a virtual switch that has no uplinks. Since we are talking about traffic that stays within the same vSwitch, we can work at blazing speeds that are not limited by the speed of the physical NIC.

If I were to mount the NFS share from a virtual machine (or a different host), we could use the external IP of the Controller VM. The problem here, is that since the external IPs are different between controller VMs, if you were to migrate your NFS client VM to a different host, everything would go over the regular network. Also, if the controller VM that you connect to as an NFS Server would be offline, your NFS share is not accessible.

The thing is, the Nutanix block is designed to work this way. It offers great flexibility when it comes to running virtualized workloads, but it is not a 100% distributed storage system. We didn’t intend on being a storage system.

It then boils down to design. Is there a way around this? Certainly.

If you want to create a distributed CIFS file share, take a look at solutions like DFS from Microsoft. You can run multiple VMs inside of a container/datastore, and just pass the disk space of the VM through. If you need more space, just add more VMs on a different node, and add capacity, and off you go. And if you run out of space on your cluster? Just add another Nutanix node, get a VM up and running, and follow the same procedure.

That way, you are actually utilizing the distributed nature of our virtual compute platform, and running your storage services in a distributed manner. Gluster FS could be a possible solution to achieve the same thing with NFS on Linux.

And like I said, if this sounds like we are not a storage company? You are absolutely right, we are not. So you might want to categorize us under a different label, put us in a different box, or create an entirely new stereotype. 😉

VAAI, VMware, vSphere

vBrownbag – VAAI on NFS session during VMworld

October 22, 2012 Bas Raayman

Well, after being in Barcelona for a week for VMworld Europe, and after some other things that I had going on, I wanted to take some time to throw out a quick blog post on somethign that I have been getting positive feedback on.

If you aren’t familiar already with the vBrownbag initiative, make sure to check it out at http://professionalvmware.com/brownbags/. To quote from the site:

The ProfessionalVMware #vBrownBag is a series of online webinars held using GotoMeeting and covering various Virtualization & VMware Certification topics.

While VMworld was going on, some of the vBrownbag crew were visiting, and set up short 10 minute sessions in which presenters could come by and discuss various topics. Topics ranging from VCDX certification, “unsupported” sessions which showed off some neat unsupported tricks, and other topics.

Fellow blogger Julian Wood, actually put up a great blog post over on wooditwork.com that directly shows you all of the recorded sessions, including my own which is titled: “VAAI tips, specifics, common pitfalls and caveats on NFS”.

It’s good that it adds video and audio commentary, but I thought I’d also add the slides, which you can find here:

http://app.sliderocket.com:80/app/fullplayer.aspx?id=34d7923b-017c-435b-8ea7-043ab0a895da

I hope that it’s of use to you, and look forward to your feedback.

EMC, Storage, VAAI, VMware, VNX, vSphere

“My VAAI is Better Than Yours” – The file side of things

February 7, 2012May 10, 2012 Bas Raayman

I have to admit it. I stole, or rather “borrowed”, part of this title from a blog post of a colleague of mine, Erik Zandboer. He just now published a post on the mindset behind VAAI, and what the actual effect is on the array itself, and on your vSphere infrastructure.

VAAI was already available in vSphere 4.1, and with the switch to vSphere 5 some new features were introduced, which means that as of this release, we now have the following situation:

Block:	File:
HW accelerated Zeroing	NFS – Full Copy
HW accelerated Copy	NFS – Extended Statistics
HW accelerated Locking	NFS – Space Reservation

Some folks will say that I left out Thin Provision Stun, which is true. And while it does help to resolve some issues, I left it out because I don’t really view it as a hardware offload, which is what I’m trying to focus on.

I took the hardware in our lab, – a EMC VNX 5300 -, for a spin in our vSphere 5 setup to show the same thing as Erik showed in his blog, but instead showing off some of the File / NFS accelerations.

To get the VNX to actually support NAS VAAI offloading and get the result you expected, you need to meet the following prerequisites:

vSphere 5 – You need vSphere 5 installed with an Enterprise or Enterprise Plus license
VNX OE for File 7.0.35.x – You need your VNX Operating Environment for File to be at least at version VNX OE 7.0.35.x or newer
NFSv3 – The offloads only work on NFSv3-based datastores
The vSphere NFS VAAI offload plugin which is referenced here

If all those prerequisites are met, you should normally be able to go in to your vSphere Client and see Hardware Acceleration as Supported:

You could also enable SSH for your ESXi host, – do this by going to the individual host, click on the “Configuration” tab, select “Security Profile” and start the SSH service -, and check the support from the command line. For block devices you could enter the following command:

esxcli storage core device vaai status getand get back a result that shows you the NAAID, the VAAI plugin name, and the primitives with their support state. By using the following command:
esxcli storage core device list you get a similar output, but again this only works for block devices, and won’t really help you when checking the support for NFS. I haven’t found any way so far to get a reliable statement back via SSH, but I’ll try to continue looking and update this post if I find something.

In case of the VNX, we can actually check on the array itself to see if we are using the primitives, so I’m actually showing you the output from the array itself, using the following command on the VNX:

server_stats server_2 -monitor nfs.v3.vstorage -type accu -i 1
First off, I went back in to my ESXi host and went in to the NFSv3 datastore that was hosting my virtual machine. In this case, a Windows 2008 server, running an SAP Enterprise Portal, and I used the vmkfstools to create a clone:

vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdkand I set off a snap using a similar command:
vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdk. All the while, I had the VNX command that I posted before running in a different window. The output from the VNX was showing that we are actually using the VAAI NFS offloading functions:

server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op Timestamp Op Max Average uSecs uSec/Op 09:07:14 09:07:15 09:07:16 09:07:17 09:07:18 vaaiFastClone 1 0 0 0 vaaiVxAttrs 3 0 1 0 vaaiRegister 5 0 0 0 09:07:19 ....... 09:08:27 vaaiOffloadStatus 1 0 0 0 vaaiVxAttrs 7 1 1 0 vaaiRegister 10 0 0 0 09:08:28 09:08:29 09:08:30 09:08:31 09:08:32 vaaiOffloadStatus 2 0 0 0 server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op Summary Op Max Average uSecs uSec/Op Minimum vaaiFullClone 0 0 83308 - vaaiFastClone 0 0 0 0 vaaiOffloadStatus 0 0 0 0 vaaiOffloadAbort 0 0 0 - vaaiVxAttrs 0 0 1 0 vaaiReserveSpace 0 0 0 - vaaiRegister 0 0 0 0 Average vaaiFullClone 0 0 83308 - vaaiFastClone 1 0 0 0 vaaiOffloadStatus 0 0 0 0 vaaiOffloadAbort 0 0 0 - vaaiVxAttrs 3 0 1 0 vaaiReserveSpace 0 0 0 - vaaiRegister 5 0 0 0 Maximum vaaiFullClone 0 0 83308 - vaaiFastClone 1 0 0 0 vaaiOffloadStatus 2 0 0 0 vaaiOffloadAbort 0 0 0 - vaaiVxAttrs 7 1 1 0 vaaiReserveSpace 0 0 0 - vaaiRegister 10 0 0 0(sorry for the formatting, I couldn’t get it to show the way it should).

Once the files are created, use a:
vmkfstools --extendedstat GI-C-SAP-EPBW.vmdk on the source file, or on the snap or clone to actually display the extended statistics. The “Capacity bytes” show the allocated space for the virtual disk, the “Used bytes” displays the blocks used for the virtual disk which in case of our snapshot is the fast clone and it’s parent. The “Unshared bytes” displays the usage of the actual fast clone itself without the parent.

I should point out that the offload did speed up my full clone operation, but it was “only” in the range of 20%. That isn’t a great deal, but using both esxtop and the vSphere Client performance graphs showed that the ESXi server was busy doing what it is supposed to do: virtualizing my resources! And that’s the most important thing, isn’t it?