EMC, Symmetrix, V-MAX, vCenter Operations, Virtualization, VMware

Setting up the EMC Symmetrix adapter on the vCenter Operations Manager 5 vApp

I present a lot on vCenter Operations Manager, a pretty neat monitoring tool from VMware. I like this tool a lot, because getting started with it is easy enough, and you have a plethora of features once you dive deeper in to it, and the best part about it? If you use the “big” version, -Enterprise or Enterprise Plus that is-, you can even monitor your applications and non-virtualized infrastructure. To monitor your things that go beyond your virtual machines, you can install so called “adapters”. In a nutshell, such an adapter is nothing more than a piece of software that tells vCenter Operations how to connect to things, and how to interpret the results it gets back. Now, EMC has created such an adapter for their VMAX and Symmetrix storage arrays, and has created a document that tells you how to set up and configure the adapter. That way, you can get loads of information from your storage system inside of vCenter Operations. Great stuff, right? Original image from: http://my.opera.com/supergreatChandu8/albums/showpic.dml?album=5466862&picture=82475462Yeah ok, maybe not so great. The biggest problem, is that the documentation seems to have been created for the normal installable version of vCenter Operations. However, VMware has also created a version in the form of an appliance, a so called vApp. You download the files, deploy the vApp, enter the IP-addresses of both virtual machines that are contained in the vApp, and away you go. Wonderfully easy to install, and besides certain limits in scalability, it offers pretty much the same functionality as the normal installer. This is where the problem starts if you want to use the EMC Symmetrix adapter. You can find almost all adapters on the Integrien FTP site, and there’s a folder containing all the files you need to get started with the Symmetrix adapter right here. My teammate Matt Cowger actually wrote a nice blog post on how to configure and set up the Symmetrix adapter. This works like a charm, except for one tiny thing that you will run in to when using the vCenter Operations vApp. When you go to create an adapter instance, you need to give it a name, indicate if you want to auto discover everything, and you need to input a path to the “EMC Symmetrix Main Input Folder”. This is the folder where you actually archive all of the performance and configuration data from your storage system. The documentation tells you that this should be:

* If the main input folder is on a remote Windows machine, you must share the folder before you add the adapter instance. Do not map the main input folder. Windows services do not work with mapped drives. * If the main input folder is on a remote Linux machine, you must mount the folder to the Collector server before you add the adapter instance.

Problem being, that if you actually have your Solutions Enabler host running on Windows, you need to input a UNC path in the format of \\servername\sharename. But the problem here is that the virtual machines inside the vApp do not come with any access methods for Windows shares. You won’t find any tools like mount.cifs, smbclient or even have the option to specify smbfs as the type of file system to mount. And that means what? Well, you will have two options to overcome this situation. You can either install the Services for Unix/Services for NFS on your Windows host and set up an NFS share on your Windows machine. Or, you can migrate your Solutions Enabler host to a Linux machine and set up everything there. OK, so how do I configure this stuff under Linux? Glad you asked. You can follow some of the steps from the post that Matt created, but I’m going to write them down here anyway so you will have one page with all the steps you need. I’m going to assume that you have already set up your Linux machine, and that you have installed the Solutions Enabler package. Go in to the following file: /usr/emc/API/symapi/config and add these following lines at the end of the file, then make sure you save your changes (create a backup of the original, this is always a good idea):

storstpd:dmn_run_spa = disable
storstpd:dmn_run_smc = disable
storstpd:dmn_run_ttp = enable
storstpd:dmn_run_ttp_on_sp = disable
storstpd:dmn_run_rtc = disable
storstpd:ttp_collection_interval = 5
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:ttp_dev_metrics = disable
storstpd:ttp_disk_metrics = disable
storstpd:ttp_dgdev_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:ttp_se_nwi_metrics = enable
storstpd:ttp_re_sg_metrics = enable
storstpd:ttp_re_nwc_metrics = enable
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:use_compression = enable

Next, restart the storstpd daemon:

/opt/emc/SYMCLI/bin/stordaemon shutdown storstpd /opt/emc/SYMCLI/bin/stordaemon start storstpd

Check if the daemon is up and running again by issuing the following command. The first line should show the Daemon State as “Running”:

/opt/emc/SYMCLI/bin/stordaemon show storstpd

Now, since the Analytics VM will be actually collecting the information from the adapter, it needs to be able to access the files from your Solutions Enabler host. Since the Analytics VM will be running the collection process as a user called “Admin”, we need to consider something. The admin user on the vCenter Operations appliance will be running with a user ID (UID) of 1000, and a group ID (GID) of 1003. That means that we should either install our Solutions Enabler using a user with the same user ID and group ID, or we need to map some things so that the admin user can actually access the files later on. In order to export the directory with the required files for the Symmetrix adapter, we will add the following line to /etc/exports: /usr/emc/API/symapi/stp *(rw,insecure,all_squash,anonuid=0,anongid=0) Obviously, this isn’t the best you can do from a security perspective, so feel free to change these options as needed for your environment, but basically what we are doing here is this:

  • The * just means that all IP-addresses have access. You can change this to for example the IP of the analytics VM.
  • RW means that the export is created with read and write access.
  • Insecure means that clients can use non-reserved ports.
  • All_squash means that all users get mapped to the anonymous user account
  • anonuid=0 means that the anonymous user ID will get mapped to the user ID 0. Be careful since this is the root account!
  • anonguid=0 means that the anonymous group ID will get mapped to the group ID 0. Again, this is the root group!

If you did install your Solutions Enabler as a different user, make sure that you map the anonuid and anonguid to the respective numerical IDs, to allow access to the files we are going to export. Now, we simply restart the NFS server, or have it re-read its config should it already be online, using:

/etc/init.d/nfsserver restart

or

exportfs –ra

We can check if the export is working, using the following command:

showmount –e localhost

Now, we create a scheduled job to archive the Solutions Enabler file. To do that, add the following line to your crontab: 2-57/5 * * * * /opt/emc/SYMCLI/bin/stordaemon action storstpd -cmd archive This will cause the job start at 2 minutes past the hour, and run in 5 minute intervals. Check under /usr/emc/API/symapi/stp/ttp, to see if you have a new directory. Normally the directory should be the serial number of your storage array, and contain compressed files inside of that directory that contain the information the Symmetrix adapter will need. Final thing to do right now, is log on to the analytics VM, and create a folder where we will mount the required files. For example create a directory called /media/VMAX. Once you have created the directory, edit /etc/fstab to contain the following line: 10.10.10.10: /usr/emc/API/symapi/stp /media/VMAX nfs rw,lock 0 0 Make sure you change the IP address to match that of your Solutions Enabler host, and then mount the directory using the following command:

mount /media/VMAX

If you don’t have a firewall blocking communication, you should now be able to traverse the subdirectories and access the files. Finally, you can now configure the adapter, and input the directory you just mounted as the “EMC Symmetrix Main Input Folder”. So, in the text field, simply enter the following as the path:

/media/VMAX/ttp

If you test the adapter now, you should see it come back successfully, and after giving it a bit of time, start working with the data you are now importing from your VMAX/Symmetrix system. 🙂

GestaltIT, vCloud Director, Virtualization, VMware, VMworld, vSphere

VMworld 2012 – Call for voting and a jiffy?!

vote! by smallcaps, on FlickrThe Twitter world has been slightly abuzz. The reason? Well, a couple of weeks ago people were allowed to submit session proposals on VMworld.com. Basically, the call for papers is a way for folks to say “Hey, this is a cool idea for a session I have. This is what I would like to talk about.”. You submitted that on the site, and a first selection was made of the submissions, before they were now put online.

What do you need to do now? Well, you need to vote! If you go to VMworld.com you can click on the “Call for Papers Public Voting” link, and then cast a vote for the sessions you would like to see at VMworld. The only thing you need is a registered account at VMworld.com, and if you don’t have an account, you can create one here.

Once your are on the site, just browse through the sessions, and click on the thumb symbol in front of the session to cast your vote. It’s as easy as that, and you can vote for all the sessions that seem interesting to you (and others).

And while you are browsing, why not also take a quick look at session number 1665? This was submitted by my colleague Jonas Rosland and myself, and is titled:

Automagically Set-up Your Private Cloud Lab Environment: From Empty Box to Infrastructure as a Service in a Jiffy!

After casting your vote, it should look like this:

In the session, we will cover setting up a fully automated vCloud Director deployment in your lab environment. Starting off with an empty server and teaching you how to automate the installation of a full Cloud Infrastructure with ESXi, vCenter, vCloud Director and vApps, combined with the power of vCenter Orchestrator. And with all of this combined, you’ll be done in a jiffy!

If you think it would be interesting, we are both thankful for your vote! 🙂

General, Virtualization, VMware

vExpert 2012 – And the award goes to…..

A while ago, I created a post about the vExpert nominations for 2012.

Well, in the meantime people were nominated, or nominated themselves, and the resulting list of folks was posted here. And the number of vExperts has increased again, from 326 in 2011, to 390 at the time of writing, and the announcement states:

Due to the large number of applications, the list is still not complete, and we will be adding a couple dozen names to it over the next week or so. If you have not yet received an email from us, please have a little patience. We will make sure to let you know the results as soon as possible.

Which means that we will likely have over 400 vExperts this year. Pretty cool! And I’m happy and honored that I’m part of this group again. So, here’s a special thank you to John Troyer, Alex Meier and all of the folks part of the vExpert program. And also my congratulations to the folks who were awarded with the title, especially the new vExperts! 🙂

Clustering, EMC, Storage, Virtualization, VMware, VPLEX, vSphere

VMware HA demo using vMSC with EMC VPLEX Metro

That’s a mouth full of abbreviations for a title, isn’t it?

So, let me give you some background info. VMware introduced something called the vSphere Metro Storage Cluster, and Duncan Epping talks about this feature here.

What the vMSC allows us to do, is to create a regular stretched vSphere cluster, but now also stretch out the storage between the two clusters. This can be done in two ways (to quote from Duncan’s article):

I want to briefly explain the concept of a metro / stretched cluster, which can be carved up in to two different type of solutions. The first solution is where a synchronous copy of your datastore is available on the other site, this mirror copy will be read-only. In other words there is a read-write copy in Datacenter-A and a read-only copy in Datacenter-B. This means that your VMs in Datacenter-B located on this datastore will do I/O on Datacenter-A since the read-write copy of the datastore is in Datacenter-A. The second solution is which EMC calls “write anywhere”. In this case VMs always write locally. The key point here is that each of the LUNs / datastores has a “preferred site” defined, this is also sometimes referred to as “site bias”. In other words, if anything happens to the link in between then the storage system on the preferred site for a given datastore will be the only one left who can read-write access it.

The last scenario described here is something that obviously can cause some issues. EMC tried to address this by introducing the “independent 3rd party”, in form of the VPLEX Witness. Some documentation states that this witness should run in a 3rd site, but I would recommend to run this in a separate failure domain.

In essence, we have created the following setup:

© VMware

Awesome stuff, because we can do new things that weren’t quite possible before. Since VPLEX is one of the key storage virtualization solutions from EMC that allow us to perform an active/active disk access, we can perform a vMotion between the two sites, and due to the nature of VPLEX, we also perform a sort of storage vMotion on the underlying disks. That, without you having to shut down the VM to do both things at the same time. Pretty neat!

Now, as Chad describes here, a new disk connectivity state was introduced with vSphere 5, called “Permanent Device Loss” or PDL. This was a great feature to communicate to your infrastructure that a target was intentionally removed. You could unmount the disk, and remove the paths to your target in a proper way.

It was also useful to indicate an unexpected loss of your target, indicating that your cluster is in a partitioned state. The problem here was that a PDL state and VMware HA didn’t work so well together. When you had an APD notification, HA didn’t “kill” your VM, and your virtual machine would usually continue to respond to pings, but that was about it.

Then along came vSphere 5 Update 1, which allows us to set a flag on each of the hosts inside our cluster, and set a different flag for our HA cluster. Now, we can actually use HA and see terminate the VMs and have it restart the virtual machines on the hosts in our cluster that still have access to their datastores in their respective preferred sites.

I’ve created a short (ok, 8 minutes) video that show exactly this scenario. You’ll get a quick view of the VPLEX setup. You’ll see the Brocade switches that will change from a config with the normal full zoneset, being switched to a zoneset that will disable the inter-switch links between both VPLEX clusters. And you’ll see the settings inside of my vSphere lab setup, with the behavior of the hosts and virtual machines.

Since I’m quite new to creating videos like this, I hope the output is acceptable, and the video is clear enough. If you have any questions, feedback or would like to see more, please leave me a comment and I’ll see what I can do. 🙂


Just a quick modification to my post, since it wasn’t actually VM-HA (or VM monitoring) responding to the PDL event, but HA terminating the VM when running in to the PDL state, as Duncan pointed out to me on Twitter. Sorry for any confusion I may have caused!

vCloud Director, Virtualization, VMware

Enabling nested 64-bit virtual hosts on vCloud Director 1.5 using MSSQL

After a crash of the database inside of my lab, I was forced to setup my vCloud Director environment once more. Before you ask, yes, I did have a backup of my database. But as Murphy would have it, it wasn’t usable for a restore.

Anyway, this allowed me to actually re-create my environment, which wasn’t a bad thing. My idea was to create an easy to use nested 64-bit vSphere environment, where I could actually quickly deploy a vSphere lab to work/test/play with.

First off, I had to enable my hosts to allow nested 64-bit vSphere guests to be installed. A way to set this up can be found here.

In summary, you can either manually add the following line:
vhv.allow = "TRUE"
to the file /etc/vmware/config on your ESXi host, or you can SSH to your ESXi host, and use the following esxcli command to set the flag (which only works if the vCloud agent has already been installed, as @lamw correctly pointed out on Twitter here):
esxcli vcloud esxvm enable64bitnested

So much for step one. 😉

But now comes the fun part, enabling this in vCloud Director. Basic instructions on how to do that can be found here, and I can only confirm the warning given there:


This is not a supported configuration by VMware and this can disappear at any time, use at your own risk!


Since the instructions found on virtuallyGhetto are a bit more targeted towards Oracle, I thought I’d might as well share the instructions for a Microsoft SQL server, since these are slightly different.

For starters, go to the SQL server that is running your vCloud Director database, open the Object Explorer, and run a query against the dbo.config table that will allow nested 64-bit systems to run inside of vCloud Director. That query should look like this:

USE ReplaceWithYourDatabaseName;
SELECT config_it, cat, name, value, sortorder
FROM dbo.config
WHERE (name = 'extension.esxvm.enabled');

From there you can simply edit the value from “false” to “true”

Next up, you need to create an additional guest operating system type. However, by default you don’t have any permissions to add values to the table, so on SQL2008, you need to first change the “IDENTITY_INSERT” setting for the table, add the new family type, and finally set the Identity Insert value back to it’s original value, which goes like this:

USE ReplaceWithYourDatabaseName;
SET IDENTITY_INSERT dbo.guest_osfamily ON;
INSERT
INTO dbo.guest_osfamily (family_id,family)
VALUES (6,'VMware ESX/ESXi');
SET IDENTITY_INSERT dbo.guest_osfamily OFF;

Next up, we need to insert the operating systems for the entry we just created. We do this once for ESXi 4.1:

USE ReplaceWithYourDatabaseName;
SET IDENTITY_INSERT dbo.guest_os_type ON;
INSERT INTO dbo.guest_os_type
(guestos_id, display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version,
supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto,
is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version)
VALUES (81, 'ESXi 4.x', 'vmkernelGuest', 6, 1, 1, 8, 3072, 7, 1, 1, 4, 8, 0, 0, 0, 0, 107, 40);
SET IDENTITY_INSERT dbo.guest_os_type OFF;

And once more for ESXi 5:

USE vmvblvcd15;
SET IDENTITY_INSERT dbo.guest_os_type ON;
INSERT INTO dbo.guest_os_type
(guestos_id,display_name, internal_name, family_id, is_supported, is_64bit, min_disk_gb, min_memory_mb, min_hw_version, supports_cpu_hotadd, supports_mem_hotadd, diskadapter_id, max_cpu_supported, is_personalization_enabled, is_personalization_auto, is_sysprep_supported, is_sysprep_os_packaged, cim_id, cim_version)
VALUES (82, 'ESXi 5.x', 'vmkernel5Guest', 6, 1, 1, 8, 3072, 7,1, 1, 4, 8, 0, 0, 0, 0, 107, 50);
SET IDENTITY_INSERT dbo.guest_os_type OFF;

Should the query analyzer give an error on the 81 or 82 values, you can increase these, because that just means that these values were already in use in the table. Just increase the numbers until the query analyzer doesn’t give you an error anymore.

And that’s it. You should now be able to see the new options when you create a new virtual machine for your vApp.

There are some additional steps to follow if you actually want to use the newly created options though. You need to restart the vCloud Director daemon on your vCloud cells, and re-prepare your hosts. Also, make sure to set promiscuous mode for the portgroups backing your vCloud network infrastructure, and you can check the post virtuallyGhetto for the details on that.

General, VMware

VMware vExpert – Nominations for 2012 now open

It’s been almost been one year since VMware made a call to nominate folks for the VMware vExpert title, and now it’s time to nominate folks once again. To blatantly quote some figures, here are the number of vExperts over the last coupe of years:

  • 2009 — 253
  • 2010 — 300
  • 2011 — 326

And that number will hopefully again increase this year.

So, what makes up a vExpert? Well, it’s basically simple. You nominate or apply for the title here: http://vmware.com/go/vexpert2012, and that’s all there is to it.

So, what makes someone a vExpert? Let me quote the description from the nomination page:

The VMware vExpert Award is given to individuals who have significantly contributed to the community of VMware users over the past year. vExperts are book authors, bloggers, VMUG leaders, tool builders, and other IT professionals who share their knowledge and passion with others. These vExperts have gone above and beyond their day jobs to share their technical expertise and communicate the value of VMware and virtualization to their colleagues and community.

In the past, this meant that you would apply and be evaluated by a group of folks inside of VMware, and then get awarded the vExpert title, or wouldn’t receive said title. However, all of the vSpecialist were judged by two categories. This year things have changed slightly, and people can classify themselves or the person they are nominating along three categories. To quote some details:

Seeing how well this program has worked so far, we wanted to grow it to include more VMware enthusiasts who may be doing their work of sharing the know-how away from the limelight of the Internet and public events. Our vExperts in the past have for the most part fallen into two implicit groups: bloggers/writers/evangelists and VMUG leaders. This year, we are making explicit three different paths to becoming a vExpert. As always, the common theme for the established and the new vExpert paths will be going above and beyond your day job to help others be successful with VMware solutions.

Evangelist Path
The Evangelist Path includes book authors, bloggers, tool builders, public speakers, and other IT professionals who share their knowledge and passion with others with the leverage of a personal public platform to reach many people. Employees of VMware can also apply via the Evangelist pathway.

Customer Path
The Customer Path is for internal evangelists and community leaders from VMware customer organizations. They have contributed to success stories, customer references, or public interviews and talks, or were active community contributors, such as VMUG leaders.

VPN (VMware Partner Network) Path
The VPN Path is for employees of our partner companies who lead with passion and by example, who are committed to continuous learning and to making their technical knowledge and expertise available to many. This can take shape of event participation, video, IP generation, as well as public speaking engagements.

Although we’re making the three paths explicit this year, there is only a single vExpert designation; we aren’t splitting the program into sections.

The exact details for the three paths can be found on the nomination page, and you will also find the other criteria and guidelines linked from there.

So, what are you waiting for?! Go and check it out, nominate yourself or someone you feel deserves it, and pay it forward! 🙂

Virtualization, VMware, vSphere

Changing a forgotten ESXi 5 root password

It shouldn’t happen, but most folks I’ve spoken to have run in to this at some point in time. You are trying to log on to your ESXi host, and for some reason your root password isn’t working anymore.

The official stance that VMware has taken on this can be found in knowledge base article 1317898, and at the time of writing, it states the following:

ESXi 3.5, ESXi 4.x, and ESXi 5.0

Reinstalling the ESXi host is the only supported way to reset a password on ESXi. Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode do not apply.


So, after searching a bit and combining infos from several folks, I’ve found a way to reset the password, but you should note that this is not officially supported by VMware!


First off, I would recommend you empty your host of any running virtual machines, and put it in to maintenance mode. Next up, inside your vSphere Client, go to the “Home” screen, and select “Host Profiles”, or just press “Ctrl + Shift + P”. Once you are there, create a new profile from an existing host, and select the host that has the unknown password, and give it a name that you can remember.

Next up, edit the newly created profile and open up the “Security Configuration” section. From there, select the “Administrator Password” option, and in the right hand drop down menu, select “Configure a fixed administrator password”.

Now, you can set a new password, but please be careful about one thing. You need to set the password with a certain complexity level. For the exact details, have a look at VMware knowledge base entry 1012033, which states that the default password complexity policy that is set with PAM has the following default:

password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min=8,8,8,7,6which actually means the following:

  • retry=3: A user is allowed 3 attempts to enter a sufficient password.
  • N0=12: Passwords containing characters from one character class must be at least twelve characters long.
    example: chars1234567
  • N1=10: Passwords containing characters from two character classes must be at least ten characters long.
    example: CHars12345
  • N2=8: Passphrases must contain words that are each at least eight characters long.
    example: software
  • N3=8: Passwords containing characters from all four character classes must be at least eight characters long.
    example: CHars12!
  • N4=7: Passwords containing characters from all four character classes must be at least seven characters long.
    example: CHars1!
  • Example: password requisite /lib/security/$ISA/pam_passwdqc.so retry=3 min= 12,10,8,8,7

If you don’t actually follow these rules, and try to apply the profile, you will get a pretty cryptic error message that in my case just stated the following:
Authentication token manipulation errorWhich isn’t that helpful.

Once you have set the new password, you just need to select the profile you have created, and you need to attach your host to this profile. Once that is done, go to the “Hosts and Clusters” view (again, click “Ctrl + Shift + H” to jump there immediately), and right click your host. Select “Host Profile” from the menu, and from there click “Apply profile”.

Now, if SSH was disabled by the applied profile, enable it again by going to the “Configuration” tab, selecting “Security Profile” and going to the properties of the “Services” part. You can start the SSH service from there. Now you can log on using the newly assigned password and that was that.

But…! There’s always a but, isn’t there? This change only works as long as you keep the host profile, or as long as the host stays withing your vCenter. So, what can you do to make the change permanent? Simple, you log on via SSH, change the password with the “passwd” command and then run the auto-backup.sh script from /sbin.

Also, if you would like to work around the password complexity policy, you can modify the following file:
/etc/pam.d/passwd to reflect your own policy. If you want to do this, create a backup of the file, modify it to reflect your own policy. After that, change the password and run the auto-backup.sh script.

Again, these last steps are not recommended by me or VMware, and this will impact the security of your system, so be extremely cautious of your changes! I’m just trying to document the steps so you might have it a bit easier should this situation occur.

EMC, Storage, VAAI, VMware, VNX, vSphere

“My VAAI is Better Than Yours” – The file side of things

EMC VNXI have to admit it. I stole, or rather “borrowed”, part of this title from a blog post of a colleague of mine, Erik Zandboer. He just now published a post on the mindset behind VAAI, and what the actual effect is on the array itself, and on your vSphere infrastructure.

VAAI was already available in vSphere 4.1, and with the switch to vSphere 5 some new features were introduced, which means that as of this release, we now have the following situation:

Block: File:
HW accelerated Zeroing NFS – Full Copy
HW accelerated Copy NFS – Extended Statistics
HW accelerated Locking NFS – Space Reservation

Some folks will say that I left out Thin Provision Stun, which is true. And while it does help to resolve some issues, I left it out because I don’t really view it as a hardware offload, which is what I’m trying to focus on.

I took the hardware in our lab, – a EMC VNX 5300 -, for a spin in our vSphere 5 setup to show the same thing as Erik showed in his blog, but instead showing off some of the File / NFS accelerations.

To get the VNX to actually support NAS VAAI offloading and get the result you expected, you need to meet the following prerequisites:

  • vSphere 5 – You need vSphere 5 installed with an Enterprise or Enterprise Plus license
  • VNX OE for File 7.0.35.x – You need your VNX Operating Environment for File to be at least at version VNX OE 7.0.35.x or newer
  • NFSv3 – The offloads only work on NFSv3-based datastores
  • The vSphere NFS VAAI offload plugin which is referenced here

If all those prerequisites are met, you should normally be able to go in to your vSphere Client and see Hardware Acceleration as Supported:

You could also enable SSH for your ESXi host, – do this by going to the individual host, click on the “Configuration” tab, select “Security Profile” and start the SSH service -, and check the support from the command line. For block devices you could enter the following command:

esxcli storage core device vaai status getand get back a result that shows you the NAAID, the VAAI plugin name, and the primitives with their support state. By using the following command:
esxcli storage core device list you get a similar output, but again this only works for block devices, and won’t really help you when checking the support for NFS. I haven’t found any way so far to get a reliable statement back via SSH, but I’ll try to continue looking and update this post if I find something.

In case of the VNX, we can actually check on the array itself to see if we are using the primitives, so I’m actually showing you the output from the array itself, using the following command on the VNX:

server_stats server_2 -monitor nfs.v3.vstorage -type accu -i 1
First off, I went back in to my ESXi host and went in to the NFSv3 datastore that was hosting my virtual machine. In this case, a Windows 2008 server, running an SAP Enterprise Portal, and I used the vmkfstools to create a clone:

vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdkand I set off a snap using a similar command:
vmkfstools -i GI-C-SAP-EPBW.vmdk CLONE-GI-C-SAP-EPBW.vmdk. All the while, I had the VNX command that I posted before running in a different window. The output from the VNX was showing that we are actually using the VAAI NFS offloading functions:

server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Timestamp Op Max Average
uSecs uSec/Op
09:07:14
09:07:15
09:07:16
09:07:17
09:07:18 vaaiFastClone 1 0 0 0
vaaiVxAttrs 3 0 1 0
vaaiRegister 5 0 0 0
09:07:19
.......
09:08:27 vaaiOffloadStatus 1 0 0 0
vaaiVxAttrs 7 1 1 0
vaaiRegister 10 0 0 0
09:08:28
09:08:29
09:08:30
09:08:31
09:08:32 vaaiOffloadStatus 2 0 0 0
server_2 NFS VAAI op VAAI Op Calls VAAI Op Total uSecs VAAI VAAI Op
Summary Op Max Average
uSecs uSec/Op
Minimum vaaiFullClone 0 0 83308 -
vaaiFastClone 0 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 0 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 0 0 0 0
Average vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 0 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 3 0 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 5 0 0 0
Maximum vaaiFullClone 0 0 83308 -
vaaiFastClone 1 0 0 0
vaaiOffloadStatus 2 0 0 0
vaaiOffloadAbort 0 0 0 -
vaaiVxAttrs 7 1 1 0
vaaiReserveSpace 0 0 0 -
vaaiRegister 10 0 0 0
(sorry for the formatting, I couldn’t get it to show the way it should).

Once the files are created, use a:
vmkfstools --extendedstat GI-C-SAP-EPBW.vmdk on the source file, or on the snap or clone to actually display the extended statistics. The “Capacity bytes” show the allocated space for the virtual disk, the “Used bytes” displays the blocks used for the virtual disk which in case of our snapshot is the fast clone and it’s parent. The “Unshared bytes” displays the usage of the actual fast clone itself without the parent.

I should point out that the offload did speed up my full clone operation, but it was “only” in the range of 20%. That isn’t a great deal, but using both esxtop and the vSphere Client performance graphs showed that the ESXi server was busy doing what it is supposed to do: virtualizing my resources! And that’s the most important thing, isn’t it?

Isilon, SRM, Storage, Virtualization, VMware

Problem with the EMC Isilon Storage Replication Adapter

VMware vCenter SRMA lot of folks out there use the VMware vCenter SRM to create and manage disaster recovery scenarios for their virtualized environments.

Besides having a button to click to fail over (parts of) your environment to a different site, it has one benefit: It forces you to think about your systems. You need to consider which systems are vital to your infrastructure, and you need to be aware of dependencies that you may have in your environment. There are numerous other things that SRM can help with, but that’s not what I wanted to highlight here.

A couple of days ago, I was at the VMware office in Munich, and was helping setting up a SRM 5.0 demo that would serve as a hands-on lab for people interested in SRM. The base of this SRM installation is a virtualized Isilon cluster, that offers the ability to easily provision storage, and offers replication between sites (a quick video overview by my colleague Nick Weaver can be found ).

While setting up the Isilon SRA which you can download from the VMware website, I ran in to a problem. When you download and extract the actual SRA, you’ll get a bunch of PDF files, and two executables. One is the installer for the actual storage replication adapter. It’s called “EMCIsilonSRASetup_1_0.exe”, and you need a current Java development kit to get that one running, but it should install correctly.

The second file is called “IsilonReplicationHelperSetup.exe”, and this is used to configure the SRA before using it in SRM. Now, when starting this helper, both me and Jase McCarty have seen errors that refer to a missing Java class (com.izforge.izpack.installer.Installer), for a program called IzPack which was used to create the installer. After extracting the actual executable, it seemed like some classes/libraries were missing from it.

I’ve been in touch with Isilon support after running in to the error, and after checking with them, they gave me an MD5 hash of a working copy of the IsilonReplicationHelperSetup.exe, which is:
416535bc1c7d7f133037af04b5502e3b However, MD5 for the executable that I got was:
4342E880A99EE2ED6DA1205F1018233DWhich obviously is different. The MD5 of the downloaded file, and the MD5 that VMware shows for the actual zip that contains the SRA matched up though.

So, I’m putting this post out there as a word of warning. It seems like one of the Isilon SRA files on the VMware website is non-functional. Should anybody out there see this, make sure to contact Isilon support and reference case 00169080, which is my case number.

I’m still working with the Isilon support to see what the next steps are going to be, and I’m sure this is going to be resolved soon enough, but I wanted to put this information out there for you in the meantime, to avoid people having to go through the same process as I did. It might save some folks a bit of time. And I’ll make sure I update this post when I get a solution from the Isilon support team.

Update – January 16th 2012:

While I’m still working with the Isilon support group to get everything sorted out, I did get a version of the IsilonReplicationHelperSetup Java archive that seems to be working. Now, I’m sharing this with you all while we try to get things resolved, and to get the working download on the VMware site, but I need to add a large disclaimer:


This file is not officially supported by EMC and/or Isilon, and while this file worked for me, your mileage may vary, and I would recommend that you do not use this file in a production environment! The file might work in a test environment, but please refrain from using it in a productive environment. Use the official files from the VMware download site, or create a case with Isilon and/or VMware support!


Now, to help you verify this file, the MD5 for the Java archive is:
FFAC907E70FD0BFC73076793B9D5FCB4and you can get the file here.


Update – February 10th 2012:

VMware has updated the Isilon SRA file, and the new MD5 for Version 1.0, (released 01/18/2012) currently is:d8b8408ab259d64ee3f5a83486e2a25eThis actually contains the working files, so you should be all set. 🙂

EMC, Storage, V-MAX, Virtualization, VMware

VMAX VSA: IT’S ALIVE!!!!!!!!!!!!!!!!

So folks, here’s a shameless copy of a blog post from one of the guys on my team. Dave was just brilliant and actually created a virtual storage appliance of the EMC VMAX. I think that’s downright awesome, and I wanted to help him get attention for what he did, so I asked him if I could copy his blog post, which is what you will find here:

young_frankenstein_doc_small

 

As the title suggests there is indeed a Symmetrix VMAX VSA. I have been working on this project since shortly after EMC World. As I look back through my emails, I received the code on 6/3/11 and I have been working on it in almost all of my free time since then.

Now finally it will make its public debut this week at VMworld 2011 as part of the EMC Interactive Demo booth on the show floor. As part of its grand unveiling I thought I would tell you a little about what makes it work.

Now to make a few things clear up front, this is a science project, I cannot distribute it, it does “work”. As part of the lab (I will publish the guide) the student actually provisions an iSCSI disk from the VSA to a ESXi 5.0 host.

One of the first things I noticed with the code when trying to virtualize it. It’s HUGE. There are 2 parts to the VSA.

1. The Service Processor (SP). In a physical VMAX this is the 1U server that is racked in the system bay. It has a special image of Windows XP and contains all of the proprietary software used to manage a VMAX. If you own a VMAX this is what you will see EMC field service personnel using when they come to work on your system. This is NOT accessible by a end-user as it requires special RSA credentials that change weekly. (one reason we can’t distribute it). Its specs are 2vCPU and 2GB of RAM and about 10GB of disk space.

2. Enginuity. This is the Operating Environment of the Symmetrix. For the purposes of this VSA it runs in a SuSE Enterprise Linux 11VM. One of the big deals with the VMAX was that Enginuity was ported from a PowerPC CPU to a Intel x86 based architecture. Without this change this VSA would never exist. Now this VM is big, so big as a matter of fact i had to use a RC build of vSphere 5 in order to even get it to work. I was finally able to scale it down a bit, but at one point it was using 32 vCPU’s 92GB of RAM and about 250GB of disk space.

Obviously one of the challenges for using this in a lab is that I needed it to use fewer resources. In the beginning this VMAX was a Single Engine model, which means it had 16 “slices” running. Each director has 4 DA (backend) directors, and 4 FA (front end) directors. I quickly found this was the biggest reason i needed so much memory and CPU. After working with one developer Chakib, who totally rocks by the way. We were able to scale this down to 1 FA and 1 DA per director. One interesting side note, when I was going down this path I asked Chakib what kind of VM he was using to test this. His reply was, “I am not using this in a VM, I have a physical Linux box with 200GB of RAM”. So I clearly had some work to do. But in its current state it uses 8 vCPU and “ONLY” 48GB of RAM. Which is still pretty darn big, but a lot better than it was when we started.

The networking requirements are pretty simple, the SP needs 1 Public NIC so that we can use its management tools. 2 Internal NICs which is used for internal communication to the directors. In our case that’s the Linux VM. The Linux VM needed the 2 internal NICs and 1 NIC to present an iSCSI target to. Then we put out ESXi host’s VMkernel NIC on the same vSwitch so it can use the iSCSI target provided by the VSA.

So that’s all great you say, but what actually works? That’s a good question.

What works is using Standard Devices, and very small ones today. One of the things I was told when I was given the code was that this WON’T and CAN’T do any I/O. Which obviously proved to be a bit of an issue. Chakib really worked his butt of to get me something that does I/O. So this is not like the Celerra UBER VSA by @lynxbat, where you can run a VM off of it. We hope we can do that one day. Thin Pools work to the extent you can create them, and put devices in a pool, but when you present it to a host it will not work. This kept me from using the VSI SPM plugin for vSphere as part of my lab, hey we always have next year! The really neat part to me is that the internal tools (SymmWin) that run on the SP fully work. It’s like having an actually VMAX, but without all the fuss of getting a few 50A power drops. As an ex-customer this to me is the coolest part, I got to put on my own BIN files, use Inlines (internal tool used to directly talk to the hardware). As a total nerd this thing is a dream come true.

So what’s next?

Well a lot of that depends on YOU! Since this is a total science project we need to show those in Symmetrix Engineering this is worth putting their time and money into. I need everyone here at VMworld this week to come try this thing, give me feedback, leave comments here, and if you aren’t at the show, express your desire for us to continue working on it. If no one is interested this will ultimately die on the vine. Please fill out this form so we can show how many of you all would like to see this project continue.

I have to give special thanks to Chad Sakac (@sakacc), Chris Horn (@horn_Chris) for getting me involved in this project and letting me run with it. Also all of the support they gave me during this process.

Here is a link to the lab guide being used this week at VMworld. Take a look and let me know what you think!

VMAX Lab Guide

Big thanks to Matt Cowger (@mcowger), Scott Lowe (@scott_lowe), and Tee Glasgow (@teeglasgow) for their help with the lab guide. Also to Rick Scherer (@rick_vmwaretips) for the blog help