Tag: Symmetrix

EMC, Symmetrix, V-MAX, vCenter Operations, Virtualization, VMware

Setting up the EMC Symmetrix adapter on the vCenter Operations Manager 5 vApp

Bas Raayman

I present a lot on vCenter Operations Manager, a pretty neat monitoring tool from VMware. I like this tool a lot, because getting started with it is easy enough, and you have a plethora of features once you dive deeper in to it, and the best part about it? If you use the “big” version, -Enterprise or Enterprise Plus that is-, you can even monitor your applications and non-virtualized infrastructure. To monitor your things that go beyond your virtual machines, you can install so called “adapters”. In a nutshell, such an adapter is nothing more than a piece of software that tells vCenter Operations how to connect to things, and how to interpret the results it gets back. Now, EMC has created such an adapter for their VMAX and Symmetrix storage arrays, and has created a document that tells you how to set up and configure the adapter. That way, you can get loads of information from your storage system inside of vCenter Operations. Great stuff, right? Original image from: http://my.opera.com/supergreatChandu8/albums/showpic.dml?album=5466862&picture=82475462Yeah ok, maybe not so great. The biggest problem, is that the documentation seems to have been created for the normal installable version of vCenter Operations. However, VMware has also created a version in the form of an appliance, a so called vApp. You download the files, deploy the vApp, enter the IP-addresses of both virtual machines that are contained in the vApp, and away you go. Wonderfully easy to install, and besides certain limits in scalability, it offers pretty much the same functionality as the normal installer. This is where the problem starts if you want to use the EMC Symmetrix adapter. You can find almost all adapters on the Integrien FTP site, and there’s a folder containing all the files you need to get started with the Symmetrix adapter right here. My teammate Matt Cowger actually wrote a nice blog post on how to configure and set up the Symmetrix adapter. This works like a charm, except for one tiny thing that you will run in to when using the vCenter Operations vApp. When you go to create an adapter instance, you need to give it a name, indicate if you want to auto discover everything, and you need to input a path to the “EMC Symmetrix Main Input Folder”. This is the folder where you actually archive all of the performance and configuration data from your storage system. The documentation tells you that this should be:

* If the main input folder is on a remote Windows machine, you must share the folder before you add the adapter instance. Do not map the main input folder. Windows services do not work with mapped drives. * If the main input folder is on a remote Linux machine, you must mount the folder to the Collector server before you add the adapter instance.

Problem being, that if you actually have your Solutions Enabler host running on Windows, you need to input a UNC path in the format of \\servername\sharename. But the problem here is that the virtual machines inside the vApp do not come with any access methods for Windows shares. You won’t find any tools like mount.cifs, smbclient or even have the option to specify smbfs as the type of file system to mount. And that means what? Well, you will have two options to overcome this situation. You can either install the Services for Unix/Services for NFS on your Windows host and set up an NFS share on your Windows machine. Or, you can migrate your Solutions Enabler host to a Linux machine and set up everything there. OK, so how do I configure this stuff under Linux? Glad you asked. You can follow some of the steps from the post that Matt created, but I’m going to write them down here anyway so you will have one page with all the steps you need. I’m going to assume that you have already set up your Linux machine, and that you have installed the Solutions Enabler package. Go in to the following file: /usr/emc/API/symapi/config and add these following lines at the end of the file, then make sure you save your changes (create a backup of the original, this is always a good idea):

storstpd:dmn_run_spa = disable
storstpd:dmn_run_smc = disable
storstpd:dmn_run_ttp = enable
storstpd:dmn_run_ttp_on_sp = disable
storstpd:dmn_run_rtc = disable
storstpd:ttp_collection_interval = 5
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:ttp_dev_metrics = disable
storstpd:ttp_disk_metrics = disable
storstpd:ttp_dgdev_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:ttp_se_nwi_metrics = enable
storstpd:ttp_re_sg_metrics = enable
storstpd:ttp_re_nwc_metrics = enable
storstpd:ttp_rdflnk_metrics = enable
storstpd:ttp_se_tcp_metrics = enable
storstpd:ttp_se_nw_metrics = enable
storstpd:use_compression = enable

Next, restart the storstpd daemon:

/opt/emc/SYMCLI/bin/stordaemon shutdown storstpd /opt/emc/SYMCLI/bin/stordaemon start storstpd

Check if the daemon is up and running again by issuing the following command. The first line should show the Daemon State as “Running”:

/opt/emc/SYMCLI/bin/stordaemon show storstpd

Now, since the Analytics VM will be actually collecting the information from the adapter, it needs to be able to access the files from your Solutions Enabler host. Since the Analytics VM will be running the collection process as a user called “Admin”, we need to consider something. The admin user on the vCenter Operations appliance will be running with a user ID (UID) of 1000, and a group ID (GID) of 1003. That means that we should either install our Solutions Enabler using a user with the same user ID and group ID, or we need to map some things so that the admin user can actually access the files later on. In order to export the directory with the required files for the Symmetrix adapter, we will add the following line to /etc/exports: /usr/emc/API/symapi/stp *(rw,insecure,all_squash,anonuid=0,anongid=0) Obviously, this isn’t the best you can do from a security perspective, so feel free to change these options as needed for your environment, but basically what we are doing here is this:

  • The * just means that all IP-addresses have access. You can change this to for example the IP of the analytics VM.
  • RW means that the export is created with read and write access.
  • Insecure means that clients can use non-reserved ports.
  • All_squash means that all users get mapped to the anonymous user account
  • anonuid=0 means that the anonymous user ID will get mapped to the user ID 0. Be careful since this is the root account!
  • anonguid=0 means that the anonymous group ID will get mapped to the group ID 0. Again, this is the root group!

If you did install your Solutions Enabler as a different user, make sure that you map the anonuid and anonguid to the respective numerical IDs, to allow access to the files we are going to export. Now, we simply restart the NFS server, or have it re-read its config should it already be online, using:

/etc/init.d/nfsserver restart

or

exportfs –ra

We can check if the export is working, using the following command:

showmount –e localhost

Now, we create a scheduled job to archive the Solutions Enabler file. To do that, add the following line to your crontab: 2-57/5 * * * * /opt/emc/SYMCLI/bin/stordaemon action storstpd -cmd archive This will cause the job start at 2 minutes past the hour, and run in 5 minute intervals. Check under /usr/emc/API/symapi/stp/ttp, to see if you have a new directory. Normally the directory should be the serial number of your storage array, and contain compressed files inside of that directory that contain the information the Symmetrix adapter will need. Final thing to do right now, is log on to the analytics VM, and create a folder where we will mount the required files. For example create a directory called /media/VMAX. Once you have created the directory, edit /etc/fstab to contain the following line: 10.10.10.10: /usr/emc/API/symapi/stp /media/VMAX nfs rw,lock 0 0 Make sure you change the IP address to match that of your Solutions Enabler host, and then mount the directory using the following command:

mount /media/VMAX

If you don’t have a firewall blocking communication, you should now be able to traverse the subdirectories and access the files. Finally, you can now configure the adapter, and input the directory you just mounted as the “EMC Symmetrix Main Input Folder”. So, in the text field, simply enter the following as the path:

/media/VMAX/ttp

If you test the adapter now, you should see it come back successfully, and after giving it a bit of time, start working with the data you are now importing from your VMAX/Symmetrix system. 🙂

General, Networking, Storage, Virtualization

My “Follow, even if it’s not Friday” list

Bas Raayman

There’s a meme on Twitter that can be witnessed each Friday. It’s called “Follow Friday” and can be found by searching for the #FollowFriday hash tag, or sometimes just simply abbreviated to #FF to save on space in a tweet.

Problem with a lot of those follow Friday tweets is that most of the time you have no idea why you are being given the advice to follow these people. If you are lucky you will see a remark in the tweet saying why you want to follow someone, or why I should follow all of these people, but in most cases it’s a matter of clicking on a person, going to their time line and hope that you can find a common denominator that gives you an indication of why you want to follow someone.

In an attempt to do some things differently, I decided to create this post and list some of the folks that I think are worth following. And I’ll try and add a description of what someone (or a list of people) do that make them worth following in my opinion. And if you are not on this list please don’t be offended, I will try to update it every now and then, but it would be impossible for me to pick out every single one of you on the first attempt.

So here goes nothing! I’m starting off this post with people that offer a great deal of info on things related to VMware, and I will try to follow up with other topics as time goes buy. Check back every now and then to see some new people to follow.

Focus on VMware:

  • @sakacc – Besides being the VP for the VMware Technology Alliance at EMC, Chad is still a true geek and is a great source of knowledge when it comes to things VMware and EMC. Also, very helpful in regards to try and help people who have questions in those areas. Be sure to check out his blog as it is a great source of information!
  • @Kiwi_Si – Simon is a great guy, and can tell you a lot about VMware and home labs. Because of the home labs he is also very strong when it comes to finding out more about HP’s x86 platform, and once again I highly recommend reading his TechHead blog.
  • @alanrenouf – This French sounding guy is actually hiding in the UK and is considered by many to be a PowerCLI demi-god. Follow his tweets and you will find out why people think of him that way.
  • @stevie_chambers – You want to find out more about Cisco UCS? Steve is the man to follow on Twitter, also for finding out more about UCS combined with VMware.
  • @DuncanYB – Duncan started the Yellow Bricks blog, which emphasizes on all things VMware, and also is a great source of info on VMware HA.
  • @scott_lowe – Scott is an ace when it comes to VMware.
  • @jtroyer – John is the online evangelist and enterprise community builder at VMware. For anything new regarding VMware and it’s community you should follow John.
  • @lynxbat – I would call it something else, but Nick is a genius. He started tweaking the EMC Celerra VSA and has worked wonders with it. I highly recommend following him!
  • @jasonboche – Virtualization evangelist extraordinaire. Jason has the biggest home lab setup that I know of, I’d like to see someone trump that setup.
  • @gabvirtualworld – Gabrie is a virtualization architect and has a great blog with lot’s of resources on VMware.
  • @daniel_eason – Daniel is an architect for a large British airline and knows his way around VMware quite well, but is also quite knowledgeable in other areas.
  • @SimonLong_ – With a load of certifications and an excellent blog, Simon is definitely someone to follow on Twitter.

Focus on storage:

  • @StorageNerve – Devang is the go-to-guy on all things EMC Symmetrix.
  • @storageanarchy – Our friendly neighborhood storage anarchist is known to have an opinion, but Barry is also great when it comes to finding out more about EMC’s storage technology.
  • @valb00 – Val is a great source of info on things NetApp, and you can find a lot of good retweets with useful information from him.
  • @storagebod – If you want someone to tell it to you like it is, you should follow Martin.
  • @Storagezilla – Mark is an EMC guy with great storage knowledge. Also, if you find any videos of him cursing, tell me about it because I could just listen to him go on and on for hours with that accent he has.
  • @nigelpoulton – Nigel is the guy to talk to when you want to know more about data centre, storage and I/O virtualisation. He’s also great on all areas Hitachi/HDS.
  • @esignoretti – If you are (planning on) using Compellent storage, be sure to add Enrico to your list.
  • @chrismevans – The storage architect, or just Chris, knows his way around most storage platforms, and I highly recommend you read his blog for all things storage, virtualization and cloud computing.
  • @HPStorageGuy – For all things related to HP and their storage products you should follow Calvin.
  • @ianhf – “Don’t trust any of the vendors” is almost how I would sum up Ian’s tweets. Known to be grumpy at times, but a great source when it comes to asking the storage vendors the right questions.
  • @rootwyrm – As with Ian, rootwyrm also knows how to ask hard questions. Also, he isn’t afraid to fire up big Bertha to put the numbers to the test that were given by a vendor.
  • @sfoskett – Stephen is an original blogger and can probably be placed under any of the categories here. Lot’s of good information and founder of Gestalt IT
  • @Alextangent – The office of the CTO is where Alex is located inside of NetApp. As such you can expect deep technical knowledge on all things NetApp when you follow him.
  • @StorageMojo – I was lucky to have met Robin in person. A great guy working as an analyst, and you will find refreshing takes and articles by following his tweets. A definite recommendation!
  • @mpyeager – Since Matthew is working for IT service provider Computacenter, he has a lot of experience with different environments and has great insight on various storage solutions as well as a concern about getting customers more bang for their buck.

Focus on cloud computing:

  • @Beaker – Christofer Hoff is the director of Cloud & Virtualization Solutions at Cisco and has a strong focus on all things cloud related. His tweets can be a bit noisy, but I would consider his tweets worth the noise in exchange for the good info you get by following him. Oh, and by the way… Squirrel!!
  • @ruv – Reuven is one of the people behind CloudCamp and is a good source of information on cloud and on CloudCamp.
  • @ShlomoSwidler – Good cloud stuff is being (re)tweeted and commented on by Shlomo.

So, this is my list for now, but be sure to check back every once in a while to see what new people have been added!

Created: May 27th 2010
Updated: May 28th 2010 – Added storage focused bloggers
Updated: July 23rd 2010 – Added some storage focused bloggers and some folks that center on cloud computing
Updated:

EMC, Storage, Symmetrix

Shorts: Trouble with symapi_db.bin causing erratic behavior

Bas Raayman

Usually when you are connected to a EMC Symmetrix array you will install the Solutions Enabler package on your system. Solutions Enabler is basically both a set of tools to help you manage your Symmetrix arrays, as well as an API. The Solutions Enabler basically creates a small database that displays what Symmetrix arrays are connected to the host you are running the software on, the so called SYMAPI database that you will find as a file on your system called “symapi_db.bin”.

Under a normal situation you will run a discover process to initially scan and fill the database with entries. To do that you can issue the command:

symcfg discover

This will start the scan operation, and depending on the amount of arrays and the configuration on those arrays you can plan anywhere from just under a minute for a scan up to several minutes. Once the file has been created you could try opening the file and searching for strings inside of the file, and you will find a lot of information about devices, device paths, disk IDs and lot’s more.

Now, in some situations after your array configuration has changed, it is useful to refresh the database file. Under normal circumstances this should all be easily done and without any issues.

However, in some cases your database file might be facing problems, without manifestation in any obvious ways. I have seen cases where new devices would simply not show up. Other examples are error messages about disks that can not be reached because of access control list errors.

If you happen to have some erratic behavior on one of your hosts, you might want to try one thing before creating a service request in Powerlink. You might want to try creating a copy of your database, removing it and then performing a new discover. Some steps to help you do just that:

  • Create a backup of your device and/or composite groups using the symdg/symcg commands.
  • Rename your old symapi_db.bin to something else.
  • Issue a “symcfg discover” to create a new symapi_db.bin
  • Import your device and/or composite groups from the backup file(s) you created.

This won’t help you in all situations, but it helped me solve several cases were we were seeing erratic behavior on our hosts, and it might do the trick for you.