Cisco, GestaltIT, Tech Field Day, UCS, Virtualization

Gestalt IT Tech Field Day – On Cisco and UCS

There are a couple of words that are high on my list as being the buzzwords for 2010. The previous year brought us things like “green computing”, but the new hip seems to be “federation”, “unification”. And let’s not forget the one that seems to last longer then just one year, it’s the problem solving term “cloud”.

Last Friday (April 9th), I and the rest of the Gestalt IT tech field day delegates were invited by Cisco to get a briefing on Cisco’s Unified Computing System or in short “UCS”. Basically this is Cisco’s view that builds on the notion that we are currently viewing a server as being tied to the application, instead of seeing the server as a resource that allows us to run that application.

Anyone in marketing will know that the next question being asked is “What is your suggestion to change all that?”, and Cisco’s marketing department didn’t disappoint us and tried to answer that question for us. The key, in their opinion, is using a system consisting of building blocks that allow me to to give customers a solution stack.

As the trend can be spotted to go towards commodity hardware, Cisco is following suit by using industry standard servers that are equipped with Intel Xeon processors. Other key elements are a virtualization of services, a focus on automated provisioning and unification of the fabric by means of FCoE.

What this basically means is that you order building blocks from Cisco in the form of blade servers, blade chassis, fabric interconnects and virtual adapters. But instead of connecting this stuff up and expanding my connectivity like I do in a standard scenario, I instead wire my hardware depending on the bandwidth requirements and that’s pretty much it. Once I am done with that, I can assign virtual interfaces as I need them on a per blade basis, which in term removes the hassle of plugging in physical adapters and cabling all that stuff up. In a sense it reminded me of the take that Xsigo offered with their I/O director, but with the difference that Cisco uses FCoE instead of Infiniband, and with Cisco you add the I/O virtualization to a more complete management stack.

The management stack

This is in my opinion the key difference. I can bolt together my own pieces of hardware and use the Xsigo I/O director in combination with VMware and have a similar set-up, but I will be missing out on one important element. A central management utility.

This UCS unified management offers me some advantages that I have not seen from other vendors. I can now tie the properties to the resources that I want, meaning that I can set up properties tied to a blade, but can also tie them to the VM or application running on that blade in form of service profiles. Things like MAC, WWN or QoS profiles are defined inside of these service profiles in an XML format and then applied to my resources as I see fit.

Sounds good, but…..?

There is always a but, that’s something that is almost impossible to avoid. Even though Cisco offers a solution that seems to offer some technical advantages, there are some potential drawbacks.

  • Vendor lock in:
    This is something that is quite easy to see. The benefit of getting everything from one vendor also means that my experience is only as good as the vendors support is in case of trouble. Same thing applies when ordering new hardware and there are unexpected problems somewhere in the ordering/delivery chain
  • The price tag:
    Cisco is not know to be cheap. Some would even say that Cisco is very expensive, and it will all boil down to one thing. Is the investment that I need to make for a UCS solution going to give me the return on invest? And is it going to do that anytime soon? Sure it can reduce my management overhead and complexity, sure it can lower my operational expense, but I want to see something in return for the money I gave Cisco and preferably today, not tomorrow.
  • Interoperability with my existing environment:
    This sort of stuff works great when you are lucky enough to create something new. A new landscape, a new data center or something along those lines. Truth is that usually we will end up adding something new to our existing environment. It’s great that I can manage all of my UCS stack with one management interface. But what about the other stuff? What if I already have other Cisco switches that are not connected to this new UCS landscape? Can I manage those using the built in UCS features? Or is this another thing that my admins have to learn?
  • The fact that UCS is unified does not mean that my company is:
    In smaller companies, you have a couple of sysadmins that do everything. They install hardware, configure the operating system, upload firewall policies to their routers and zone some new storage. So far so good, I’ll give them my new UCS gear and they usually know what goes where and will get going. Now I end up in the enterprise segment where I talk to one department to change my kernel parameters, a different to configure my switch port to auto-negotiate and the third one will check on the WWN of my fibre-channel HBA to see if this is matching to the one configured on the storage side. Now I need to get all of them together to work on creating the service profiles, although not all will be able to work outside of their knowledge silo. The other alternative would be to create a completely new team that just does UCS, but do I want that?

Besides the things that are fairly obvious and not necessarily Cisco’s fault, I think that Cisco was actually one of the first companies to go this way and one of the first to show an actual example of a federated and consolidated solution. Because that is what this is all about, it’s not about offering a piece of hardware, it’s about offering a solution. Initiatives like VCE and VCN only show us that Cisco is moving forward and is actually pushing towards offering complete solution stacks.

My opinion? I like it. I think Cisco have delivered something that is a usable showcase, and although unfortunately I have not been able to actually test it so far, I do really like the potential it offers and the way it was designed. If I ever get the chance to do some testing on a complete UCS stack, I’ll be sure to let you know more, but until then I at least hope that this post has made things a bit clearer and removed some of the questions you might have. And if that’s not the case, leave a comment and I will be sure to ask some more questions on your behalf.

The sponsors are each paying their share for this non-profit event. We, the delegates, are not paid to attend. Most of us will take some days off from our regular job to attend. What is paid for us is the flight, something to eat and the stay at a hotel. However as stated in the above post, we are not forced to write about anything that happens during the event, or to only write positive things.

Data Robotics, Drobo FS, GestaltIT, Storage, Tech Field Day

Drobo announces their new Drobo FS

In November 2009, Data Robotics Inc. released two new products, the Drobo S and the Drobo Elite. Yesterday I was lucky enough to be invited to a closed session with the folks from Data Robotics as they had some interesting news about a new product they are announcing today called the Drobo FS.

When we visited the Data Robotics premises with the entire Tech Field Day crew last November, one of the biggest gripes about the Drobo was that it relied on the Drobo Share to allow an ethernet connection to the storage presented from my Drobo. The newly introduced Drobo S added an eSATA port, but also didn’t solve this limitation since it wasn’t even compatible to the Drobo Share. As such the Drobo Share was not the worst solution ever, be it for the fact that it connects to the Drobo via a USB 2.0 connection, thus limiting the maximum speed one could achieve when accessing the disks.

Front of the new Drobo FSWell, that part changes today with the introduction of the Drobo FS. Basically this model offers the same amount of drives as the Drobo S, namely a maximum of 5, and exchanges the eSATA port for a gigabit ethernet port. The folks from Data Robotics said that this would mean that you will see an estimated 4x performance improvement when comparing the Drobo FS to the Drobo Share, and you also get the option of single or dual drive redundancy to ensure that no data is lost when one or two drives fail.

Included with all configurations you will receive a CAT 6 ethernet cable, an external power supply (100v-240v) with a fitting power cord for your region, a user guide and quick start card ( in print) and a Drobo resource CD with the Drobo Dashboard application, help files, and electronic documentation. The only thing that will change, depending on your configuration, is the amount of drives that are included with the Drobo FS. You can order the enclosure without any drives at all, this would set you back $699.- (€519,- / £469,-), or you can get the version that includes a total of 10 terabyte of disk space for a total of $1499.- (€1079,- / £969,-).

As with the other Drobo’s you are able to enhance the function of your Drobo with the so called DroboApps. This option will for example allow you to extend the two default protocols (CIFS/SMB and AFP) with additional ones such as NFS. Unfortunately we won’t be seeing iSCSI on this model since according to the guys from Data Robotics they are aiming more towards a file level solution than a block level solution.

Back of the new Drobo FSOne of the newer applications on the Drobo FS is something that caught my eye. This application is targeted towards the private cloud and uses “Oxygen Cloud” as a service provider to provide file access to a shared storage. This means that you can link your Drobo’s together (up to a current limit of 256 Drobo units) and allow these to share their files and shares. This will include options like access control and even features such as remote wipe, but a more complete feature list will follow today’s release.

One feature that was requested by some users hasn’t made it yet. The Drobo dashboard which is used to control the Drobo is still an application that needs to be installed, but Data Robotics is looking at the option of changing this in to something that might be controlled via a browser based interface. However no comments were made regarding a possible release date for such a web interface. What is also under development on is an SDK that will allow the creation of custom DroboApps. Again, a release date was not mentioned in the call.

I will try to get my hands on a review unit and post some tests once I have the chance. Also, I am looking forward to finding out more about the device when I meet the Drobo folks in person later this week during the Gestalt IT Tech Field Days in Boston, so keep your eye on this space for more to come.

GestaltIT, Tech Field Day

Getting ready for the Gestalt IT Tech Field Day 2010 – Boston

Last year in November I was fortunate enough to be invited to the Gestalt IT Tech Field Day which took place in San Jose. A recap of what happened there can be found here.

Some of you might not be aware of the concept of the tech field days, so let me give you an overview.

The origin might be found in an event that is called “Tech Day” and was initiated by HP. Basically HP invited several bloggers from around the globe and offered them a technical discussion and a more in depth view of several of their products.

Gestalt IT’s Stephen Foskett
was one of the bloggers invited to this event who felt that this might be a good basis for that which now makes up the tech field days.

So, in a nutshell the tech field days brings together a group of independent people that are present in the various social media (think of Twitter, blogs, podcasts, the works) and have a technical background. These good folks then are packed with two days of presentations, discussions and hands on from the sponsors of this event.

You might want to think of this as vendor love, but you wouldn’t be quite right. First of all there is no obligation to communicate about any of the things that are presented to you. What is even more important, when you decide to actually report on what happened, you can give your honest opinion, be it good or bad. Secondly, since the group of people that are invited have a very broad background, the service or products presented will usually get a very broad array of questions fired at them. These will range from very detailed questions that could be about the choice of an algorithm to something more general like for example the value of deduplication in a virtualized environment.

Because we are talking about people with backgrounds in (among others) backgrounds in storage, virtualization, operating systems, hardware, networking and analysts you will find that the questions asked are usually tough on the presenters. These are people that know their stuff and this is also why presenters get the recommendation to not make this in to a marketing show.

This is an event for the community and as such the people who attend are very aware of this fact and looking at the first event, you will see a lot of feedback coming from the people who attend. This is not just limited to the on-site discussions. We had discussions put on video in the pub, there were dynamic conversations in the hotel lobby where the delegates discussed ideas or even took the time to explain concepts to the other delegates who were not experts in the same area.

So, here’s a list of the delegates that will be attending the event:

Jason Boche JasonBoche
Carlo Costanzo VMware Info CCostan
David Davis VMwareVideos DavidMDavis
Greg Ferro EtherealMind
Gestalt IT
Edward Haletky The Virtualization Practice Texiwill
Robin Harris Storage Mojo
ZDNet Storage Bits
Greg Knieriemen Storage Monkeys
Simon Long The SLOG
Gestalt IT
Scott D. Lowe Tech Republic
John Obeto Absolutely Windows JohnObeto
Devang Panchigar StorageNerve
Gestalt IT
Bas Raayman Technical Diatribe BasRaayman
Simon Seagrave TechHead Kiwi_Si
Matt Simmons Standalone Sysadmin StandaloneSA
Gabrie van Zanten Gabe’s Virtual World GabVirtualWorld

If you check out the profiles of the attendees, you will see that these people should make for an interesting mix. What’s more, I am certain that these folks are able to ask questions that are not always easy to answer.

Cisco Systems
Data Robotics
EMC Corporation
Hewlett-Packard Company

So, look for some interesting posts coming from the delegates and on Gestalt IT. You can follow what happens online on Twitter by using the hashtag #TechFieldDay, and be on the lookout for lot’s of interesting things to be coming on April 8th and 9th.

One final thing that should be said.

The sponsors are each paying their share for this non-profit event. We, the delegates, are not paid to attend. Most of us will take some days off from our regular job to attend. What is paid for us is the flight, something to eat and the stay at a hotel. However as stated in the above post, we are not forced to write about anything that happens during the event, or to only write positive things.

GestaltIT, Performance, Storage, Tiering

“Storage tiering is dying.” But purple unicorns exist.

Chris Mellor over at the Register put an interview online with NetApp CEO Tom Georgens.

To quote from the Register piece:

He is dismissive of multi-level tiering, saying: “The simple fact of the matter is, tiering is a way to manage migration of data between Fibre Channel-based systems and serial ATA based systems.”

He goes further: “Frankly I think the entire concept of tiering is dying.”

Now, for those who are not familiar with the concept of tiering, it’s basically moving data between faster and slower media in the background. Clasically tiering is something that every organization is already doing. You consider the value of the information, and based on that you decide if this data should be accessible instantly from your more expensive hardware, and even at home you will see that as the value decreases you will archive that data to a media that has a different type of performance like your USB archiving disk or for example by burning it to a DVD.

For companies the more interesting part in tiering comes with automation. To put it simply, you want your data to be available on a fast drive when you need it, and it can remain on slower drives if you don’t require it at that moment. Several vendors each have their own specific implementation of how they tier their storage, but you find this kind of technology coming from almost any vendor.

Aparrantly, NetApp has a different definition of tiering, since according to their CTO tiering is limited to the “migration of data between Fibre Channel-based systems and serial ATA based systems”. And this is where I heartily disagree with him. I purposely picked the example of home users who are also using different tiers, and it’s no different for all storage vendors.

The major difference? They remove the layer of fibre channel drives in between of the flash and SATA drives. They still tier their data to the medium that is most fitting. They will try to do that automatically (and hopefully succeed in doing so), but just don’t call it tiering anymore.

As with all vendors, NetApp is also trying to remove the fibre channel drive layer, and I am convinced that this will be possible as soon as the prices of flash drives can be compared to those of regular fibre channel drives, and the automated tiering is automated to a point that any actions performed are transparent to the connected system.

But, if NetApp doesn’t want to call it tiering, that’s fine by me but I hope they don’t honestly expect customers to fall for it. The rest of the world will continue to call it tiering, and they will try to sell you a purple unicorn that moves data around disk types as if by magic.

as a Service, General, GestaltIT

Jack of all trades, master of… the solution stack?

Stevie Chambers wrote something in a tweet last night. He stated the following:

The problem with an IT specialist is that he only gets to do the things he’s already good at, like building a coffin from the inside.

And my first thought was that he’s absolutely right. A lot of the people I know are absolute cracks or specialists in their own area. I’ll talk to the colleagues over in the Windows team, and they can tell you everything about the latest version of Windows and know each nook and cranny of their system. I’ll talk to the developers and they can write impossible stuff for their ABAP Web Dynpro installations.

But then I ask my developers what effect a certain OS parameter will have on their installation. Or perhaps how the read and write response times from the storage array in the back-end might influence the overall time an end user spends while he’s waiting for his batch job to complete. And you know what answer you get a lot of times? Just a blank stare, or if you are lucky some shoulders being shrugged. They’ll tell you that you need to talk to the experts in that area. It’s not their thing, and they don’t have the time, knowledge, interest or just simply aren’t allowed to help you in other areas.

So what about our changing environment? In environments where multiple tenants are common? Where we virtualize, thin provision and dedupe our installations and create pointer based copies of our systems? Where oversubscription might affect performance levels? Fact is that we are moving away from isolated solutions and moving toward a solution stack. We no longer care about the single installation of Linux on a piece of hardware, but need to troubleshoot how the database in our Linux VM interacts with our ESX installation and the connected thin provisioned disks.

In order to be an effective administrator I will need to change. I can’t be the absolute expert in all areas. The amount of information would just be overwhelming, and I wouldn’t have the time to master all of this. But being an expert in only one area will definitely not make my job easier in the future. We will see great value in generalists that have the ability to comprehend the interactions of the various components that make up a stack, and are able to do a deep dive when needed or can gather expertise for specific problems or scenarios when they need to.

Virtualization and the whole “* as a Service” model isn’t changing the way any of the individual components work, but they change the interconnect behavior. Since we are delivering new solutions as a stack, we also need to focus on troubleshooting the stack, and this can’t always be done in the classical approach. In a way this is a bigger change for the people supporting the systems than it is for the people actually using those systems.

Clariion, CX3, EMC, GestaltIT, Storage

The Asymmetrical Logical Unit Access (ALUA) mode on CLARiiON

I’ve noticed that I have been getting a lot of search engine hits relating to the various features, specifications and problems on the EMC CLARiiON array. One of the searches was related to a feature that has been around for a bit. It was actually introduced in 2001, but in order to give a full explanation I’m just going to start at the beginning.

DetourThe beginning is actually somewhere in 1979 when the founder of Seagate Technology, Alan Shugart, created the Shugart Associates Systems Interface (SASI). This was the early predecessor of SCSI and had a very rudimentary set of capabilities. Only few commands were supported and speeds were limited to 1.5 Mb/s. In 1981, Shugart Associates was able to convince the NCR corporation to team up and thereby convincing ANSI to set up a technical committee to standardize the interface. This was realized in1982 and known as the “X3T9.2 technical committee” and resulted in the name being changed to SCSI.

The committee published their first interface standard in 1986, but would grow on to become the group known now as “International Committee for Information Technology Standards” or INCITS and that is actually responsible for many of the standards used by storage devices such as T10 (SCSI), T11 (Fibre Channel) and T13 (ATA).

Now, in July 2001 the second revision of the SCSI Primary Commands (SPC-2) was published, and this included a feature called Asymmetrical Logical Unit Access mode or in short ALUA mode, and some changes were made in the newer revisions of the primary command set.

Are you with me so far? Good.

On Logical Unit Numbers

Since you came here to read this article I will just assume that I don’t have to explain the concept of a LUN. But what I might need to explain is that it’s common to have multiple connections to a LUN in environments that are concerned with the availability of their disks. Depending on the fabric and the amount of fibre channel cards you have connected you can have multiple paths to the same lun. And if you have multiple paths you might as well use them, right? It’s no good having the additional bandwidth lying around and then not using it.

Since you have multiple paths to the same disk, you need a tool that will somehow merge these paths and tell your operating system that this is the same disk. This tool might even help you achieve a higher throughput since it can balance the reads and writes over all of the paths.

As you might already have guessed there are multiple implementations of this, usually called Multipathing I/O, MPIO or just plainly Multipath, and you will be able to find a solution natively or as an additional piece of software for most modern operating systems.

What might be less obvious is that the connection to these LUNs don’t have to behave in the same way. Depending on what you are connecting to, you have several states for that connection. Or to draw the analogy to the CX4, some paths are active and some paths are passive.

Normally a path to a CLARiiON is considered active when we are connected to the service processor that is currently serving you the LUN. CLARiiON arrays are so called “active/passive” arrays, meaning that only one service processor is in charge of a LUN, and the secondary service processor is just waiting for a signal to take over the ownership in case of a failure. The array will normally receive a signal that tells it to switch from one service processor to the other one. This routine is called a “trespass” and happens so fast that you usually don’t really notice such a failover.

When we go back to the host, the connection state will be shown as active for that connection that is routed to the active service processor, and something like “standby” or “passive” for the connection that goes to the service processor that is not serving you that LUN. Also, since you have multiple connections, it’s not unlikely that the different paths can also have other properties that are different. Things like bandwith (you may have added a faster HBA later) or latency can be different. Due to the characteristics, the target ports might need to indicate how efficient a path is. And if a failure should occur, the link status might change, causing a path to go offline.

You can check the the status of a path to a LUN by asking the port on the storage array, the so called “target port”. For example, you can check the access characteristics of a path by sending the following SCSI command:


Similar commands exist to actually set the state of a target port.

So where does ALUA come in?

What the ALUA interface does is allow an initiator (your server or the HBA in your server) to discover target port groups. Simply put, a group of ports that provide a common failover behavior for your LUN(s). By using the SCSI INQUIRY response, we find out to what standard the LUN adheres, if the LUN provides symmetric or asymmetric access, and if the LUN uses explicit or implicit failover.

To put it more simply, ALUA allows me to reach my LUN via the active and the inactive service processor. Oversimplified this just means that all traffic that is directed to the non-active service processor will be routed internally to the active service processor.

On a CLARiiON that is using ALUA mode this will result in the host seeing paths that are in an optimal state, and paths that are in an non-optimal state. The optimal path is the path to the active storage processor and is ready to perform I/O and will give you the best performance, and the non-optimal path is also ready to perform I/O but won’t give you the best performance since you are taking a detour.

The ALUA mode is available on CX-3 and CX-4, but the results you get can vary between both arrays. For example if you want to use ALUA with your vSphere installation you will need to use the CX-4 with FLARE 26 or newer and change the failover mode to “4”. Once you have changed the failover mode you will see a slightly different trespass behavior since you can now either manually initiate a trespass (explicit) or the array itself can perform a trespass once it’s noticed that the non-optimal path has received 128,000 or more I/Os than the optimal path (implicit).

Depending on which software you use – PowerPath or for example the native solution – you will find that ALUA is supported or not. You can take a look at Primus ID: emc187614 in Powerlink to get more details on supported configurations. Please note that you need a valid Powerlink account to access that Primus entry.

GestaltIT, Virtualization, VMware

Virtualization and the challenge of knowing what your customer understands

Customers only hear what is useful to them. There, I’ve said it out loud and now I’ll just wait for my lapidation to begin. Or perhaps you will bear with me while I try to explain?

As I wrote in previous blog posts we deliver a lot of our systems virtualized. You could call our VM implementation level a big hit, but we keep running in to the same problem, and it’s something that comes close to selective hearing. We have dozens of presentations, slides and info sessions that explain the benefits of a virtual server. And there are several benefits. Things like vMotion or the Site Recovery Manager can help you achieve a certain service level and add value by automating failovers or minimizing downtime due to planned maintenance.

The problem is the translation that is made by the customer. In our case the various internal teams are our customer, and the amount of knowledge around the operating system or the features that a virtualized landscape offers them are not always fully comprehended. You can tell a customer what these features do and how virtualization adds redundancy to the hardware stack. Usually all that arrives is “Less downtime because I can vMotion the virtual machine to a different piece of hardware.”, and the customer will automatically link this to a higher availability of the application that is running on this host.

The problem is this higher availability will not cascade through the entire stack. Sure, a vMotion might save you from a downtime due to hardware maintenance, but it won’t save you from the proverbial blue screen or kernel panic due to a defunct driver. You can write down everything in an SLA, but the thing that stuck in the customers head was the thing that he deduced from the presentation.

Fellow Blogger Storagebod already wrote a piece on “Asking the right questions“, and I fully agree with him that we as a service provider need to start asking the right questions to our customers. We have no other way to find out and offer the service he wants.

But, we also need to find out if the customer is certain that he understands the services we are offering. It’s not about leaning back in your chair and saying “we only offered a 99.5% uptime, it was there black on white”. It’s about going the extra mile and making it clear that “We offer an infrastructure that can give you 99.999% uptime, but your application isn’t laid out for that same uptime”. It’s about asking the right questions, but it’s also making sure that your customer heard the entire message, not just the parts he wanted to hear.

Virtualization is a tool or a way you can help your customer obtain higher uptimes. It can enable, but won’t guarantee availability of the entire stack.

GestaltIT, Virtualization, vSphere

The RAM per CPU wall

There is one thing that keeps coming up lately in our office. We install a lot of systems virtually, and have over 8000 virtual servers installed.

RAM.jpgNow, as anyone who has installed SAP will probably tell you, our software can do a lot of things, but one of the things it swallows up is RAM. We seem to be doing a bit better on the central instance side, but moved a lot of the “RAM hunger” elsewhere. This is especially true for our new CRM 7.0 which will show you that your application server will need a lot more of it, when compared to older versions.

Now, since these kind of application servers are ideal candidates for virtual machines. With the new vSphere release you can actually give a VM up too 255 GB or RAM, and for larger CRM implementations you might actually consider this limit for your application servers. No problem at all, but you will face an entirely new problem that has been creeping up and has left most people unaware.

Since there’s no real name for it, I just call it the “RAM per CPU wall”, or perhaps even the “RAM per core wall”.

So, what about this wall?

Well, it used to be that computational power was quite expensive, and the same could be said about storage. Times have changed and right now we see that processing power is not that expensive anymore. We have seen a slow moving shift toward the x86 instruction set and it’s x64 extension. With the development of the multi-core processor we are seeing applications being optimized to run in parallel on the chipsets, and with solutions like vSphere we are using more and more of those features. Examples in vSphere would be “Hyperthreaded Core Sharing” or a setting like “cpuid.coresPerSocket”.

It’s great that we can tweak how how our VM platform and our VMs handle multiple cores, and I am certain that we will be seeing a lot more options in the future. Why I am so sure of that? Because we will be seeing the octo-core processors in the not too distant future (I’m betting somehwere around the end of Q2 2010). Just think about it, 8 cores on one CPU.

Now, combine that with an educated guess that we will be seeing 8 socket mainboards and servers, and we are talking about something like the HP DL780 or DL785 with a total of 64 cores. Quite the number right? Now just imagine the amount of RAM you can put in such a server.


The current HP DL785 G5 has a total of 32 cores and a limit of 512 GB, assuming you are willing to pay for the exuberant memory price tag for such a configuration. I’m assuming support for 128 GB RAM modules will take a bit longer, so just assuming 64 cores and 512 GB of RAM will give you 8GB of RAM per CPU core.

It might just be my point of view, but that’s not that much per core.

Now I can hear you saying that for smaller systems 8 GB of ram per core is just fine, and I hear you very good. But that’s not the typical configuration used in larger environments. If you take the CRM example I gave before, you can load up a bunch of smaller application servers and set your hopes on overcommitting, but that will only get you so far.

And the trend is not changing anytime soon. We will be seeing more cores per cpu coming, and the prices and limits of large amounts of RAM are not keeping up, and I doubt they will do so in the future.

So, will we continue to see larger systems being set-up on physical hardware? Actually, I honestly think so. For service providers and large environments things need to change if they want to get a big benefit out of the provisioning of larger machines. If things don’t change, they are facing the “RAM per CPU wall” and will be stuck provisioning smaller instances.

All in all I can only recommend digging down and asking your customers for their requirements. Check the sizing recommendations and try to get some hands on information or talk to people who have implemented said solutions to see how the application handles memory. Make a decision on a per-use case for larger environments. Virtualization is great, but make sure that the recommendation and implementation suits the needs that your customer has, even if that means installing on a physical platform.

EMC, FAST, GestaltIT, Storage

EMC’s FAST, take 1. Action!

As you might have read in my earlier blog post, EMC has announced the release of the first version of their product called “Fully Automated Storage Tiering” or in short “FAST”.

Now, to describe the purpose of this technology in a very simple form, we are talking about the performance of your storage and some logic that will help you put those things that need performance on the fastest bits available in your storage environment.

And that’s about as far as we can go with the concept of simple. Why? Because if this technology is to add value, you need it to be really clever. You would almost need it to be a bit of a mind reader if you will. You will want it to know what your application is going to do, and you will want to know where it does that on the most granular level of your storage, namely the blocks on the disks. Or more simply, you don’t want it to react, you want it to behave proactively.

So, let’s start with some mixed news:

  • FAST v1 is available on Symmetrix V-Max, Clariion CX4 and Celerra NS

  • As some of you will notice these three platforms have something in common. EMC tried to get rid of using custom ASICs in favor of using commodity x86 based hardware for as much as they could. In the new V-Max you will only find a custom ASICs that resides on the Virtual Matrix Interface controller, and is responsible for the coordination of local and remote memory access.

    This swap to x86/x64 and a 64 bit architecture was done on all three mentioned platforms. On its own this is a good thing, but it would also be a good explanation why EMC as of now is not supporting older arrays. EMC is bound to get requests for this new technology for their older arrays like the CX3 or the DMX4. There are two likely options there:

      1: It’s not going to happen.

      Porting the code to a different hardware platform is a pain. The logic behind it is still the same, but the question is, up to where would you backport it? DMX3? DMX2? Where would you draw the line? Combine that with the fact that not all the newer features are available on the older machines and you can probably imagine that it would be easier to just not make these features available on older arrays.

      2: They are almost done and will release it sooner than anyone thought.

      EMC has a lot of developers. Chances are they were also working on FAST for the other platforms and will be releasing it in the not too far future.

    Since we will be seeing arrays being removed from the product purchase portfolio, my money is on option number one. You won’t have the option of buying a DMX3 within the next half-year. And you can also replace half a year with 1.5 year for the DMX4. Sure, you can get extended support which will add four or five years to the life cycle of your array, but implementing new features for arrays which will not be sold anymore in the near future? I find that sort of unlikely.

  • FAST v1 will only work on a LUN level.

  • As explained before, normally your application won’t be updating the data on the entire LUN. Usually you have a few so-called “hot zones” which are just blocks of data are being accessed by reads and/or writes more frequently. An excellent graphical example of this fact is something called a “heat map”. This heat map is created by an (unfortunately) internal EMC application called SymmMerge but fortunately fellow blogger Barry Burke, a.k.a. “The storage anarchist” allowed me to use some images from his blog.

    So, this would be the situation in a fairly common environment:


    Note that in this image we are talking about actual disks, but the image will also work if we just simply replace the word “drives” with “blocks”. The green blocks are doing fine, but the red and orange blocks are the ones that are being accessed a lot.

    The ideal solution would normally be to put the red and orange blocks on a faster medium. EMC would normally tell you that the ideal medium for these kind of blocks would be EFDs or “Enterprise Flash Drives”. And you could put the green blocks on a medium that might not need quite as much performance or the same response times as regular fiber channel drives or perhaps even cheaper SATA drives for bulk storage. Each class of drive (EFD, FCD, SATA) is called a tier, hence the term “Tiering”.

    After a redistribution your image would look something like this, where all blocks would be on a storage class that suits their individual performance needs:


    Now, probably one of the biggest pain points for a lot of people is that this version of FAST is not capable of doing this on a block level. Version 1 is only capable of moving data on to a different tier on a LUN level. But your database/CRM/BW/etc. normally does not read and/or write to the entire LUN.

  • The value of policies.

  • So with this version of FAST you actually put a lot more data on a faster tier than you would actually need to. On the other hand EMC stated that the key value for FAST version 1 is not so much in the fact that you move your LUNs to different tiers, but in the fact that you can set user policies to have the system do this for you. It takes some of the effort involved and handles things for you.

    Now, you can create up to 256 different tiers which in its current version allow you to define tiers based on RAID levels, the speed of a drive and the drive type. It should be noted that the tier definitions will differ when using dynamic or static tiering. Currently disk size and rotational speed are not considered when you create a dynamic tier, so a dynamic tier may contain disks of differing performance characteristics, but a tweet from Barry Burke stated that FAST is actually aware of the RPMs, and knows the latency impacts of contention and utilization. Or at least “it will be in the future

    Now, you can create a policy for a storage group, which is basically a group of disks that are managed as a set, and have that policy associate a storage group with up to three tiers, depending on the tiers you actually have in place. Now, combine that with setting limits for the percentage of capacity that is on a single tier and you will see that you could for example say that you want 80% of you capacity to reside on SATA disks and 20% on EFDs.

    Fast will now apply your policy and, depending on the choice you made, automatically move the data around across those tiers or give you a recommendation on what would be a sensible choice. It can even relocate you data to a different RAID type on the other tier, and your SymmDev ID, your WWN, your SCSI ID and all external LUN references will remain unchanged during the move. If you have replication set-up, that stays active as well.

    Now since this all stuff that might have a performance impact if done during peak loads on your box, the default is that all the moves are performed as lowest priority jobs during time slots or windows that you as the end-user can define. Just keep in mind that you are limited to 32 concurrent moves and to a maximum of 200 moves per day.

  • What will it cost me?

  • Prices start at $5,000 USD for the entry-level systems, and will set you back $22,000 USD for the Symmetrix V-Max. But that is the starting price, and the unbundled price. You could also consider a bundle called the “Symmetrix FAST Suite” that includes the optimizer and priority control & cache partitioning. All to be delivered as a “firmware upgrade” for your array.

  • So do we need to wait for FAST v2?

  • Well, I’ve got mixed feelings on that point. I can see how this first version can add some value to your environment, but that will depend on your environment. People who only use one tier might not have as much value, and adding the cost of new disks in to the equation will not make it any easier. Especially when we take the release of FAST v2 into account that is “planned for GA in 2nd Half 2010” and will also provide support for thinly or virtual provisioned LUNs and be able to move stuff around at the block level.

    I know there is value in this release for some customers that are actually using the V-Max. The automated tiering can at least help you meet a certain service level, but that added value is highly dependent on your environment. Personally, I’d probably wait for the release of version 2 if possible. On the other hand, EMC needs to gain traction first and they were always open about the fact that they would release two versions of FAST, and stated that version 1 would not have all the features they wanted, and that the rest of the features were planned for version 2. I have somewhat of a hard time with some of the analysts who are now complaining that FAST v1 is actually that what EMC said it would be. Did they just ignore previous statements?

  • To sum it all up

  • It’s the same story as usual. Every storage vendor seems to agree on the fact that automated storage tiering is a good thing for their customers. Some have different opinions whether or not the array should be the key in this automation, because you are at risk of the array making a wrong decision.

    EMC started off their journey with some steps towards automated tiering, but they delivered just that, the first steps toward an automated tiering vision. If we would remove the argument of a price tag, I would be almost positive I’d recommend version 2 too any possible customers. For version 1, the answer is not that simple. You need to check your environment and see if this feature makes sense for you, or adds value to your setup.

    Besides FAST we’ve also seen some new cool features being introduced with the new “firmwares” that were released for the various arrays, such as thin zero reclaim and dedupe enhancements. Look for coming posts that will go in too more detail on the new Flare 29 and Enginuity 5874.