EMC VNX Pool LUNs + VMware vSphere + VAAI = Storage DEATH

**Cliffs notes – a bug in the VNX OE causes massive storage latency when using vSphere with VAAI enabled – disabling VAAI fixes issue**

Hello, and welcome to my very first blog post! I’ve owned this domain and WordPress subscription for nearly a year and a half and am finally getting around to posting something on it. Considering I’ve spent the last 3 years focused on end user computing, and the majority of that being done with Citrix products, I always figured my first post would be in that domain…but alas, that was not the case.

The problem…

I recently started a new gig and one of the first orders of business was untangling some storage and performance issues in a vSphere 5.5 environment running on top of a Gen 1 EMC VNX 5300.  It was reported that there was very high storage latency, often resulting in LUNs being disconnected from the hosts, during certain operations like a Storage vMotion or deploying a new VM from template.

After a general review of the environment I was able to rule out a glaringly obvious mis-configuration, so I turned to a couple useful performance monitoring tools – Esxtop and Unisphere Analyzer.  While I am by no means an expert with either tool, with a little bit of Google-fu and the assistance of a couple great blogs (which I’ll link to later in this post), I was able to get the info I needed to verify my theory – a bug involving VAAI that was supposed to be addressed in the latest VNX Operating Environment (which at the time of this posting is 5.32.000.5.215) still exists.

I started out by doing some performance baselining with VisualEsxtop (https://labs.vmware.com/flings/visualesxtop) so I could get a picture of what the hosts were seeing during operations that involved VAAI (Storage vMotion, deploy-from-template/clone, etc.)  As you can see in the below screenshot, the VNX is quite pissed off.  The “DAVG” value represents disk latency (in milliseconds) that is likely storage processor or array related.  The “KAVG” value represents disk latency (in milliseconds) associated with the VMkernel.  Obviously, the latency on either side of the equation is nowhere near a reasonable number.  Duncan Epping has a great overview of Esxtop (http://www.yellow-bricks.com/esxtop), I highly recommend you give it a read if you’re newer to the tool like I am.

1

The next step was to use EMC’s Unisphere Analyzer to get a picture of what was occurring on the storage side during these operations.  If you’re not familiar with Unisphere Analyzer, an EMC employee created a brief video on how to capture and review data with it (http://www.youtube.com/watch?v=yCMZ_N7-p7A) – it’s a relatively simple tool that you can garner a lot of valuable information from.  I used it to capture storage side performance metrics during the two following tests.

2

The first test consisted of a Storage vMotion of a VM with VAAI enabled on the host (1 Gb iSCSI to the VNX).  This test moved the VM from LUN_0 to LUN_6, starting at 9:46:44 AM and finishing at 10:01:53 AM.  If you look at the corresponding time period on the Unisphere Analyzer graph you’ll see that response time is through the roof.  While it did not occur during this test, the hosts would often lose their connection to the LUNs during these periods of high latency…not good, obviously.

These warnings always show up in the vSphere Client when this issue occurs (yeah yeah, I’m not using the Web Client for this):

3

The second test consisted of a Storage vMotion of the same VM with VAAI disabled.  This test moved the VM back to LUN_0 from LUN_6, starting at 10:04:13 AM and finishing at 10:14:03 AM.  This time, the Unisphere Analyzer data looks MUCH better.

2

Here is some an example of what Esxtop looked like during the test with VAAI disabled:

4

The LUNs with ~ 1400 read/write IO are obviously the ones involved in the Storage vMotion…notice the lack of “SAN choking”.  I re-ran this test multiple times using other LUNs with identical results…it was obvious at this point that there is still an issue with VAAI being used on this VNX OE.  Fortunately, our production datacenter utilizes 10 Gbe for the iSCSI network and Storage vMotions finish in just a minute or two.  I could see this flaw being particularly problematic in larger environments where Storage vMotion is frequent or something like VDI where VM’s are frequently spun up, tore down, or updated.

The solution…

Obviously, disabling VAAI in vSphere is a guaranteed “work around”.  I wouldn’t necessarily call this a “fix” as the VAAI feature is unusable, but it will stop the high latency and disconnects when vSphere tries to offload certain storage tasks to the array.  Once I had some hard evidence in hand, I did open up a ticket with EMC, and the support engineer was able to confirm this was indeed still a bug and has not been addressed by the latest OE version.

This VMware KB article details the process of disabling VAAI (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665).  I found that just the “DataMover.HardwareAcceleratedMove” parameter in the article had to be disabled.  The EMC support engineer also mentioned they had some success increasing the “MaxHWTransferSize” parameter while leaving VAAI enabled, but that it hadn’t worked for everyone.

You can see more information from this KB article (https://support.emc.com/kb/191685 – you may need an active support account, I had to login to view this page).  I decided to just disable VAAI and call it a day until a valid bug fix was released in some future OE version.  ***update 11/06/15*** it has come to my attention the preceding EMC KB191685 can no longer be accessed at the supplied link…I searched through the support portal and could not find a replacement so I don’t know if they pulled the KB documenting this issue entirely or if it’s been merged into another.  I did however find a support bulletin from June 2015 saying that the VAAI improvements had been added into the .217 firmware.  At one point I did request the .217 firmware only to find out they’d pulled it due to some issue it was causing.  I can only assume the VAAI improvements would’ve been added into some subsequent firmware version but no longer have my VNX’s in production, nor are they under support, and I won’t be able to personally test.

Hopefully this information will be beneficial to someone at there…luckily I found my way through the rabbit hole, but there wasn’t a whole lot publicly available regarding this issue when I was initially seeking a cause.

Advertisements

11 comments

  1. Fantastic write up. I’m experiencing similar issues (although not as severe) specifically when doing any type of operation involving a pool. Funny enough, when using just old school RAID groups, VAAI seems to perform like a champ, cutting a 7 min deploy from template to 2 min.

    1. Yep, it only affects storage pools so traditional RAID group LUNs can still leverage VAAI without issue. Obviously, disabling VAAI on the host side is going to affect all attached LUN types.

    2. Just a heads up, I was discussing this issue with EMC support recently and they stated there is now a hotfix for this specific issue. I’ll be applying it to my arrays in the next few weeks so stay tuned…

    1. 5.32.000.5.215 when I found the problem, it was supposedly resolved in subsequent firmwares…though I just got an advisory email from EMC last week saying that the .217 firmware has a VAAI bug that causes SP panics or something. So it sounds like something still isn’t quite right. Luckily we have migrated away from that storage to a new platform where VAAI does work just fine.

    1. I was surprised to see this, I believe at the time I was having this issue about 2 years ago, only gen 1 VNX’s were affected and gen 2 worked fine. I know there were subsequent gen 1 firmwares that were supposed to fix it but ended up not, and we EOL’d that array before it was fixed. Hopefully you will have better luck with a quick resolution, it’s a bummer not being able to use VAAI if you frequently need that feature.

      1. Thank you, Neal. My main concern is that Storage DRS will be affected by this issue. I will let you know when have more information from the vendors.

      2. We also had exactly the same issue with a second gen VNX 5600 and ESX 5.1i – Disabling VAAI fixed the latency issues..and they were through the roof ( 7500ms ) at times. Mainly occouring overnight..something to do with backups I suspect. So this issue is defo still around.
        What is annoying is that Vmware still show a compatability list that declares the VNX second gen is compatable with VAAI..which it clearly isnt still. We installed and applied latest OS around May 2016. Very annoying. Great post btw and thanks for this, saved some bacon I can tell you.

      3. Sorry to hear you are experiencing this too – I’m surprised a fix hasn’t been put in place yet considering VAAI is an integral feature. I “fixed” this issue over 2 years ago now, and I know they were having these problems with the arrays at least a year or two prior, if not for the whole time the VNX was owned…so we’re talking 3-4+ years now.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s