Army ELA: Weapon Of Mass Confusion?

When the U.S. Army inked what was -- and still is -- VMware's largest enterprise licensing agreement ever in September 2013, it thought it was getting a sweet deal that would result in significant savings and operational efficiencies.

Yet the $44.8 million VMware ELA, and a subsequent $33.3 million six-month extension, have led to higher costs for some Army commands due to certain aspects of the contract that appear to be tilted in VMware's favor, sources with knowledge of the deal told CRN in interviews over the past two months.

At least one subordinate Army command -- the Army National Guard -- declared a moratorium on VMware purchases earlier this year because it received larger-than-expected bills from the VMware ELA and needed to stay within its IT budget. That moratorium remains in effect today.

[Related: Department Of Defense Cancels VMware's Proposed $1.6 Billion Enterprise Licensing Deal]

In a broader sense, VMware's Army ELA has called into question the viability of ELAs -- contracts often structured as "all-you-can-eat" IT buffets that have been big revenue generators for VMware and other enterprise vendors.

Some industry watchers say ELAs are more about locking in customers than saving them money, and could eventually become obsolete as the pay-as-you-go cloud computing model catches on.

The issues the VMware ELA is causing for the Army also have raised the question of who is responsible for determining the ultimate cost of complex contract vehicles -- whether it's the customer, the vendor or the solution provider that works with the customer.

As IT departments seek to transform themselves into internal service providers, these sorts of ELA questions are coming to the fore. Yet in the case of VMware's Army ELA, no one seems to have any answers.

In reporting this story, CRN spoke with enterprise vendor executives, government solution providers and industry analysts, and most believe there's plenty of blame to go around for the problems the VMware ELA is causing.

Some sources said VMware's federal sales team pushed an ELA that included overly advantageous terms for the vendor. Other sources question whether high-ranking executives in the Defense Information Systems Agency (DISA) -- the Department of Defense branch in charge of IT purchasing -- did enough due diligence on the VMware ELA to ensure the vendor wasn't overstepping its bounds.

Some of CRN's sources think the Army's CIO office -- known as the CIO/G-6 -- didn't adequately prepare Army commands for the ELA's implications. The Army's current CIO, Lt. Gen. Robert S. Ferrell, took office in December 2013, two months after VMware's Army ELA went into effect. However, Ferrell was in his role when the VMware ELA extension was executed.

Finally, there are questions about what role Carahsoft, the VMware government reseller that was awarded the ELA, played in setting expectations on the ultimate cost of the contract for the Army.

VMware, despite being a leading provider of software that powers private and public clouds, has been slow to make the transition to being a cloud service provider. Many VMware partners view this as the biggest challenge it has faced in its 17-year history.

Like many enterprise software vendors, VMware, Palo Alto, Calif., still relies heavily on software maintenance fees. In fiscal 2007, maintenance accounted for about 25 percent of VMware's total annual revenue. In fiscal 2014, maintenance accounted for about 50 percent of VMware's $6 billion-plus annual revenue, according to the vendor's 10-K filings.

DISA in February solicited a five-year, $1.6 billion VMware joint enterprise licensing agreement (JELA) for the DOD covering the Army, Navy, Air Force and other DOD agencies, but canceled it a month later after four rival vendors protested, arguing it would give VMware an unfair competitive advantage.

The DOD JELA would have enabled VMware to show that the DOD -- the world's largest employer with more than 3.2 million people -- believes in its "software-defined data center" vision, which involves not just server virtualization, but storage and network virtualization, as well as hybrid cloud services.

Meanwhile, the Army has been operating without a VMware ELA in place since the end of March. DISA said it's working with VMware on a new Army ELA, which it expects to solicit in late May. DISA also expects to solicit a new JELA in the federal government's fiscal 2016, which starts in October.

The big question now is whether VMware's new ELA will be structured differently than the old one. If it's not, sources told CRN they think Citrix Systems, which was shut out of Army desktop virtualization business by VMware's old ELA, could challenge the deal. And any delays in getting the new Army ELA done could make it harder for VMware to get its new DOD JELA approved, said the sources.

Inside The Army ELA

In September 2013, the CIO/G-6 paid $44.8 million for a one-year VMware ELA that covered 19 products, including server virtualization, management, desktop virtualization and end-user computing software. The ELA was meant to cover the Army's VMware needs while DISA hammered out details of the JELA.

In addition to covering software upgrades and maintenance renewals for VMware products the Army already had bought, the ELA allowed for unlimited downloads of additional VMware product licenses. The VMware ELA was set up so that the Army would be billed for the additional product licenses at the end of the one-year term.

"The exact cost to your command will be determined after license allocation," according to VMware ELA documentation on the Army's Computer Hardware Enterprise Software and Solutions (CHESS) website. "License allocation will be re-evaluated yearly to determine the new cost per license for each command."

During the contracting process for the VMware ELA, the Army's Independent Government Cost Estimate concluded that the ELA "provided cost savings when compared to procuring licenses and maintenance in a decentralized manner," a spokesman for the Army CIO/G-6 said in an emailed statement.

VMware's ELA pricing represented savings of more than 40 percent from what the U.S. General Services Administration charges for VMware licensing and support, the spokesman said. The ELA "also established mechanisms to provide command IT and resource managers centralized visibility into procurements and costs," said the spokesman. It also "required IT and resource manager approval for technical and budgetary requirements before licenses could be acquired and activated."

Yet despite these safeguards, some Army commands evidently didn't realize that each time they downloaded additional product licenses under the ELA, they were racking up ongoing software maintenance and support charges that would cause sticker shock down the road. And for reasons that remain unclear, the Army's CIO/G-6 office, which was in charge of billing Army commands for their usage under the ELA, didn't send bills at the end of the one-year deal.

Sources familiar with the matter told CRN it wasn't until January, 16 months after the original VMware ELA took effect, that Army commands began receiving bills. And some of the bills, sources said, were significantly larger than what commands previously had been paying for VMware software.

The Army National Guard received a larger-than-expected VMware bill from the CIO/G-6 and responded by declaring a moratorium on VMware purchases, Rick Breitenfeldt, chief of media relations for the National Guard Bureau, confirmed to CRN May 21.

"The Army National Guard implemented the VMware moratorium in order to remain within current budgetary constraints," Breitenfeldt said in an email. "The moratorium on VMware purchases in the Army National Guard remains in effect."

Breitenfeldt declined to comment on the size of the bill the Army National Guard received, or when it received it, referring these questions to the Army CIO/G-6 office. Sources told CRN the Army National Guard's VMware ELA bill was between $14 million and $15 million.

Sources told CRN that the Army's Warfighter Information Network-Tactical (WIN-T), which provides satellite and terrestrial communications network infrastructure for soldiers in battlefields around the world, also received a bill of around $15 million from the CIO/G-6 in January stemming from the VMware ELA. WIN-T's bill was about three times larger than what it had previously been paying for VMware software, said one of the sources.

It's unclear if WIN-T also declared a moratorium on VMware purchases, and its representatives didn't respond to a request for comment.

The U.S. Army Corps of Engineers is another command that's unhappy with the size of the bill stemming from the VMware ELA, sources said. CRN reached out to it for comment on how much it paid, but hadn't heard back as of press time.

Before the VMware ELA, Army commands negotiated their own ELAs individually with VMware, and managed maintenance agreements directly with the vendor. The Army CIO/G-6 did the VMware ELA in order to standardize the process of buying VMware software, get cheaper pricing and consolidate management of multiple licenses and contracts. The ELA was mandatory for all commands that wanted VMware software.

DISA didn't respond to a request for comment about the Army ELA.

CRN sent roughly two dozen questions about the VMware ELA and the bills to Gregg Judge, chief of the Army CIO/G-6 enterprise licensing division, on March 29. He acknowledged receiving them on April 7. As of press time, neither Judge nor other Army CIO/G-6 officials had answered the vast majority of CRN's questions.

The Army spokesman told CRN that CIO/G-6 ELA staff conducts "annual data calls" with each Army organization to verify and validate their current VMware inventory.

"This is how the Army updates VMware inventory and clarifies costs for organizations under ELAs managed by CIO/G-6," said the spokesman. "The centralized visibility that ELAs provide to the Army enable us to better monitor and manage software products in use by organizations."

CRN also sent several questions to Carahsoft regarding its role in preparing the Army commands for the ELA, as well as other matters, but its representatives didn't respond.

A VMware spokesman said in an emailed statement to CRN that the U.S. Army has "achieved significant cost savings and a return-on-investment in the millions of dollars" from the VMware ELA.

The VMware spokesman declined to respond to CRN's specific questions about the ELA, referring them instead to the Army CIO/G-6 office. "How the Army or any of our clients conduct internal billing for usage across their agency is an area VMware does not have visibility into; this would be a question for the Army," the VMware spokesman said.

What's Behind The Sticker Shock?

Sources told CRN that some Army commands viewed the VMware ELA as an all-you-can-eat buffet that already had been paid for, and didn't realize that each time they downloaded additional software licenses they were accruing additional software maintenance costs.

DISA expected the DOD-wide JELA to be in place by the time the Army ELA expired in September 2014, but for reasons that remain unclear, it wasn't ready. So the Army extended its VMware ELA for six months, paying an additional $33.3 million, bringing the total cost of the ELA to $78.1 million.

Although the ELA extension looks like a hefty price hike, a VMware spokesman told CRN the $44.8 million cost of the original Army ELA included discounted pricing that took into account some earlier ELAs Army commands had done that were still active at the time.

However, two sources with knowledge of the matter said the $33.3 million reflected the additional software maintenance costs the Army took on when it downloaded additional VMware software licenses during the first year of the contract.

A source close to VMware told CRN the Army CIO/G-6 funded the first year of the VMware ELA on its own and didn't intend to recoup that cost by billing individual commands. But the Army CIO/G-6 did send bills to commands to fund the Army's portion of the upcoming DOD JELA, and sources said those bills were larger that expected for a couple of reasons.

In addition to unlimited software downloads, the VMware ELA also required Army commands to download entire suites of products to get commonly used products such as vSphere server virtualization and View desktop virtualization, sources told CRN. To get vSphere, Army commands would have to download the entire vCloud Suite Enterprise, which also includes vCloud Automation Center (now known as vRealize Automation), vCenter Operations Management Suite (now known as vRealize Operations), vCloud Director and three other products.

"The ELA does not allow you to ’purchase' vSphere separately from vCloud Suite Enterprise," according a Carahsoft FAQ on the Army ELA.

Carahsoft's Army ELA FAQ was publicly available on the CHESS portal until sometime in early April, when it vanished from the site without explanation. Neither Carahsoft nor the Army CIO/G-6 responded to a request for comment on why it was removed and whether it's still publicly available.

The VMware spokesman told CRN the vendor began letting Army commands download stand-alone vSphere approximately three months after the Army ELA went into effect. "VMware did this to meet specific needs of the Army commands," said the spokesman.

However, there's no mention of this on the Army CHESS portal. The VMware spokesman said the stand-alone vSphere option was listed on the Army's download portal for the ELA. The VMware spokesman didn't respond to a follow-up question about whether Army commands could also download VMware View on a stand-alone basis.

When Army commands downloaded the server and desktop virtualization suites, some were getting additional products that they didn't need, two industry sources familiar with the ELA told CRN. Making vSphere only available through vCloud Suite Enterprise is one way VMware can create the impression in the marketplace that vSphere works best when used in conjunction with its other products, sources said. It also helps VMware show Wall Street that its newer products are selling.

VMware charged the Army $4,953 for each download of vCloud Suite Enterprise, on a per-CPU basis, according to a document on the CHESS portal billed as a "VMware ELA Cost Calculator." The VMware ELA Cost Calculator also vanished from the CHESS portal sometime in early April without explanation. Neither Carahsoft nor the Army CIO/G-6 responded to a request for comment on why it's no longer available on the website.

Army commands that downloaded vCloud Suite Enterprise, but just installed vSphere, would still be on the hook for maintenance costs for the other seven products in the suite, even if they didn't use them. VMware referred CRN's question about this aspect of the ELA to the Army CIO/G-6 office, which didn't respond.

Why Some Fingers Are Pointing At VMware

Some of CRN's sources feel that VMware, by convincing DISA to agree to an ELA that required Army commands to download suites, took advantage of a situation to sell commands products they didn't need.

A VMware spokesman told CRN that the U.S. Army uses "all components of the vCloud Suite in production." The spokesman didn't respond to a follow-up question about how many of the 51 Army commands listed in the VMware ELA are actually doing so today.

Given how quickly word travels in government IT circles, VMware's silence on the Army ELA issues could cause other government agencies to question whether ELAs are as advantageous as they appear, said one high-ranking industry executive.

"Unless VMware starts explaining what's going on here, they're going to start making enemies inside the Army, because the perception of what's happening here is not good," the executive told CRN. "They're going to look like the bad guys here if they don't start speaking up."

Meanwhile, downloading unlimited quantities of VMware software licenses expanded the total install base that the Army CIO/G-6 used to calculate maintenance costs for the JELA, which also helped fuel the larger bills to commands, according to CRN's sources.

Further complicating matters, CRN's sources said Army commands began receiving the bills in January, about four months after the expiration of the original VMware ELA. It's not clear how these bills were calculated, or what time frame they covered, but the sources said it's possible that commands that downloaded additional licenses during the six-month VMware ELA extension will see additional maintenance costs.

VMware referred CRN's questions about this to the Army CIO/G-6 office, which didn't respond.

Another factor fueling the big bills is that the VMware DOD JELA included many more products than the Army ELA, including newer VMware offerings such as NSX software-defined networking, VSAN storage virtualization and vCloud Government Service, the vendor's FedRAMP-certified public cloud.

Because the JELA was much larger in scope, the Army CIO/G-6 had to come up with more funds to cover its share of the DOD JELA -- particularly the unlimited downloads. Instead of calculating Army commands' bills based on their actual requirements, the CIO/G-6 instead based them on the amount of VMware licenses the commands had downloaded previously, the sources said.

The sources said they don't think the Army CIO/G-6 sufficiently explained the mechanics of the VMware ELA. "VMware's Army ELA was like telling a bunch of kids to go into this candy store, eat whatever you want, and we'll send the bill to your parents at the end of the year," said one of the sources.

While it's the Army CIO/G-6's responsibility to make sure individual commands understand how ELAs work, "VMware should have been more proactive about reporting to the Army CIO/G-6 the potential for abuse," said one longtime solution provider who has previously worked on government ELAs.

But although VMware federal sales may have been overly aggressive in this case, the CIO/G-6 should not have been blindsided by the bills, one government solution provider executive told CRN. "It's the CIO/G-6's fault for not realizing what they were getting themselves into when they signed the VMware ELA. How do you not run a model when you're taking on this kind of ELA?" said the source.

"What government needs to understand is, not only is it industry's job to be transparent with their offerings, it's also the government's responsibility to be diligent with managing and forecasting the operations and maintenance costs of ELAs," the source added.

The JELA Arrives, Fizzles

At the same time VMware was putting together and executing the Army ELA, it was working on the DOD JELA, which was a much larger, more aggressively constructed volume licensing agreement. When DISA solicited the JELA in February of this year, it proposed a five-year, $1.6 billion deal that was structured in much the same way as the Army ELA, including unlimited product licenses with billing in arrears, and suites of products instead of stand-alone products.

The JELA included not only server and desktop virtualization and management, but also newer products such as NSX software-defined networking, VSAN storage virtualization and AirWatch mobile device management. The DOD JELA would have given VMware a high-profile reference customer for its software-defined data center pitch, which involves using all of these products in concert with each other.

On the surface, the VMware JELA looked like a win-win for VMware and the DOD. In addition to volume discounts on VMware software, the JELA would consolidate the Pentagon's more than 2 million VMware product licenses and 9,270 separate procurement transactions signed over the past five years into a single agreement. With a simpler agreement, the DOD could reduce the number of staff needed to manage its VMware licenses, thereby saving taxpayers money.

Consolidating licensing agreements is a primary goal for DISA, which in the past two years has inked JELAs with Microsoft, Cisco Systems and Adobe Systems. None of those deals appear to have caused the kinds of problems the Army is experiencing with the VMware ELA.

JELAs are part of the DOD's campaign to make its operations more efficient, John Slye, a research analyst at Deltek, a Herndon, Va.-based software vendor that works with the federal government, said in an interview. "They've been saying, with multiple commands buying licenses you're getting overlap, and that gets to be expensive. They felt they were buying too much, and they want to consolidate it and manage it better," Slye said.

As it did with the Army ELA, DISA argued that only VMware and its resellers could handle the work involved in the JELA, and sought to award the contract to them without opening it up for competitors to bid on. DISA raised the possibility of security problems that could occur if VMware software wasn't maintained and updated properly.

In DISA's JELA proposal, all of these products would have been made available to DOD agencies. The JELA would have also anointed VMware as the Pentagon's exclusive public cloud provider for the duration of the contract. Sources told CRN this aspect of the contract was particularly ambitious, since VMware came late to a public cloud market where Amazon Web Services, Microsoft and Google are widely regarded as the leaders.

VMware, like many enterprise vendors, is looking to carve out its turf in a federal cloud market that's still in its infancy. IDC expects federal government public cloud spending to grow from around $173 million in fiscal 2014 to more than $3 billion in fiscal 2017.

The biggest problem with the DOD JELA, according to multiple sources familiar with the agreement, is that it included several VMware products that are sold by other vendors. Which is why challenges began almost immediately after DISA solicited the JELA RFP Feb. 9.

First, Amazon Web Services, Citrix, Nutanix and Microsoft -- through its government reseller Minburn Technology Group -- filed protests against the JELA with the Government Accountability Office (GAO), arguing that the DOD JELA would give VMware an unfair advantage in the government market.

The vendors each had their own specific reasons for protesting the JELA. AWS and Microsoft sell public cloud services, Nutanix sells storage virtualization, and Citrix sells desktop virtualization. "None of the protesters wanted to risk being shut out of any portion of the potential market opportunity for government customers," said one longtime government solution provider who tracked the deal.

Cisco, which sells software-defined networking, didn't file an official protest of the JELA, but did complain directly to DISA, two sources close to the action told CRN. A Cisco spokesman declined comment.

On March 12, DISA canceled the JELA and the GAO dismissed the vendors' protests.

A VMware spokesman said the vendor "is extremely proud of the technology support" it has provided to the Army and the DOD. "While VMware cannot comment on specifics of any of our client's confidential contracts with our company, nor the recent decision taken by the DOD, we look forward to our ongoing partnership, supporting their critical mission.

"While both [the Army ELA and the DOD JELA] are structured very differently, both agencies have achieved significant cost savings and return on investment through their use of VMware technology," the VMware spokesman said.

VMware first referred to the JELA in a quarterly earnings call last July, indicating at the time that it had already worked on the deal for a year. Yet in VMware's first-quarter earnings call in April, VMware President and COO Carl Eschenbach didn't sound optimistic about the JELA's prospects.

"That's something the government controls," Eschenbach said in response to an analyst's question about the deal.

Eschenbach also suggested that the JELA might end up being smaller in scope than planned. "We continue to work with all of the different agencies across the federal government," he said on the call. "Some of them could participate in the [JELA] if it came out and it was closed and some may decide not to."

ELAs overall accounted for 30 percent of VMware's bookings during the quarter, and even without the JELA, VMware's federal business grew "in the midteens," year over year, Eschenbach said on the call. "So we continue to power along despite this thing out there for multiple quarters," he said.

As DISA and VMware work on a new JELA proposal, sources familiar with the matter said the relationship between the two is coming under scrutiny from other enterprise vendors. What's more, certain details of the JELA remain shrouded in secrecy.

DISA's JELA document contains several large chunks of redacted text, including one entire section titled "Determination Of Fair And Reasonable Cost."

In other parts, DISA has redacted specific figures, such as VMware's percentage of the DOD's virtualized environments and the total amount the DOD has
invested in VMware software licenses. The redacted portions have fueled industry speculation about why these and other aspects of the contract were deemed unfit for the eyes of the public.

DISA's rationale for awarding the Army ELA and DOD JELA to VMware without opening it up to competition is also suspect, one industry executive who's been tracking both deals told CRN. "Typically, the justification for sole-sourcing contracts to a vendor is that they only cover maintenance, yet these contracts obviously weren't maintenance-only," said the source.

What Happens Now?

CRN's sources have differing views on who gets the lion's share of the blame for the VMware ELA mess and the cancelation of the JELA.

Jamie Shepard, senior vice president of strategy and health care at Lumenate, a Dallas-based VMware partner that sells to government customers, said he believes the CIO/G-6 should have done a better job of preparing the Army commands for the ELA.

"I cannot for the life of me understand how they missed the fine print," Shepard told CRN. "VMware has always been crystal clear on what you get in an ELA and how much it costs. I have done several ELAs with them, for both private and public sector customers, and they are fairly straightforward and similar."

One government solution provider told CRN he thinks the CIO/G-6 office's resolution to the situation, which was to charge the individual commands for the additional license costs, defeats the purpose of having a VMware ELA in the first place.

"Why have an ELA when you're turning around and charging commands?" said the source. "Clearly, the Army CIO/G-6 should have had much better internal controls over the understanding and deployment of the agreement."

On the other hand, some of CRN's sources feel that VMware took advantage of a chance to railroad the Army into buying products it didn't need -- specifically by making vSphere only available in a suite of products.

Whether this limits future opportunities for VMware is hard to say, Deltek's Slye told CRN. "They are definitely not the first firm to undergo a protest and have things stepped back. But this is a relationship business, so if VMware rankled people, that news travels," Slye said.

Some sources think VMware was too aggressive when it convinced DISA to include SDN, storage virtualization and public cloud services in the DOD JELA. Now that the VMware JELA has attracted protests, whatever revised JELA the DISA solicits in the future is sure to get studied intensely by other vendors, said the sources.

Meanwhile, the Army has been operating without a VMware ELA in place since the end of March. VMware has agreed to provide free technical support and maintenance to the Army until a new ELA is in place.

DISA could announce a new ELA any day now, but if it includes unlimited downloads of desktop virtualization software, sources said it's possible that Citrix -- the vendor most harmed by the first VMware Army ELA -- will protest. Citrix didn't respond to a request for comment on why it didn't protest.

Any challenge to the new VMware Army ELA could further complicate VMware's relationship with the DOD. Sources said it's possible the new VMware ELA will only cover software maintenance in order to avoid protests from other vendors.

As for the JELA, CRN's sources think that DISA may remove the public cloud services component -- and possibly other technologies -- and renew its efforts to award it to VMware or one of its resellers. In any event, the JELA, if it is re-solicited, is almost certain to be much smaller in size than the original, sources said.

The DOD has standardized on VMware and runs mission-critical systems on its software and that's not going to change anytime soon, even though open-source software is making inroads in the market. That said, some sort of resolution to the Army's VMware ELA bills is needed, one government solution provider told CRN.

"The issue is, who should be held responsible? We have a predicament that needs to be resolved. Taxpayer dollars are in limbo with regard to how to pay those bills. Something has to happen to unclog this pipe," said the source.

PUBLISHED JUNE 3, 2015