Modernizing the Air Operations Center – Robert Gates Wouldn’t Be Happy

The USAF held its Air Operations Center (AOC) Modernization industry day at Hanscom Air Force Base last week. I attended thinking that it would be a quick way to get caught up on the program’s direction. I’ve been doing defense work for about seven years now, so I know how this process works. I went in expecting the day to be heavily scripted and pro forma. Even so, it was so pointlessly byzantine and disconnected from any concrete reality in the field that I left the venue completely depressed.

The AOC (aka Falconer) is like Cape Canaveral for controlling an air war. It is the progeny of the HQ’s in those World War II movies where the airplanes are being pushed around on the big map table in the middle of the room. It’s the place where the Joint or Coalition Forces Air Component Commander conducts command and control. There are big screens on the wall for sharing situational awareness and lots of people doing assessment, planning, and execution processes. When things are busy there are many hundreds of people working there. When it’s not busy, it’s basically a room full of computers but few people.

FE_DA_080609combat_12312



And it really does need to be modernized.

The AOCs today are a dog’s breakfast of systems designed and built independently and then brought together after the fact with little thought given to how the system as a whole functions. Worse, many of the key systems were built to solve completely different business processes from the ones our people are using to fight the current wars. In particular, the “engine of the AOC” is a system called Theater Battle Management Core System (TBMCS). It was built after the first gulf war to handle the production and distribution of very large numbers of interdiction-style mission orders. From a process point of view it’s like a highly specialized word processor. It’s main output is something called the Air Tasking Order which is delivered via a big USMTF mail merge. Unfortunately, in a world of rapid change, drones, dynamic targets, and pre-positioned weapons delivery systems its relevance to our current conflicts has been sorely tested and Excel has found a surprisingly useful role as its sometimes replacement.

That’s not what is bumming me out though. That’s just the way it is; systems built for one thing often have less relevance for another thing. What is depressing me is the way our fear of post-award protest has created a culture so focused on “fairness” that it ensures that absolutely nothing relevant is addressed. After all, if you want to guarantee a fair competition, and fair is more important than effective, just don’t tell anyone anything. This wasn’t a discussion of the AOC and what the people using it care about, this was an exposition on the Federal Acquisition Rules and how constraining they are when over-stringently applied. These days attending an industry day is like listening to a political speech. Much ado, but little said that can be pinned on the candidate later. I didn’t learn a single relevant thing during the entire day, unless you count not learning anything as being at least partially relevant.

Well, I did pick up a few things.

The plan for modernization runs through 2017 and will roll out initial operating capability to stateside AOC’s first. It appears as though the CENTCOM AOC, the one actually prosecuting two wars, isn’t slated to get anything till the very end. This strikes me as backwards but it probably makes perfect sense to an Air Force whose primary focus for modernization is configuration control. The whole notion of “AOC as a Weapons System” seems to distill down to iron clad SOA’fied configuration management. But configuration management of what? What should it DO? The little bit that did make it through the FAR filters merely hinted at the dual chimeras of agility through SOA and a single configuration for every AOC. It’s the paradox of control without an articulated vision.

I think what was most difficult for me was that there was absolutely no mention of the two wars that we are fighting right now or what the users over there need. The words Iraq and Afghanistan were never uttered. Nor were AOC capability priorities linked to the shortcomings in the existing Falconers and how they are impacting those conflicts, particularly those relating to coalition partner participation. The only time a user was mentioned it was with a disparaging remark about “Sgt Snuffy and his local county options.” The first thing I said to my colleagues when I walked out of the theater was “If Bob Gates was here, he’d be pissed.” The meeting was a microcosm of everything he’s been fighting in the way we acquire stuff during wartime.

Maybe software is just too abstract. Maybe airplanes are easier to understand. That idea got me thinking about an imaginary industry day during the Korean conflict for the F-100 going something like this:

“Hi, welcome to industry day. The F-86 is getting a bit long in the tooth and it’s time to modernize. The F-100 needs to be built using the latest trends in aircraft design and construction, so it is going to have a 45 degree swept wing. It will burn JP-5 and it’s aluminum skin needs to be a standard composition widely available from any foundry. We will be carefully following a user board based process for vetting requirements and we’ll be rigorously testing the aircraft against requirements at every step in it’s development. It’s critically important that the design be completely documented and repeatable. If it fails at any stage of testing you will have to redesign it to correct those flaws. We want the new aircraft to be ready for stateside squadrons by 1954 and we’ll roll it out to overseas units by 1958. Furthermore, If you were involved in any of the design or construction of the F-86 you will need a comprehensive organizational conflict of interest plan. None of the people who worked on the F-86 can be involved in the F-100 development in any way, including the RFP response. This is to ensure that your bid doesn’t have an unfair advantage from previous experience. Are there any questions?”

“Ahem, yes, I have a question. What do you want the F-100 to do? Is it an air superiority fighter? Ground attack? How will you be evaluating things like maximum altitude, speed, maneuverability? How will our coalition and service partners participate? And especially, how does it factor into the fight in Korea?…”*

“Let me cut you off, we aren’t going to discuss that here. You can read the Capability Definition Document which provides an overview and then refers to the applicable detailed requirements documents. Once the bidder’s library has been made available you’ll find them all there. If those documents don’t answer your question, you can submit a written question to our office and the answer will be distributed later to all potential bidders.”

The whole thing was just weird, even as these things go. All FARS and process and no real discussion of what problem was being solved, and for whom. There was only one slide that hinted at the service’s end goals from the acquisition – with four blue bubbles labeled with “speed of command,” “cost of capability,” and two more that I’ve already forgotten. That was it. And why even conduct “one on one” discussions if they are nothing more than a venue for sliding written questions across the table? I think the government people were as frustrated by the absurdity of it all as we were.

Well, since the Air Force is too constrained by the FARS to say anything, here’s what I would have said if given the chance…

“Hi, welcome to the AOC Modernization Industry Day. As many of you know, each of the Air Operations Centers grew up somewhat haphazardly as theater HQ’s. As a result they were cobbled together with systems that were designed and built under independent contracts and because of that they don’t integrate well or at all. Naturally this interferes with user experience and mission effectiveness and makes training expensive. Also, there are significant variations from one AOC to the next. This regional flavoring of capability has allowed local commanders to tailor capabilities to their needs, but from a corporate USAF point of view it has made them more difficult to staff and much more costly to maintain. They are self-contained and localized which makes it possible for the commander to chest poke under-performers, but it makes it difficult to share resources or effort across Falconers. A Falconer in a wartime theater will be stressed and busy while the others leave their people and computing power un-utilized.

Five years ago with the Air Operations Center Weapon Systems Integrator (AOC WSI) contract the USAF set out to re-make the AOC’s as corporate assets by converting them all to a common trainable and maintainable baseline called AN/USQ-163 Falconer version 10.1. The AOC would be a weapons system and would be managed like one. While that was an important step (mostly in establishing the notion that it is a corporate / enterprise asset) it hasn’t resulted in the capabilities we need to fight a broad spectrum of conflicts. In fact, despite five years of work, the users are frustrated as hell because they’ve seen very little change at their end. Entire careers are served without any visible improvement in capability.

Users tried to take things into their own hands a few years ago and introduce a dynamic planning application called TBONE, but there was no requirement in the existing CDD for a dynamic planning application. The effort was eventually abandoned (after struggles worthy of Han-era court politics), sacrificed at the alter of configuration management. Instead of seeing local initiative at the edge as a source of innovation, and designing to encourage it, we’ve been mistakenly writing it off as a ‘county option’ and squelching it.

Today’s AOC is under better configuration management and is more consistent across locations than the ones inherited at the beginning of the AOC WSI contract, but there is still vast room for improvement. In particular, its capabilities are still postured more toward the high sortie count interdictions that are the signature of high intensity conflict. There are still key challenges when fighting the dynamic targeting kinds of conflicts found in our current lower intensity wars. Also, and perhaps more critically, the AOC still is almost exclusively focused on planning and high level execution in the Air domain. It is Combat Air Forces (CAF) focused, but integration to the rest of the Theater Air Control system is weak. Also, very little has been done to meaningfully include space and cyber domain operations. Finally, the AFFOR and TRANSCOM missions that are so critical in supporting the component commander in his mission are mostly non-existent in the AOC.

In our messianic focus on configuration control we have created processes too cumbersome and rigid to support real world mission and system evolution. We’ve unintentionally traded chaotic configuration for a rigid and brittle system. As a result warfighters are fighting too much of the war with PowerPoint and Excel. We don’t think that the issue is with configuration management per se, the problem is that we defined the “weapon system” at too high a level. The various Falconers are serving very different missions and need to be able to have different capabilities (and evolve at different rates). We continue to believe in the concept of the AOC as a configuration managed weapon system, however, now we are focusing in on the enabling platform. We call it a Capability Delivery Platform. You might also think of it as an Air Operations Center as a Platform (AooP) if it helps you to think about it in terms of today’s world of cloud computing.

We renamed the CDD to PDD to make our intent clear. The Platform Definition Document doesn’t address each Falconer’s end user capabilities directly, instead it defines the platform capabilities that the core weapon system platform must offer to a broad spectrum of capabilities developers and integrators. From large integrators to local SETA contractors, and even warfighters, we now realize that supporting innovation means supporting development from the corporate core out to the operational edge. Furthermore, the PDD doesn’t address a particular end state. It’s most important platform requirement is to support agility which means that the platform itself is also expected to be in a state of continuous rigorously managed change.

A key goal of such a platform approach is to recognize the tension between configuration management and agility, and satisfy both through a combination of technology, business model, and intellectual property approach.

For too long we have focused on SOA as the sole enabler of integration and flexibility. But ‘SOA’ and all of the ESB baggage that goes with it, is often too much about low level integration of silos and not enough about what happens ‘at the glass,’ where the user is. This is an area where I think the Army is ahead of us. I wouldn’t pick CPOF/CoMotion as our UX environment, but at least they are stressing composability at the glass and they institutionally recognize that that’s where you make a difference for the user. All our talk of SOA for the last seven years has simply been too abstract to mean anything to the users that are actually fighting two wars.

Furthermore, our sole focus on ‘SOA’ as the technical construct behind the AOC has had us ignoring for too long concepts like private clouds and other forms of horizontal run time services. It also leaves data bottled up in disparate systems rather than looking at it as a corporate asset. Making data available from every silo via services simply isn’t the same thing as a comprehensive enterprise data and content strategy. Our approach today is better than those pre-WSDL days, but not much.

Rather than publishing cumbersome standards, we intend to encourage commonality going forward by building compelling runtime services that will become de facto standards through adoption rather than through unenforceable mandate. The work done at NASA on the Nebula project is a very good starting point for what we have in mind.

Outcomes are as much about the business model as they are about technology, if not more. So, we need to ask ourselves the question: ‘How do we use our available contract types and things like award fees to encourage companies of all sizes to innovate and participate in this AOC ecosystem?’ To put a finer point on it, how could we have structured things so that the previous innovation efforts like TBONE would have been facilitated rather than rejected? We think the answer lies in using smaller IDIQ task orders where future work and award fees will be based on use, re-use, and adoption metrics. If you bring an idea to the table like TBONE we’ll incent the big guys to facilitate it’s integration and we’ll award you follow up work if the initial delivery is adopted by users. If it isn’t adopted, it will die. Quickly.

Today the users are just too far removed from the people building these systems. Requirements documents are like low band pass filters and too many of the subtleties that impact user experience just don’t make it through. So we are going to take a different approach to building the ‘little c’s’ on top of the capability platform going forward. We plan to institutionalize the way we accidentally built ADOCS in Korea years ago. We are going to put development teams side by side with our users and ask them to quickly deliver increments of capability in our key theaters where we are fighting right now.

These efforts will take advantage of the runtime API’s available through the certified and accredited Capability Development Platform. Improvements that are adopted by users in theaters actually doing something will be made available for adoption in others. By automating testing, conducting continuous integration and vulnerability analysis, we think we can get the users what they need quickly without sacrificing the configuration management and quality code that we need.

Finally, to make all of this work we are going to demand that work funded under this modernization be handled consistently from an intellectual property perspective. We want the source to be open across our contractor base and made available in forge.mil or an equivalent government owned repository. While contractors will continue to innovate (and may protect their IP) at the application and capability level, the Capability Delivery Platform is going to be either COTS, GFE, or Open Source. We simply aren’t going to permit contractors to hold proprietary positions in those layers.

Are there any questions? Feel free to ask, we’ll answer all of your questions here and now. We are videotaping the Q&A and it will be available online tomorrow for those potential bidders who may not have been able to get here. So ask away, it’s all on the table.”

*actually, this is complete fiction since the F-100 was derived from an unsolicited proposal.

Comments

  1. Chris Gunderson - December 21, 2009 @ 4:48 pm

    Jim, great post. … a friend told me that the RFP for the first Airliner (TWA) was one paragraph long. That turned out pretty well…

    I went to an AOC WS industry days a couple months ago and had a similar experience to yours. I wish AOC WS focus on bureaucracy was atypical, but we both know it is not. E.g. I was at meeting with some engineers discussing a new standard associated with military communications. The question “should we bring NATO allies into the discussion?” came up. The OSD rep’s answer was essentially…”NO! That would complicate the paperwork badly.” I mentioned that if we didn’t bring the allies into the conversation up front, it would add years onto the implementation timeline at the back end. The answer was “what’s your point?”

    Good news is that there seems to be a growing center of mass of technical people in the trenches who are fed up with the madness. I’m hoping we’re close to a tipping point!

  2. Perry - December 21, 2009 @ 6:40 pm

    Acquisition is screwed up, and it got this way because… Well, I don’t really know enough to say how it got this way. I was going to say, “Because everything needs to be specified down to a gnat’s ass because if not the contractors will take the government for a ride.” (“Oh, you wanted tires on that truck… well, I delivered the spec and it didn’t say anything about tires. The truck has special hubs, and any tires can only be created using my proprietary specs, and are $10M a piece.”)

    However, I realize that is probably unfair since all I’ve been is a user (in the military) or a government type. I know that changing/unclear specs causes all sorts of problems also, as well as many other things.

    I don’t know what needs to be done, and I’m sure any process to reform would end up completely ugly, and probably produce something worse than the Congressional process over health care. But here’s a thought on how to do it to remove all the BS: Grab 20 people with the highest reputations for integrity, common sense, and intelligence (in that order) and lots of experience in the field, from both sides of the government/industry. Have them study and address all the issues and come up with the best way to buy everything, from ships to software, bullets to bedding, and whatever they say goes.

    Here’s the kicker: pay them $250M each for the two years of work. With the caveat that they, or anyone in their immediate family, cannot work in a company involved with DoD contracting for the remainder of their lives. Any ethical lapse will cause immediate loss of all payment, plus LONG sentences in a max security federal pen. The idea is to give them a very big carrot to stay good and a very big stick to prevent to avoid being bad, so they can make decisions simply for the good of the person at the tip of the spear and then the services. That’s $5B, and a lot of money, but do you think it would be worth $5B to just unBLEEP the current system?

  3. Bob Gourley - December 22, 2009 @ 11:15 am

    Jim I really appreciate you spelling things out like this. I have to admit I had hoped there was more “platform” thinking in their approach. There are a growing number of seniors who “get it” in this regard. I’ll forward this on to those I know who are with you on this.

    Cheers,
    Bob

  4. Garry Trexler - December 23, 2009 @ 1:47 pm

    Jim, wonderful job laying out the issue. I spent the last 12 years of my active duty career in and out of the AOC business…the majority of that time as a commander. The frustration was tremendous.
    Early on in the 1990s every commander wanted his own AOC and many built them. They truly were a hodgepodge of kluged together systems. Something had to be done, and recognizing the AOC as a weapon system and introducing configuration management was the right thing to do at the time. But, as you pointed out so well, the bureaucracy could not make changes, which were needed at the local level, fast enough. Innovation was stifled! Upgrades delivered were “late-to-need” and in most cases unusable once they were fielded. Early on, most fielding problems were caused because “no one” was in charge or held responsible for integration.
    As you pointed out, many of the problems were caused because the government acquisition personnel and the private sector engineers did not understand how commander’s employed air and space power. Additionally and more importantly, they did not take into account the skill level and backgrounds of the personnel operating the systems. What might be intuitive to a software engineer is not necessarily intuitive to an “augmentee” brought into the AOC for 90 days. Couple this with the fact that in some cases the “augmentee” may be from a foreign country.
    The Government is its own worst enemy; however, industry does not get a pass. Your discussion regarding not allowing contractors to hold “proprietary” positions in the layers is extremely important. Of course their innovations and intellectual property must be protected; but this can be done while still demanding COTS, GFE, and Open Source.
    Finally, “Flexibility is the key to Airpower” and that basic tenet is forgotten by those who write the FARs.

    It’s probably time to take a “platform” approach to supporting the warfighter’s requirements.

  5. Carmine Wiggins - March 4, 2010 @ 7:34 pm

    What is funny about this is all the MAJCOMs w/AOCs sends reps to all the requisite working groups that manage (vote too) what goes into or does not go into the AOC WS. I cannot tell you how many times these reps voted for something they had no clue about what they are putting into our AOCs. We need qualified blue-suiters in those voting seats and extremely tech savy folks in the engineering dept that know how useless things like SOA based architecture might be. As far as TBONE, that was a mistake on Tommy Crawford-he decided on his own to break the current system of record by taking all the money to fix it properly. So we get what we got. It’s time for total truths, TBMCS should never have been fielded, it was old by the time it was and too much money wasted.

Leave a Reply

Your email address will not be published / Required fields are marked *