Grappling Titans

It’s Friday afternoon and I found a little bit of time to catch up on my reading, a bunch of which focused on big DoD programs that are going through big pains (Including an article in the Wall Street Journal I can’t link to called “Is the U.S. Government outourcing its brain?”). And this got me thinking…

What happens when your customer behaves as though it thinks that you, as an industry, are “contractor scum” and when you, attempting to serve that customer, feel like you are dealing with the line at the Post Office the week before Christmas writ large?

What happens when that disgruntled customer makes more and more rules to ensure that the providers that it really doesn’t trust can’t take advantage of it? And when the suppliers of systems and services to that customer, ever more constrained in their ability to be innovative and efficient, by the very customer they are attempting to serve, are forced to more and more complicated efforts to deliver profits to their shareholders sufficient to keep the capital from finding greener pastures? And in doing so struggle to maintain the cultures that made them great in the first place?

While Sarbanes-Oxley, by raising the cost of doing business for public companies, is having the unintended effect of forcing organizations to merge for scale where there is no fundamental business reason to do so. The beuracracy and complexity of doing business with the federal government continues to encourage even greater unnecessary scale and makes SarbOx look like a school yard game.

So… what does it look like when a giant, beuractratic, policy-by-the-pound issuing customer and the five remaining monolithic, defensive, culturally misanthropic providers do business? 🙂


• • •

Open Technology II –

I recently finished a post on open technology in the DoD by posing the question: “could a significant infrastructure program such as SOSCOE be developed with one of the various open source software development models found in the commercial world?”

The models I considered might be loosely labeled the “IBM Apache” model, the “JBOSS” model, and the “grass roots” model.

Despite the success of Linux, which was arguably a grass roots model for significant periods of its early development, I don’t really see that model being applicable here. Perhaps I’m being unnecessarily hesitant, but I simply don’t see FCS application developers coordinating effectively at this stage to develop something as complex as SOSCOE with no central program. Even Linux essentially had a “core program” during its early years.

The “IBM Apache” model is predicated on the assumption that some core or useful seed already exists prior to the involvement of the paying entity. IBM didn’t get involved in Apache until it was already a viable web server with significant market share. And, well, If IBM had started the Apache project then it would really just be the JBOSS model again.

So… Let’s focus our attention on the JBOSS, or, more appropriately, the “funded open product development” model. Lets just imagine that the SOSCOE program were managed as though it were SOSCOE Inc. with goals similar to JBOSS’ but with some minor twists for suitability with the DoD environment. It would probably work something like this:

The DoD recognizes that SOSCOE, as a widely distributed infrastructure intended to support an entire eco-system of capabilities and applications, shares many of the attributes of a commercial open source “stack” and there will be benefits to treating it like one. With this recognition, the DoD decides to run the SOSCOE program as a funded gated-community open source development program. So…

– It awards a contract to an agile highly competent product and service provider to provide core SOSCOE “product” development activities including design, development, testing, and etc. This contract would be very similar to the contract currently held by Boeing and SAIC today but with some important differences…
– The contractor, rather than background as “integrator”, might have significant competencies in the core underlying technologies of SOSCOE (e.g. real time OS, networking, etc.) and would have experience with product development methodologies.
– The contractor team would be responsible for implementing community processes and tools to encourage and facilitate the participation of external contributors. Ideally they would have previous experience with open product development.
– As part of that community process they would, at any point during the development cycle, readily release source to any developer qualified to be inside the SOSCOE gated community (more on the scope of that community in a minute).
– Source released as “GLOSS” (Government Licensed Open Source Software) would be subject to a “GGL” (General Government License) license.*
– They would actively and aggressively work with the community of FCS application developers, the Integrated Computer System (ICS) designers, and other interested and appropriately credential-ed parties to incorporate bug fixes, code submissions, feature requests, and even design feedback into the product development cycle. Over time members of these communities would even earn commit rights within the community.
– Where requirements for things like POSIX compliance or other API’s exist within the program, the primary contractor would build a comprehensive set of test harnesses to facilitate test automation both within the communities configuration management repository and allow these tests to be distributed to members of the community for local testing prior to code submission.

I already discussed in the previous post why this would be great (e.g. more bullet proof software, engaged developer community, more competitive level playing field etc.); but what things would stand in the way of this today? What issues must be surmounted by changes in culture and policy?

Get it right and the DoD gets to do it’s reusable infrastructure projects with all of the benefits of the JBOSS model, do it wrong and it will end up open but irrelevant (e.g. Apple’s Darwin project; sure it’s open but it doesn’t seem to have generated a very vibrant community). In the remainder of this post I’ll highlight what I see as current-day issues with moving forward with a GLOSS / GGL approach. I’ll address solutions and policy recommendations in future posts.

First issue to address is barriers to participation.

“Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects” by Lakhini and Wolf provides important insights into why developers participate in OSS projects and how and if their efforts are funded.

It turns out that the primary motivation is intrinsic, hackers hack for the sense of flow they get when their contributions perfectly match their capabilities; and the sense of creativity they feel when in that flow. I believe that this reason for contributing can be readily harnessed in DoD GLOSS projects (and would greatly enhance the opportunities for rewarding roles in DoD industry); however, with the caveat that OSS contributors are generally self-selecting and this fact has wide ranging consequences. It is unlikely, given this primary motivation, that the DoD or its contractors can direct OSS participation artificially and achieve the same results. This has implications on how trans-contract GLOSS participation must be funded to be effective.

It also turns out, according to the paper referenced above, that a significant percentage of OSS contributors are funded by their companies (although frequently without the company’s knowledge or explicit consent). OSS contributors want to “be in the flow” but often not at the expense of spending weekends with their families. The implication here is that, particularly in the DoD, with its ubiquitous charge codes and DCAA audits, the application eco-system developers will not contribute to GLOSS projects unless their time is explicitly funded under a current contract except for those rare cases where they will do it on their own time.

So,… how to create an opportunity for GLOSS developers to self-select for participation while ensuring that they can also be funded; all while simultaneously ensuring that we don’t simply create a system of unspecified T+M work ripe for abuse? A question for another day, but It isn’t as simple as apportioning core-program requirements to the application developer eco-system and then spreading those dollars uniformly across the application contracts (and frankly, that wouldn’t be simple even if it would solve the problem).

The next big issue is intellectual property and its ownership. In an environment where contractors work hard to leverage IP ownership as a method for locking in their participation in future contract renewals and related programs, the idea of GLOSS will be anathema. The contractor working the core product development would be developing significantly less lock-in / barrier to exit when compared to traditional contracts. A GPL-like GGL license would be useful to the government in the sense that it would prevent forking, eliminate the need for IP attribution within the source, and ensure that every participant could use the entire source file for subsequent innovation and wide re-use, but it would be hated by the firm in prime contractor or LSI role. Naturally, as the biggest contributor to the source, they wouldn’t view the other contributor’s asymmetric contributions as worthy of equivalent rights. The government of course would have to resolve this through a combination of contractual requirements related to source sharing declared at acquisition time and allowances for development-time integration of proprietary run-time components that protect proprietary IP.

This next issue is in some ways more fundamental.

DoD contracts operate under the often erroneous assumption that a software project can be perfectly understood in all of its myriad details at contract award time, and then delivered on with certainty – the right amount of earned value accruing to the prime and all its associated subs in a well-choregraphed dance. Even if that were regularly true, how would earned value (or requirements apportionment for that matter) work when an indeterminate amount of the effort involved in infrastructure development is to be done by a distributed group of self-selecting contributors? (And just to be clear, the earned value usually does accrue as planned, it’s the software that doesn’t always follow the plan).

I doubt that JBOSS relies on the contributions of external contributors to meet their committed feature launch dates (I may be wrong on this, but I doubt they HAVE to rely on it). However, they at least have to account for the impact of addressing third-party requests, bug fixes, code contributions, and other interactions; not all of which can be entirely predicted in advance.

Let’s be honest, software development is rarely 100% predictable (not if the problem it solves is interesting anyway) but the widely shared delusion in this industry is that it is. The problem with variable-degrees of community OSS contribution is that it makes it impossible to pretend that schedule and effort will be 100% predictable. Linux is really cool, but who in 2000 would have known with certainty what features would be available when?

What prime contractor / LSI would sign up for a contract today where a fuzzy, indeterminate, amount of work would be required (or permitted) by unfunded (by the prime contract) self-selecting contributors in order to deliver on the requirements? Or, even if the prime was adequately funded to deliver the entire program, and third party community contribution was unnecessary to deliver on the core requirements but might still add uncertainty into the schedule because of other unpredictability’s, would they be willing to absorb the responsibility of coordinating those unknown third parties in advance? There is both real financial risk and tremendous cultural adaptation required for this to happen.

Finally, there is the question of community scope.

The people in government responsible for important infrastructure programs like SOSCOE are not going to be inclined to put them out on The idea that they would spend $B’s of U.S taxpayer sweat on infrastructure and then freely share it with our global competitors just wouldn’t sit well. However, the benefits of OSS accrue more readily when the pool of potential contributors is bigger rather than smaller. So, how to define the GLOSS gated community pool?

The “traditional” way, refined a bit for GLOSS, would be to open the source repository to those companies specifically under contract to build SOSCOE. My view is that this just isn’t big enough. Not only are all of these people already working on it, but it leaves out all of the opportunity for a slight twist on NetCentric principles, the unanticipated contributor. At a minimum the community has to include the community of developers who will actually be using the infrastructure in question for their applications. However, for systems where the code base is UNCLASS I believe one can make a very strong argument that the source repository should be open to any US citizen who is working on any DoD software contract.

As I said earlier, I will come back to each of these four issue areas in later posts to dissect further and attempt to suggest policies that will defuse or mitigate them. However, in the meantime let’s conduct a little thought experiment right here at our computers.

Open a browser to and poke around. However, as you are looking at the community tools and methods pretend you are at and everywhere you see the word “jboss” replace it with “soscoe.” Not only would the entire source and design documentation be at your finger tips, but there would be discussion between the core developers and the developers of consuming applications, there would be a transparent bug and issue list, a continuous integration test environment, recognition of “contributor of the month” and all of the other indicators of a vibrant broad community. Now imagine nearly a 1000 developers across the DoD participating to varying degrees in this community; the 400 or so current SOSCOE developers and 600 or so third-party community members who today are boxed out from even seeing a compiled instance of SOSCOE. Imagine 50 or more them actively contributing features, code, bug fixes, and design suggestions. Pretty cool huh?

* Note: If you have never heard of GLOSS or GGL you aren’t alone. I just made them up for this post. GLOSS is my stand-in term for a F/LOSS project executed within a government-sponsored gated community. Where an OSS project is sponsored by the government to satisfy the specific infrastructure requirements of the government customer I’m assuming that the government won’t be keen to share those developments with foreign, potentially hostile, governments. SOSCOE, even though it is 85% COTS/OSS today, probably still meets this definition if only to assuage the fears that completely opening it would create. GGL is a riff on the GPL. The implication is that the government could do itself a huge favor by creating a small portfolio of two or three licenses modeled on the knowledge gained from licenses such as GPL and Apache but with specific language to deal with the gated community issue noted above as well as other specific considerations unique to this domain and then widely using them (as opposed to creating licenses unique to each major program). To alleviate the coming “many to many” problem of contributors to projects, one or more of these standard licenses could be widely written into virtually every DoD contract as a consistent “master services agreement” term.

• • •

A Big Bet

I had dinner with a colleague from a previous company this week whose firm is making a massive bet on a new software platform. The company, which is in the financial services arena, has been spending 10% of gross revenues on R+D for three years and expects to continue doing so for three more in a huge effort to modernize and globalize their core offering. When it is all said and done the development of this new platform will probably cost half a billion dollars.

We read every day about how Google, Amazon, and Yahoo do software development and we try to take lessons from them back into the DoD. Each of these companies offers important lessons to the way we work, but the analogies are not always directly applicable because of the size of the projects involved. Often these companies are building a large number of relatively small projects that may or may not have any relation to each other. The project my colleague is working on is different in that it is one massive platform development and the company’s future is literally dependent on its success.

I think it is really interesting to look at how this company most people have never heard of is delivering such a massive project.

Of the approximately 650 developers and technologists working on the project, at least 400 of them are in India, 100 work for the company that is commissioning the system here in the states, and the balance work for as many as eleven contributing technology product firms.

Considering that those 400 developers in India are probably billing out at less than 1/3 of the fully loaded rate of a DoD contractor, this project would probably cost more like $1.2B or more if it were being developed at DoD contractor rates (assuming the same efficiency was maintained). This is a big software project even by DoD standards.

From the start of the project until the final instance of the legacy system it is replacing is turned off will probably take a decade or even as much as fifteen years. However, it is important to note that three years into the project one customer of some of the early capabilities is already being implemented. The company views this as important to prove its technology and to begin generating development-offsetting revenue early.

How else does this effort differ from an equivalent DoD project?

For starters the business owners who are sponsoring the project (and whose customers will be consuming it) are in the same building, and on the same team, with the technologists who are developing it. From the beginning the company signed up customers committed to early adoption whose representatives also participate in the development process in very direct ways).

Because of the number of developers involved, and the breadth of organizations that they represent, the overall program manager has set up a highly choreographed delivery cycle that has a fully tested and deliverable release coming out of the development every four to six months. This approach is being taken to ensure that all of the parts are integrated and working and to keep the risk down.

What is notable is that, though the company outsourced the bulk of the development and purchased a number of best-in-breed components, all of the project management and key product management roles as well as sizable architectural and development roles are being maintained in house.

These in house managers are, as expected, incented to deliver the platform on time. However, because of the impact of all this spending on the company’s EBIDTA and the fact that much of their incentive compensation is tied to company performance (bonuses and options) they are equally incented to keep costs in line. This is in contrast with a fully outsourced model where contractors can be subtly or not-so-subtly financially incented to grow scope to enhance the revenue opportunity.

It is probably also worth noting that as long as the program continues to track toward success, the leaders of the program (both business and technical) are very well compensated and the company is able to attract talented players in key positions of leadership.

Additionally, that team of 100 in house developers are presumably going to have to live with their creation after it is done. Subpar quality or incomplete functional scope during the development will not be opportunities for future business, they will be failures that these folks will have to live with long after the offshore contractors have moved on to their next gig.

The key insight though is probably that all of the program leadership, business and technical, is in the same building. This is not a situation where leadership is coming from multiple requirements and program management organizations and their widely dispersed SETA contractors. All of these equivalent functions for this very large program are in one building.

It is well documented that large programs in the commercial space struggle to get finished on time and on budget and this one will face all of the same risks. But when I look at the way the DoD does similar-sized software projects the structural choices that are forced by the acquisition and government hiring practices in my view seem to push the risk past the tipping point in way too many cases. Sometimes a few key choices can make a tremendous difference.

• • •

Open Technology

I can’t believe it has been a month since my last post. Blogging is a bit like taking vitamins every morning; if you forget for a day suddenly you realize you’ve forgotten for a month.

I attended the AFEI Open Technology conference in DC last week and am still absorbing what I heard.

Chuck Riechers, Principal Deputy Assistant Secretary of the Air Force for Acquisition, in his keynote described a three step plan for the DoD’s adoption of open source / open technology. Start with the acceptance of open standards, interfaces, and data (crawl) before moving toward the adoption of Open Source Software concepts and methods for internal DoD developments (walk), and finally leverage shared source repositories (run).

Unfortunately, the rest of the conference seemed stuck in the rut of “Really, you can use open source software like COTS.” It is difficult to engage in a discussion of the more complex use of OSS concepts and approaches to build DoD infrastructure when the culture is still absorbing the more basic idea of using COTS OSS (e.g. “really, you are allowed to crawl”).

The standard concerns of “who will support it?”, “Is it secure?”, “How can I control feature direction?”, and “How can we get it certified and accredited if no vendor is advocating for it?” keep coming up, and will keep coming up for a long time. These and similar concerns seem to be rehashed ad nauseum at each open technology discussion that I have attended in the last year leaving little time for discussion of the more interesting “walk” and “run” stages.

I guess I shouldn’t be surprised. It has been less than a decade since the idea of using any COTS systems really started to get some traction in the DoD. For instance it was only in the early 90’s that systems like the predecessor of GCCS were running on purpose-built Honeywell hardware. The Army’s AFATDS fire control system shipped for the first time only a few years ago with a 1980’s era TP4-based proprietary networking system. In the Navy it is still a very early trend to use open architecture shipboard. Historically everything from silicon to disk drives were designed from scratch for each new capability that came down the pike.

There are some positive trends.

Brig Gen Justice from C3T at Fort Monmouth discussed the fact that recent versions of FBCB2 are now based on Linux. Linux seems to be turning up everywhere including in the foundation of the Army’s System of Systems Common Operating Environment (SOSCOE).

Despite these wins I mostly just want to say “so what.”

In the commercial world there is a well defined trend toward cost sharing as the infrastructure stack moves to open source. From core IP network elements to the OS and now onto higher-order middleware infrastructure. The trend is for widely used shared infrastructure elements to go to open source.

Where is this trend in the DoD? It is one thing to consume from the bottom of the commercial OSS infrastructure stack, but the DoD has some mission-specific infrastructure needs. When will that DoD-specific infrastructure stack start being addressed in a more open way?

Take something like SOSCOE as an example.

SOSCOE is supposed to be the Army’s foundation for an entire eco-system of applications and capabilties across the wide-ranging FCS program (and maybe, ultimately, wider). Essentially it is a traditional operating system with intermittent P2P networking and service abstraction built in (maybe an over-simplification, but that’s basically it).

Imagine if SOSCOE were an open source initiative being developed incrementally to satisfy near term requirements with source-level involvement from the FCS application development community (and those capabilities were going to be rapidly fielded in Strykers and elsewhere). In other words, imagine if it were being developed the way Linux was. Short incremental versions that actually go into production with a high degree of involvement from the consuming / contributing community throughout the process of development.

The benefits of such an approach would include:
– a better more bullet proof product whose usefulness was proved at every step of development
– an opportunity to gain the value from early increments, and prove those increments while the software moves up the capability curve
– an engaged development community whose insights and inputs from actually using the product would feedback in the form of useful code and design insights
– a much more even playing field for competition within the developer community / SOSCOE ecosystem resulting in lower cost and better capability for the Army.

Unfortunately it doesn’t work this way. The Army won’t even “own” SOSCOE until it is “finished” and the DD250 (Material Inspection and Receipt Form) is signed years from now. In the meantime incremental “releases” are being delivered to the government, but their use and distribution is severely restricted.

Imagine if Linux had been in development for ten years, with increments being inspected and tested on a limited basis but were restricted from being used in a production environment, and then it was suddenly shipped to millions of users and placed into production in data centers all over the country.

The way the acquisition rules are being interpreted for software infrastructure places an unreal expectation that there is a moment in time where it is done (DD250 signing day); before which it can’t be used, and after which it is widely used with maximum capability.

There are a number of reasons why things like SOSCOE aren’t developed in the DoD the way they would (or could) be in the outside world. But maybe the most fundamental reason has nothing to do with technology or the acquistion rules per se. It fundamentally boils down to incentives.

In the commercial world markets are not bound. The pie can get bigger. In such an environment it makes perfect sense for IBM to support the development of Apache or for Sun to support OpenESB. By commoditizing the infrastructure in a cost-sharing way they can focus their efforts higher up the stack and build a model around support, integration, and higher-order dependent products.

In another example, it makes perfect sense for a variety of competitive Wall Street firms to collaborate in the development of the commodity messaging platform AMQ. They can spend cost-shared dollars to achieve a highly scalable commodity product.

To look briefly at one more model, the dynamics of today’s commercial markets also make the model(s) pursued by JBOSS and OpenSQL make sense as well. Essentially these companies conduct near-traditional product development but open the process to intentionally achieve commoditization of the product for wide distribution and then make money on ancillary services or related products (and also gain the benefits of an engaged participating user group).

Would any of these models make sense for the DoD in the context of infrastructure such as SOSCOE?

Could Boeing act like IBM or Sun and essentially manage an open source community (either gated within the DoD contracting community or wide open) to “layer” the needed additional capabilities on top of Linux?

Could they act like JBOSS and essentially do what they are doing now but open the repository and process to the community (again, gated or not depending on the sensitivity of the effort)?

Or, could Boeing be eliminated from the process altogether by simply having the FCS application development community coordinate their own efforts to develop some or all of SOSCOE as an OSS collaborative community?

I will address the advantages and barriers to each of these models as the basis for future posts.

• • •