March 27, 2007 by jimstogdill
Open Technology II – www.soscoe.org
I recently finished a post on open technology in the DoD by posing the question: “could a significant infrastructure program such as SOSCOE be developed with one of the various open source software development models found in the commercial world?”
The models I considered might be loosely labeled the “IBM Apache” model, the “JBOSS” model, and the “grass roots” model.
Despite the success of Linux, which was arguably a grass roots model for significant periods of its early development, I don’t really see that model being applicable here. Perhaps I’m being unnecessarily hesitant, but I simply don’t see FCS application developers coordinating effectively at this stage to develop something as complex as SOSCOE with no central program. Even Linux essentially had a “core program” during its early years.
The “IBM Apache” model is predicated on the assumption that some core or useful seed already exists prior to the involvement of the paying entity. IBM didn’t get involved in Apache until it was already a viable web server with significant market share. And, well, If IBM had started the Apache project then it would really just be the JBOSS model again.
So… Let’s focus our attention on the JBOSS, or, more appropriately, the “funded open product development” model. Lets just imagine that the SOSCOE program were managed as though it were SOSCOE Inc. with goals similar to JBOSS’ but with some minor twists for suitability with the DoD environment. It would probably work something like this:
The DoD recognizes that SOSCOE, as a widely distributed infrastructure intended to support an entire eco-system of capabilities and applications, shares many of the attributes of a commercial open source “stack” and there will be benefits to treating it like one. With this recognition, the DoD decides to run the SOSCOE program as a funded gated-community open source development program. So…
– It awards a contract to an agile highly competent product and service provider to provide core SOSCOE “product” development activities including design, development, testing, and etc. This contract would be very similar to the contract currently held by Boeing and SAIC today but with some important differences…
– The contractor, rather than background as “integrator”, might have significant competencies in the core underlying technologies of SOSCOE (e.g. real time OS, networking, etc.) and would have experience with product development methodologies.
– The contractor team would be responsible for implementing community processes and tools to encourage and facilitate the participation of external contributors. Ideally they would have previous experience with open product development.
– As part of that community process they would, at any point during the development cycle, readily release source to any developer qualified to be inside the SOSCOE gated community (more on the scope of that community in a minute).
– Source released as “GLOSS” (Government Licensed Open Source Software) would be subject to a “GGL” (General Government License) license.*
– They would actively and aggressively work with the community of FCS application developers, the Integrated Computer System (ICS) designers, and other interested and appropriately credential-ed parties to incorporate bug fixes, code submissions, feature requests, and even design feedback into the product development cycle. Over time members of these communities would even earn commit rights within the community.
– Where requirements for things like POSIX compliance or other API’s exist within the program, the primary contractor would build a comprehensive set of test harnesses to facilitate test automation both within the communities configuration management repository and allow these tests to be distributed to members of the community for local testing prior to code submission.
I already discussed in the previous post why this would be great (e.g. more bullet proof software, engaged developer community, more competitive level playing field etc.); but what things would stand in the way of this today? What issues must be surmounted by changes in culture and policy?
Get it right and the DoD gets to do it’s reusable infrastructure projects with all of the benefits of the JBOSS model, do it wrong and it will end up open but irrelevant (e.g. Apple’s Darwin project; sure it’s open but it doesn’t seem to have generated a very vibrant community). In the remainder of this post I’ll highlight what I see as current-day issues with moving forward with a GLOSS / GGL approach. I’ll address solutions and policy recommendations in future posts.
First issue to address is barriers to participation.
“Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects” by Lakhini and Wolf provides important insights into why developers participate in OSS projects and how and if their efforts are funded.
It turns out that the primary motivation is intrinsic, hackers hack for the sense of flow they get when their contributions perfectly match their capabilities; and the sense of creativity they feel when in that flow. I believe that this reason for contributing can be readily harnessed in DoD GLOSS projects (and would greatly enhance the opportunities for rewarding roles in DoD industry); however, with the caveat that OSS contributors are generally self-selecting and this fact has wide ranging consequences. It is unlikely, given this primary motivation, that the DoD or its contractors can direct OSS participation artificially and achieve the same results. This has implications on how trans-contract GLOSS participation must be funded to be effective.
It also turns out, according to the paper referenced above, that a significant percentage of OSS contributors are funded by their companies (although frequently without the company’s knowledge or explicit consent). OSS contributors want to “be in the flow” but often not at the expense of spending weekends with their families. The implication here is that, particularly in the DoD, with its ubiquitous charge codes and DCAA audits, the application eco-system developers will not contribute to GLOSS projects unless their time is explicitly funded under a current contract except for those rare cases where they will do it on their own time.
So,… how to create an opportunity for GLOSS developers to self-select for participation while ensuring that they can also be funded; all while simultaneously ensuring that we don’t simply create a system of unspecified T+M work ripe for abuse? A question for another day, but It isn’t as simple as apportioning core-program requirements to the application developer eco-system and then spreading those dollars uniformly across the application contracts (and frankly, that wouldn’t be simple even if it would solve the problem).
The next big issue is intellectual property and its ownership. In an environment where contractors work hard to leverage IP ownership as a method for locking in their participation in future contract renewals and related programs, the idea of GLOSS will be anathema. The contractor working the core product development would be developing significantly less lock-in / barrier to exit when compared to traditional contracts. A GPL-like GGL license would be useful to the government in the sense that it would prevent forking, eliminate the need for IP attribution within the source, and ensure that every participant could use the entire source file for subsequent innovation and wide re-use, but it would be hated by the firm in prime contractor or LSI role. Naturally, as the biggest contributor to the source, they wouldn’t view the other contributor’s asymmetric contributions as worthy of equivalent rights. The government of course would have to resolve this through a combination of contractual requirements related to source sharing declared at acquisition time and allowances for development-time integration of proprietary run-time components that protect proprietary IP.
This next issue is in some ways more fundamental.
DoD contracts operate under the often erroneous assumption that a software project can be perfectly understood in all of its myriad details at contract award time, and then delivered on with certainty – the right amount of earned value accruing to the prime and all its associated subs in a well-choregraphed dance. Even if that were regularly true, how would earned value (or requirements apportionment for that matter) work when an indeterminate amount of the effort involved in infrastructure development is to be done by a distributed group of self-selecting contributors? (And just to be clear, the earned value usually does accrue as planned, it’s the software that doesn’t always follow the plan).
I doubt that JBOSS relies on the contributions of external contributors to meet their committed feature launch dates (I may be wrong on this, but I doubt they HAVE to rely on it). However, they at least have to account for the impact of addressing third-party requests, bug fixes, code contributions, and other interactions; not all of which can be entirely predicted in advance.
Let’s be honest, software development is rarely 100% predictable (not if the problem it solves is interesting anyway) but the widely shared delusion in this industry is that it is. The problem with variable-degrees of community OSS contribution is that it makes it impossible to pretend that schedule and effort will be 100% predictable. Linux is really cool, but who in 2000 would have known with certainty what features would be available when?
What prime contractor / LSI would sign up for a contract today where a fuzzy, indeterminate, amount of work would be required (or permitted) by unfunded (by the prime contract) self-selecting contributors in order to deliver on the requirements? Or, even if the prime was adequately funded to deliver the entire program, and third party community contribution was unnecessary to deliver on the core requirements but might still add uncertainty into the schedule because of other unpredictability’s, would they be willing to absorb the responsibility of coordinating those unknown third parties in advance? There is both real financial risk and tremendous cultural adaptation required for this to happen.
Finally, there is the question of community scope.
The people in government responsible for important infrastructure programs like SOSCOE are not going to be inclined to put them out on Sourceforge.net. The idea that they would spend $B’s of U.S taxpayer sweat on infrastructure and then freely share it with our global competitors just wouldn’t sit well. However, the benefits of OSS accrue more readily when the pool of potential contributors is bigger rather than smaller. So, how to define the GLOSS gated community pool?
The “traditional” way, refined a bit for GLOSS, would be to open the source repository to those companies specifically under contract to build SOSCOE. My view is that this just isn’t big enough. Not only are all of these people already working on it, but it leaves out all of the opportunity for a slight twist on NetCentric principles, the unanticipated contributor. At a minimum the community has to include the community of developers who will actually be using the infrastructure in question for their applications. However, for systems where the code base is UNCLASS I believe one can make a very strong argument that the source repository should be open to any US citizen who is working on any DoD software contract.
As I said earlier, I will come back to each of these four issue areas in later posts to dissect further and attempt to suggest policies that will defuse or mitigate them. However, in the meantime let’s conduct a little thought experiment right here at our computers.
Open a browser to www.jboss.org and poke around. However, as you are looking at the community tools and methods pretend you are at www.soscoe.org.mil and everywhere you see the word “jboss” replace it with “soscoe.” Not only would the entire source and design documentation be at your finger tips, but there would be discussion between the core developers and the developers of consuming applications, there would be a transparent bug and issue list, a continuous integration test environment, recognition of “contributor of the month” and all of the other indicators of a vibrant broad community. Now imagine nearly a 1000 developers across the DoD participating to varying degrees in this community; the 400 or so current SOSCOE developers and 600 or so third-party community members who today are boxed out from even seeing a compiled instance of SOSCOE. Imagine 50 or more them actively contributing features, code, bug fixes, and design suggestions. Pretty cool huh?
* Note: If you have never heard of GLOSS or GGL you aren’t alone. I just made them up for this post. GLOSS is my stand-in term for a F/LOSS project executed within a government-sponsored gated community. Where an OSS project is sponsored by the government to satisfy the specific infrastructure requirements of the government customer I’m assuming that the government won’t be keen to share those developments with foreign, potentially hostile, governments. SOSCOE, even though it is 85% COTS/OSS today, probably still meets this definition if only to assuage the fears that completely opening it would create. GGL is a riff on the GPL. The implication is that the government could do itself a huge favor by creating a small portfolio of two or three licenses modeled on the knowledge gained from licenses such as GPL and Apache but with specific language to deal with the gated community issue noted above as well as other specific considerations unique to this domain and then widely using them (as opposed to creating licenses unique to each major program). To alleviate the coming “many to many” problem of contributors to projects, one or more of these standard licenses could be widely written into virtually every DoD contract as a consistent “master services agreement” term.