Open Technology Conference Wrap Up – Where the Geeks At?

Yesterday I sat on the panel that I referred to here. I thought I’d follow up with a brief post about one topic of our panel conversation.

To start the panel we were asked “what’s bugging us?” This started an interesting conversation about some specific open source roadblocks in defense. In particular, Bdale Garbee made the point that open source projects rely heavily on personal reputation. Even when major corporations participate in open source community, it’s the reputation of the individual that determines whether and how contributions make it into the project repository. People get commit rights, not companies. This can be problematic in the defense space.

I added that many key contributors to open source projects have self-selected to participate. The ability to self select is important to the ability of a project to find people with high levels of commitment and expertise. Look at the list of contributors on the Apache web server project for example. While there are certainly participants that represent major corporations, I would estimate from looking at the list that at least a third self selected. And that third is important as it is often the source of key (and difficult to find) skills. In fact, even many of the company sponsored contributors self selected and were later hired because of their participation.

Unfortunately, for a variety of reasons, within the DoD it can be much more difficult to self-select to participate. In defense work every hour is accounted for and must match a specific project plan line item. Community participation often requires a contributor to assist with things that don’t have an exact corresponding work breakdown structure element from the program that is paying them. In defense work, if you don’t have a charge code, you don’t work. There’s simply less wiggle room for participation that doesn’t directly relate to the program that is funding you at that moment.

We also touched on a bunch of other issues that impact the ability to participate in or contribute to open source projects. Things like export controls, copyright, culture, etc.

These specific issues that impact defense contribution and participation have broad implications if defense is to be able to effectively leverage the work going on in open source communities. One of the things that makes open source community tick is the right to fork. Knowing that you can fork the source if the project direction deviates from your own direction is important to alleviating risk. The antidote to forking is community participation and the development of trust. The more you participate, and the more you develop trust, the more you or your organization can influence the direction of a project or at least make sure that your specific needs can be met. With all of the rules that currently make meaningful participation difficult, it is very difficult for defense contractors to participate in the upstream software value chain. The result is perpetual forking.

It will work like this. A defense contractor does a trade space analysis and decides that they can save a lot of money for the government by using a particular open source project, so they include it in their bid. They win and they build the system using the open source component, however, they realize that they have to modify it in a few critical ways to satisfy some specific requirements. They can’t participate in the community so their changes never get offered back, and never make it into the trunk. A few years later, under a follow up maintenance and sustainment contract, they do an upgrade of the system and, because their changes never made it into the core project, they have to repeat the work again on the newest version of the open source project.

In the not too distant future there will probably be whole classes of software infrastructure that are effectively only available as open source. It simply won’t be economic for a proprietary software firm to compete in areas that have been completely commoditized. Therefore, it’s imperative that the Department figures out how to resolve the issues that are preventing their own people or their contractors from participating meaningfully in the communities that they will be forced to rely on.

That’s probably enough on that. There was one other thing I wanted to touch on in this post. This was the fourth year of this conference and, maybe I’m just an impatient person, but I’m getting really bored of the same old remedial conversations with a bunch of suits (full disclosure, I was in a suit too). Or as John Scott put it to me during a break, “Where the geeks at??” Too much of the conversation is still about whether or not Linux qualifies as CoTS in the FARS and that sort of thing. Where are the breakout groups on open geo tools? Where’s the presentation from the guys using XMPP as a cheap messaging stack in some major program? Where are the non-DoD geeks who are attending because they are participating in an open source community that was started in defense but is now being widely used to solve all kinds of other problems? Where are people trying to build an open service bus that will deal with intermittent service end points that you find on a battlefield? Where are the SOSCOE developers talking about how they used JXTA’s service advertising mechanisms? Etc…

It’s time to move from the basics into the advance course kinds of stuff; the stuff you talk about when you are actually doing it. It’s time for DoD policy makers and decision makers in key programs to really start to push; push for expertise, program outcomes, and key policy initiatives that will alleviate the kinds of road blocks we discussed (again) in our panel. In short, it’s time to stop talking about open source in defense and start using it at such a meaningful scale that next year the room won’t be full of suits, but will be full of geeks and practitioners.

• • •

Open Source in Defense: Consuming it is Nice, but Building it is Better

I’ll be participating in a panel discussion next week at the 4th Annual DoD Open Technology Conference in DC. The Panel is about open source software in defense and will be moderated by John Scott, an author of Sue Payton’s Open Technology Development Roadmap (pdf). Dan Risacher, who recently discussed the DoD CIO’s upcoming policy memo with GCN, will also be on the panel. We’ll be talking about what makes open source valuable to the department – consuming it, contributing to it, and even building it outright. We’ll also be talking about the policy, legal, and accidental process roadblocks that make it more difficult today than it should be.

Yesterday, while I was doing some preparation, I ran across this sources sought on Fed Biz Ops. It got me thinking about the down to Earth practical stuff that is necessary to make a difference in encouraging open source in defense. I am going to come back to the details of this sources sought in a moment.

The DoD has broken the seal when it comes to consuming open source, at least in packaged form. I’m not certain where I got this factoid, but I think the US Army is now Redhat’s single biggest customer. But like I’ve said before, consuming open source is no big deal and really isn’t occasion for a big celebration. Where the DoD stands to gain much more value is in producing open source software.

Every industry has at least some domain-specific software needs. The stuff that makes up their industry-specific “stack” and that isn’t readily provided in the cross-industry products from the major software vendors. For example, the financial services industry depends on things like high speed messaging buses and high availability transaction monitors. Web firms use things like Perlbal, Hadoop, and of course Apache that help them build a massively horizontally scaled web presence. Telecom has specifications like H.323 and now SLEE and SIP and products built on them.

In the old days, if the industry was big enough, domain-specific software vendors would spring up to provide them with the infrastructure that they needed (e.g. Tibco’s Rendezvous). If they were REALLY big, a large software vendor might even offer a domain-specific product, or at least a version of their product.

These days though there is an alternative, Open Source Quasi- Joint Ventures. Well, nobody really calls them that, but that’s how I think of them. They are like accidental joint ventures that do resource sharing the way a traditional joint venture would, but they rely on open source licensing to make the risk of participation low. Plus, they avoid most of the legal and ownership wrangling that happens in a real joint venture.

A great example of this approach is the Advanced Message Queuing Protocol (AMQP) (and associated AMQ implementations such as OpenAMQ). It was initiated by JPMorgan but has grown to include many large banks as participants. The banks don’t give up any competitive advantage by participating because messaging is about passing information to trading partners, but they save money by more efficiently providing for their own infrastructure.

Things like OpenID and Hadoop also fit into this mold. Companies like Yahoo and SixApart are taking active roles in funding and guiding the development of their industry-specific technology. Again, it’s far enough down in their stack that they aren’t giving up a competitive position; but they are saving money by sharing their development resources.

I don’t mean to say that these aren’t normal open source projects in every way. I’m simply making a distinction about how and why they are funded in terms of the specific needs of the industry that is funding them. By joining together to build components for an industry-specific stack and then intentionally commoditizing it within that industry, these projects seem to be filling in where JV’s or domain-specific software companies might have focused before. This open source approach is better than a traditional JV though because new participants can join up at any time and they avoid many of the up front issues of starting a JV.

Back to the DoD. Defense has saved a great deal over the last decade recognizing that it can leverage COTS hardware and software. However, it still has many unique needs for its information technology stack – the DoD operates in a different operational environment and has many specialized requirements. So, while the DoD today is beginning to consume “package” commodity open source projects such as Linux, there is still a great opportunity to steal a page from JPMorgan’s playbook and build defense-specific infrastructure as open source. The DoD builds defense-specific stack components all the time, but they rarely do it in a way that makes it easy for other programs to adopt them (or even know about them). An open source approach would better spread the funding and would also ensure that once the money is spent the pieces could be widely used and adopted.

This brings me back to the WebTAS sources sought.

WebTAS started life years ago to simplify the process of conducting data analysis that spanned database instances and DBMS’s. Over time it evolved to be something like generalized middleware and application framework for data analytics applications (and has been used in even more general applications since then). The sources sought I linked to describes basically a “business as usual” approach to continuing the program as it plans to support continued R+D of the core framework as well as for at least some of the analytics work that will be done with it (which is why top secret clearances are required).

It’s probably worth asking the question whether there is even still a need for a government funded program to build a database connectivity and analytics suite (especially for a program that is expected to cost as much as $300M – that’s almost 25% of the $1.4B value of the Linux Operating System!). A lot of time has passed since the program was started and there are many more commercial and open source technologies available in that space today than there were when WebTAS started. However, for the purposes of this discussion I’m going to assume that WebTAS is continuing to provide unique capabilities to meet DoD-specific requirements. However, with that assumption in place, I’m going to argue that WebTAS should be developed in the fashion of an Open Source Quasi-JV.

Because WebTAS was developed under government contract, it can theoretically be used in any government contract. The government could furnish it to a contractor as Government Furnished Equipment (GFE) for any program. However, in practice this rarely happens. In defense IT, infrastructure software tends to be used only on contracts delivered by the contractor that built the infrastructure. As an example of this tendency, all three of the projects mentioned by the sources sought, SWIC, PANACIA, and MAAP are built primarily by the same contractor that currently delivers WebTAS.

If the government is interested in getting more value out of their investments in infrastructure like WebTAS (and would like to quench the proprietary lock in business model that they are stuck with today) they need to take concrete steps. As the sources sought indicates, the contract is expected to have a five year term. Once that contract is issued, if an open source approach isn’t built in, there won’t be another opportunity to change the approach for five years.

So, it’s great that Ms. Payton’s office wrote the OTD Roadmap and that the DoD CIO is about to issue a clarifying open source policy. However, if I were Ms. Payton, I would take another step and have my staff directly engaging with program managers of programs like WebTAS to ensure they let contracts that would directly support OTD in general and open source in particular.

For example, why not define in the contract’s CDRL’s (basically the stuff that is delivered) that the vendor must establish and govern an open community and the associated code repository? Why not include award fee metrics that incent open community – stuff like the number of other government programs or contractors that are participating in the community and using the software for their projects in order to incent open community development and marketing? While using IDIQ style contracts, why not make the awards to a range of potential contributors so that the contract is positioned to support an eco-system of contributors and ensure that each of them understands the Intellectual Property approach that will be used? Why not split the contract to develop the infrastructure from the use of it for analytics work so that a wider group of contractors (that don’t have to take on the cost of TS clearances) can participate?

The goal should be to establish rules and incentives in the contract that encourage the development of widely available open software with an effective community.

Savvy contractors will realize on their own that taking the initiative to open source infrastructure like WebTAS themselves is good for business. Assuming the code is valuable, aggressively commoditizing it will contribute to wider adoption and more opportunity – after all, these are all services businesses. However, it’s early and we haven’t reached that tipping point yet. So, if the government wants to achieve real strides with open initiatives they need to do more than provide OTD policy guidance. They need to aggressively work programs like WebTAS to establish contractual terms and incentives that will push their contractors past the tipping point before another round of long term contracts freezes progress for half a decade.

• • •