February 2, 2007 by jimstogdill
A More Fun Way to Build C2 Systems
A large Command and Control system that will remain anonymous in this post recently went through a Program Management Review. From what I gather it didn’t go well and the on again off again release of “dot” versions will continue for the forseeable future.
I started in this industry mid way through 2003 and in that time two “dot” versions have failed to field, and one has been fielded after a period during which it looked like it would be held up.
A few reasons spring to mind:
– Bad technology choices made to obtain competitive advantage in the weird world of DoD acquisition rather than based on sound technical choices.
– “Big bang” development with all of the expected difficulty of downstream pre-release integration.
– The integration of vendors in a forced marriage (who in fact basically hate each other and whose behavior remains competitive even during delivery).
– Things move slowly, so innovation shorts to ground, further confounding the politics of the situation.
So now a small tiger team has been tasked to re-group and put together a re-set plan to figure out how to get something done and ready for fielding this year, and then some more for next year.
Nobody asked me what we would do if we were leading that team, but it would be a fun challenge, and if they did ask, this is how we would go about it:
First, assign an overall “product manager” to the effort, and feature area product managers to support him / her. Somebody has to start being the single belly button to poke for the vision on this thing. And having a vision is key. Most complex products aren’t created by blindly canvassing for requirements; great products that wow their users appear at the intersection of well understood user requirements and a coherent vision (with liberal sprinklings of good design on top).
Next… We’d lock the product managers and a cross functional team of actual customers, acquisition folks, and the contractors that will build it in a room (less than ten people) for two days with white boards and paper. Tell them they only have two days, and when they come out they should have a succinct statement of what success looks like. It might take the form of a press release, or it might take another form, but the intent is to express in writing what an outstanding outcome looks like and make it visible to the entire team.
The other big thing they would have to produce is a product backlog that has these columns and that is sorted by value / priority:
– Story or Feature, Description (one paragraph or less),
– value (1-5)
– SWAG (hours)
– Complexity (1-5)
– and for future use a “committed” column.
At this stage these backlog items might be high level. That’s ok. There would be no requirements document.
Each team would be cross functional with analysts, testers, developers, documenters, and, where appropriate, UI designers. The teams and their occasional pair of pizzas would move into newly renovated team spaces. Essentially little war rooms where they can post stuff, see each other, hold impromptu meetings, and basically interact in a way that makes tons of documentation less necessary as a way to convey information between the members of a team, because they are all right next to each other.
By the way, when I say renovated I mean really simple 20’ by 20’ open spaces with tables / desks that face toward the center of the room. I’d also give each team a small budget to fit it out with the things they’ll need to make it both useful and homey… comfy chairs, maybe a little fridge, some mobile white boards. That kind of thing.
The very first team I would staff would be a Continuous Integration and Build Master team to set up a bullet proof continuous build and testing environment. This environment would start off simple but will grow rapidly as integration, acceptance, and performance tests are added with each new feature or capability is added to the core application suite.
The goal of this team will always be something like “make sure that a developer can check in code at any time and know within a five minutes if they broke the build, within thirty minutes if they broke an integration test, and won’t have to wait for more than overnight to find out if they broke an acceptance test or busted a performance metric whether they are in this building or work for a participating vendor somewhere else.”
This team would also be responsible for managing community processes and tools because this system is bigger than a single contractor. It will be necessary to augment the core teams funded by the acquisition contract with a gated community of developers who work for related programs.
The community process will ensure that they get early and frequent looks at the source code of key integration components of the core C2 system. They will be encouraged to test their own systems in concert with those early builds and offer feedback and source fixes and enhancements. They will do this by accessing source where appropriate and a dev net well-stocked with data where that is appropriate.
An additional benefit of early-and-often community process participation would be the early and loud screeching sound that would result from bad technology choices. A little bit of dissertation-defense mentality goes a long way toward keeping the BS out of the technology selection process and prevents those decisions from being made uni-laterally for the wrong reasons.
The other teams would be built around related feature areas within the overall system and will work really closely with the product managers mentioned earlier. Like every day, to make sure that features and capabilities are being built to match the vision.
The product managers will work to hard develop relationships with lots of users and customers so that they can in turn pick up a phone at any time to get their developers talking directly to a customer when it is needed. They will do this through both formal and informal channels. Informal is often better. They will work hard to facilitate DIRECT relationships between teams and users so that developers, analysts, and designers will be able to pick up the phone and talk to a user when they have a question.
Here’s where it gets fun.
We would put a big sign visible from every team space with the release date and time to release posted in very large text.
We would teach the developers test first development and expect them to check in code at least once per day, preferably more. No big integrations at the end.
We would encourage pair programming to get benefits of better information sharing, improved code quality, and faster more consistent implementations.
We would figure out ways to get as many of our team members as possible out of the office and into the C2 nodes to actually watch users work in as realistic conditions as possible (e.g. attend support exercises and experiments, training courses, tag along on biz dev trips, etc.). We would do this as frequently as possible. We would set up brown baggers so they could convey lessons learned to their peers.
We would ask each team to commit to what line items in the backlog that they will deliver at the end of a month. “Deliver” would mean fully tested and documented code that could be deployed (barring Certification and Accreditation) within one month. Teams would be self managing and would take that commitment to be a very real, personal, and visible commitment.
Each team would hold a stand up meeting every day where they would quickly have each member tell what they did yesterday, what they plan to do today, and any issues or roadblocks they are running into.
Every team would post their progress against commitments in a very visible location in their team spaces every day. These “burn down charts” would also be visible on the program portal to all stakeholders.
If more than one company was involved we would still have them do two-pizza teams and continuous integration into the common repository (to eliminate the downstream integration nightmare).
If possible I would move everyone under one roof. Where that wasn’t feasible I would have each two-pizza team assign a representative to an uber-team that would meet every day to discuss what each team did yesterday, what they are doing that day, and any issues or roadblocks they were running into.
To keep everyone playing nice I would make sure that every company, and every person working for them, had incentives that were based on the delivery of capability to the end user. Performance based payments down to individual bonuses would be paid when a high quality product was delivered to users in the field. Competition is appropriate for the vendor selection phase, but collaboration is critical during delivery.
When we hired people, we would hire for passion as much as for technical skill. We would do lots of little things to nurture a passion for quality delivery of great product to the warfighter across the program.
We would set up a blog roller for the entire development community (inside the program and including the gated community). We would encourage teams and individuals to establish blogs to add to the “ambient communication” happening across the program.
At the end of each month each team would demonstrate the actually working, tested, and documented software to any interested stakeholder from the warfighting, development, and acquisition communities. Related programs participating in the community processes would also be invited. If we were doing well, everyone would know it every month. If we were behind or delivering crap, they’d only have to wait a month to know that too.
After the demonstration each team, with their product manager, would meet for half a day to commit to the next month’s delivery and plan the work. The product backlog would be re-evaluated continuously by the product managers on the basis of current priorities or new “business” demands. The product backlog would become more refined as the project progressed and requirements became better understood.
Quarterly we would go through the process of packaging up a “release” for delivery even if none was expected so that we could find issues then rather then when the real release was expected. We would roll that release into a real world integration environment to validate that our continuous integration testing environment was comprehensive enough and we would subject it to real user interaction and testing to get user feedback. (Even better, we would convert this over time to a SaaS model so that we could release much more continuously).
We would make sure that the team had highly competent user experience, information design, and interaction design capabilities to ensure that the product delivered wasn’t just capable, but was usable. We would have a program “anthropologist” to validate that requirements were being met by real users doing real tasks in real world conditions, not just that some feature was “available.”
Well, I suppose that covers the high level and I should get back to some other things.
The main thing to try to achieve though would be a highly focused collection of small manageable and passionate teams, who would be continuously integrating and testing their efforts, demonstrating those increments to real users and stake holders frequently, and teams that were absolutely committed to a quality product delivered on time to users who need it. I think it would be fun.
My Dad always said “If you are going to do it, do it like it matters.”
Correction: When I first posted this I was under the mis-understanding that no new capability had been released in over three years. That statement was in error – thank you to the individual who pointed that out to me. A previous version, that for a while looked like it would be held back to just wait for the follow on, was in fact fielded in 2005/2006. I have corrected the first two paragraphs.