Here's Obama's plan to prevent future IT disasters like the Healthcare.gov rollout

The government has long been bad at managing information technology. It's a problem that was starkly highlighted by the disastrous 2013 rollout of Obamacare's Healthcare.gov website. Ever since then, senior Obama administration officials have been looking for ways to improve the government's IT capabilities — creating two new agencies for that purpose in 2014.

And now the White House is turning to Congress for additional help. Earlier this month, Rep. Steny Hoyer (D-MD), the House's number two Democrat and a White House ally, introduced legislation to provide $3.1 billion for modernization efforts.

The White House has touted the fund as a way to improve the government's information security, something that's definitely needed after someone — perhaps the Chinese government — stole millions of sensitive records from the Office of Personnel Management. But it's not just about security. In an interview with me last week, Hoyer argued that the fund would also encourage government agencies to make important reforms to how they manage IT projects.

Traditionally, the government has used a top-down approach to software development that left little room for experimentation and user feedback. Often, this meant massive projects that cost hundreds of millions of dollars, took years to complete, and were obsolete before they were ready to use.

But now, leaders in the White House and Congress are starting to recognize that the government needs to adapt more flexible methods — known in the private sector as agile software development — that are focused on iteration, decentralization, and rapid response to user feedback. These reforms, they think, could make the government's famously dysfunctional IT systems work a whole lot better.

The FBI's decade-long IT disaster

Photo by Win McNamee/Getty Images

To illustrate the problems with the government's traditional approach to development software, it's helpful to look at a classic example: the FBI's struggles to build an electronic case tracking system between 2000 and 2010. Using traditional contracting methods, the FBI struggled for nine years and spent hundreds of millions of dollars without producing a working system — until the agency made a major change that finally turned things around.

The story starts in 2000, when Congress gave the FBI $380 million for a three-year project to upgrade the agency's computer systems. The bureau wanted to replace its cumbersome paper-based case filing system with a modern, electronic system called the Virtual Case File (VCF).

This project gained added urgency after the terrorist attacks of September 11, 2001, since improved search capabilities might have helped agents identify terrorist threats. So Congress provided still more cash and encouraged the FBI to accelerate its development plan for VCF. The FBI hired a contractor to write the software for the new system and asked it to deliver the project within two years.

But that didn't happen. After the contractor delivered what was supposed to be the finished product in 2003, the FBI began testing it in real-world situations. The FBI discovered more than 400 ways that VCF it didn't live up to the agency's requirements. The project dragged on for another year before being abandoned in 2005.

A chastened FBI then started another new project called Sentinel to replace the failed VCF. They quickly ran into the same problems. A 2010 report from the Department of Justice's inspector general predicted that the project — which at that point was already behind schedule and over budget — could take another $350 million and six years to complete. If that prediction had come true, the FBI would still be working on the software today.

Agile development methods saved the FBI

But starting in 2010, the FBI changed its approach to developing the Sentinel system, and finally managed to escape from its IT quagmire.

In July 2012 — less than two years after the IG said there was still six years of work left to do — the FBI completed work on the Sentinel project and was able to shut down its old case management system.

What explains the dramatic turnaround? In October 2010, the FBI announced that it was overhauling the way it managed the Sentinel project and adopting a new approach called "agile development methodology" that had become popular in the private sector.

In a 2012 presentation, software consultant Brian Wernham explains how agile techniques helped the FBI finish the project more quickly. Agile programmers break their work into two-week "sprints," each of which is designed to deliver one concrete feature or enhancement to users. At the end of each sprint, programmers solicit feedback from users and incorporate it into subsequent sprints.

In the government's old model, developers did eventually seek feedback from users. But users were often asked to give feedback on prototypes or mockups rather than actual, working systems. And their feedback tended to get routed through senior project managers, who would then incorporate the feedback into changes in the official requirements document. The result: It could take months for the feedback to reach the developers who actually created the software, at which point the feedback might not even be applicable any more.

Also, in the agile model, rank-and-file developers have more discretion to change a product's design in response to user input. Users are able to communicate directly with the programmers who create the software, and they often see their suggestions implemented in a matter of weeks rather than months or years. That's good for user morale. And because developers are constantly receiving feedback from actual users, there's less danger that the project will be obsolete before it's even completed.

Can agile methodology work across the government?

Photo by Olivier Douliery-Pool/Getty Images

President Obama, of course, gained firsthand experience about the importance of software development methodology when Healthcare.gov failed spectacularly on October 1, 2013. The Healthcare.gov development process showed many of the same flaws as the early FBI development process. After the initial disaster, Obama recruited a team of elite programmers as part of a "technology surge" to save the site. The team brought with them many of the same agile programming methods that had worked so well for the FBI.

Once the "tech surge" was over, Obama hoped to institutionalize the lessons of the project and spread greater IT savvy across the federal government. The administration has created two organizations that are quietly trying to change the federal government's IT culture.

The US Digital Service operates as Obama's personal IT swat team. It's a team of talented programmers — many with experience in tech companies like Amazon or Facebook. They can be dispatched to work on projects that are priorities for the president, like improving the software of the Veterans Administration to avoid a repeat of the 2014 VA scandal.

The second agency, known as 18F, takes a more bottom-up approach. They operate as an internal consulting shop for other federal agencies. Agencies hire 18F to work on IT projects, but they have to agree to let 18F use agile programming methods. The hope is that as 18F develops a track record for delivering software on time and under budget, word will spread to other agencies about 18F in particular and agile programming methods in general.

Earlier this month, Rep. Steny Hoyer introduced legislation designed to modernize the federal government's IT infrastructure. The legislation would create a $3.1 billion IT Modernization Fund. Agencies that need to upgrade their infrastructure can apply for funds. Their requests will be vetted by an independent board of technical experts, who will look for projects that improve government services while saving government money.

Once projects are complete, agencies will pay back into the fund — hopefully using funds saved by the IT upgrade. If all goes well, that should provide an ongoing pool of funds for other agencies to use in the future. In this way, an initial $3.1 billion investment could enable improvements in IT systems indefinitely.

And because grants are provided on a competitive basis, agencies will have a strong incentive to adopt modern software development techniques in their proposals.

In an interview last week, Hoyer told me the fund was designed for projects that operate "in relatively discrete, agile forms so that those projects are done quickly." True to the agile methodology, Hoyer said, they should allow parts of the system to be tested by users before the entire project is finished.

And according to Hoyer's office, Rep. Darrell Issa, a prominent Republican from California, has agreed to co-sponsor the legislation.

Why agile programming is easier said than done

Agile programming methods have won a lot of converts, but they also have their share of skeptics. These critics point out that while agile principles — decentralization, rapid iteration, responsiveness to user feedback — sound good in theory, they don't always produce good results.

A prime example is Healthcare.gov itself. In late 2013, agile skeptic Michael Daconta argued that the Obamacare website — the original, failed version — was built using agile methods. "I have seen some of the developer documentation, and it clearly discusses sprints, user stories and incremental testing — all of which are hallmarks of an agile process."

"The reality is that the developers did use agile, and the project failed miserably," Daconta wrote. In Daconta's view, this showed that agile methods are not the silver bullet advocates claim.

Of course, the fact that Healthcare.gov project managers talked about agile methods doesn't necessarily mean they followed them. This is something the Atlantic's Robinson Meyer addresses in its 2015 feature article on the effort to save Healthcare.gov after the initial flop:

The government was eager to embrace agile methods, but it didn’t always understand them. The first time the team and the government tried to implement them together, government representatives drew up a plan for a three-month plan, complete with five carefully scheduled development sprints.

"And I’m like, how is that agile? That’s a three-month plan—down to like, a plan every day of those three months. ‘What if you learn something on like the third week that changes the rest of the plan?,’" developer Loren Yu remembers asking. "And they were like, oh, well it’s the rest of the plan, so it can’t change."

The mistake project managers made was thinking that they could have their programmers embrace agile methods while managers themselves retained the top-down control they enjoyed under conventional software development methods. But that doesn't work. The whole point of agile programming methods is to empower rank-and-file programmers to adjust their plans based on the progress of the project and feedback from users.

And project managers aren't just reluctant to do this because they're close-minded or set in their ways. Traditional hierarchical management systems strongly discourage this kind of improvisation. If a manager allows the programmers under him to deviate from the official plan based on feedback from users, the manager could get in trouble with his supervisors for not delivering on the agreed-upon feature set. So even if some features are of no actual use to the user, the safe course of action is to build them anyway.

And this means that successfully adopting agile methods requires buy-in from the highest levels of the organization. A key part of the FBI's successful rescue of the Sentinel project was for CIO Chad Fulgham to take over management of the project from contractor Lockheed Martin. By taking personal responsibility for the success of the project, Fulgham was able to give subcontractors and project managers the breathing room they needed to reprioritize tasks and tweak designs based on user feedback.

And this is ultimately why improving government IT is so difficult. The problem isn't that agile methods are unknown in government or are inherently difficult to practice. The problem is that they run strongly against the grain of the government's hierarchical culture. To empower programmers and users, managers need to feel like they have breathing room they need to experiment and sometimes fail — while iterating quickly enough that they can recover from the failures and deliver the project in a timely manner.

That's not how senior government officials are used to operating. But they're going to have to get used to it if they don't want to suffer from more software fiascos in the future.

Explanatory journalism is a public good

At Vox, we believe that everyone deserves access to information that helps them understand and shape the world they live in. That's why we keep our work free. Support our mission and help keep Vox free for all by making a financial contribution to Vox today.

One-Time Monthly Annual

$95/year

$120/year

$250/year

$350/year

Other

$

Yes, I'll give $120/year

Yes, I'll give $120/year

We accept credit card, Apple Pay, and Google Pay. You can also contribute via