HealthCare.gov: what saved the site was not just adding more people

A real, public case. Hábil did not take part in HealthCare.gov. We analyze it based on reports from the U.S. government and the press; the sources are at the end.

It is midnight on October 1, 2013. The contracted staff who run HealthCare.gov's account and identity system watch the site start up. It has just opened, and millions of people are supposed to use it to compare and buy health insurance. One of them would later describe the moment to the auditors: "We were sitting in the office at midnight when it started running and it wasn't looking good. Everything was turning red on our screens" [1].

That day, 250,000 users hit the site at the same time, many times more than planned. The crashes began two hours after launch. By the end of the day, according to the report by the Inspector General of the U.S. Department of Health, only six people managed to submit an application and select a plan [1].

Six. On the site that was supposed to serve millions of people in 36 states [1].

This is the story of how that site recovered in about two months. I am not telling it for the politics or for the health insurance. I am telling it because every technology executive has been, on a smaller scale, in the same room: a system that cannot fail, a date that will not move, and someone at the table saying "let's throw more people at it." What makes the case interesting is that that had already been tried, and on its own it was not enough. The site recovered when, on top of adding people, the way of working changed: a small core of experienced engineers, integrated at a single table with the agency and the contractors, system-wide metrics, and priorities cut ruthlessly. The auditors point to several causes, not just one [1].

Seventy-two hours before

The problem did not start on October 1. The government's auditors trace it back years. The Inspector General (OIG, the office that audits the Department of Health) interviewed 86 officials and contractors and reviewed thousands of documents. Its main conclusion fits in one line: the most serious problem was the absence of clear leadership, which delayed decisions, left tasks poorly defined and kept anyone from seeing the size of the problem as it grew [1].

The last days before the launch read like a movie script.

On September 24, the chief information security officer of the agency building the site (CMS, the Centers for Medicare & Medicaid Services) warned that a central component did not reasonably meet security requirements. On the 27th, instead of a normal approval, the CMS Administrator signed a temporary authorization to operate for six months [1].

On September 26, less than five days before go-live, CMS technicians ran load tests and found that the site could handle far fewer simultaneous users than planned. They drove straight to the infrastructure provider's offices and asked it to double all capacity in 72 hours. A manager at the provider put it this way: "The request was to double everything 72 hours from launch. […] renting planes to get hardware here that was intended for other clients" [1].

On September 29, the account and identity system, the door every user had to go through, suffered outages. That same day, the CMS Administrator wrote to the U.S. Chief Technology Officer that the site would be ready on October 1 [1].

There is one detail the report underlines that applies to any company: several people on the team believed the site's date was written into the law. It was not. The law required coverage to begin no later than January 1, 2014, and left it to the Department of Health to set the enrollment period, which by regulation started on October 1, 2013 [1]. On top of that came politics: the White House had committed publicly to that date, and people in the Department feared that a delay would hand arguments to those seeking to repeal the law in Congress [1]. The date nobody dared to move did not come from the law, but from a regulation and from politics. And no one in CMS leadership formally discussed moving it [1].

And one more: there was no complete end-to-end testing. One contractor summed it up for the auditors: "End-to-end testing is critical. It's suicide not to do it" [1].

Adding people had already been tried

Here is the fact almost nobody mentions about the case.

In August 2013, two months before launch, CMS saw that the project was far behind schedule and reacted the way almost any organization would: it temporarily assigned 60 more people from other areas to the 51 already working on the site. A 117% increase [1].

It was not enough. The report itself records how some of those involved felt: that it was already too late to bring others in, given the steep learning curve, and that they were embarrassed to "add others into a mess" [1].

In 1975, Frederick P. Brooks Jr., who had led development of the OS/360 operating system at IBM, wrote a sentence in The Mythical Man-Month that is now known as Brooks's law: "Adding manpower to a late software project makes it later" [5]. His reasoning: the work does not split into independent pieces, every new person needs someone to bring them up to speed, and the lines of communication grow faster than the team.

It is worth being precise. The OIG report does not cite Brooks, and it cannot be claimed that those 60 people delayed the site. What it does document is that adding them late, without clear leadership and onto a disorganized project, did not prevent the outcome. The link to Brooks is mine, and I offer it as a reading, not as a finding of the auditors.

Two weeks flying blind

After October 1, the first reaction was, again, to send more people. Within a few days, staff from CMS and from the prime contractor set up in the contractor's command center in Herndon, Virginia, and other contractors gradually joined [1].

But there was an underlying problem: nobody could see the whole system. Each contractor monitored its own component and could not see the others, even though they all worked together. There were several "command centers" in different places, one in Herndon and another in Columbia, Maryland, each with its own tools, and no leader or team with a view of the entire system and the authority to act. No one was formally serving as systems integrator [1].

One of the engineers who would arrive later said it at a conference in 2014: "There was no place to look to find out whether the site was up today or not except CNN, which was literally how we found out about problems a good part of the time in the beginning" [1].

To complicate things further, on the morning of the launch the federal government shut down for lack of funding. The shutdown lasted 16 days, and staff in the project's key offices was cut to about a third [1].

Meanwhile, the site was barely enrolling anyone. According to TIME, 26,794 people enrolled through the federal marketplace in all of October [4]. And the real capacity, according to a member of the team that later fixed it, was "maybe a few thousand users," while the official figure spoke of 50,000 [4].

October 17: patch it or scrap it?

On October 17, according to journalist Steven Brill's reconstruction in TIME, the White House was debating whether the site was worth repairing or whether to throw it out and start from scratch [4]. That day, Todd Park, then the White House Chief Technology Officer, changed jobs without changing offices: "On Oct. 17, I went from White House CTO to full-time HealthCare.gov fixer" [4].

Park recruited a group of engineers who had not been involved in building the site. The OIG describes them as "several leaders and engineers from top technology firms" [1]. Among them was Mikey Dickerson, a site reliability engineer at Google, that is, someone whose daily job was keeping an enormous site from going down [4]. One detail in the report says a lot about how they came in: for the most part they donated their time, although, under government employment rules, they were paid by the hour through one of the contractors [1].

The first thing they did was not write fixes. It was to measure. On the night of Friday, October 18, according to TIME, Gabriel Burt, chief technology officer of a data analytics company, and others spent about five hours putting together a dashboard, because the site had no quick way of knowing what was going on [4]. With that dashboard they found one of the first quick fixes: on the night of October 22, a page's load time dropped from eight seconds to two [4].

That same October 22, the Secretary of Health publicly announced what she called a "tech surge": Jeff Zients, an executive with experience running companies, was joining as management adviser, together with "experts and specialists" from government, the contractors and industry, "including veterans of top Silicon Valley companies" and "a handful" of Presidential Innovation Fellows [3].

The first major decision, made with that group: refactor the existing code, do not rebuild from scratch. Starting over would have taken time they did not have, and the agency's and contractors' staff were willing to work with the newcomers [1].

The room in Columbia

In late October, the command center moved to a room of about 370 square meters (4,000 square feet) that one of the contractors rented in an office park in Columbia, Maryland [4]. Over time it would become the site's formal operations center [1].

Dickerson called the first stand-up meeting on October 24. From then on, every day, weekends included, through October and November: at 10:00 and at 18:30, about 45 minutes each, with a phone line open 24 hours a day so that anyone working elsewhere could talk to the others as soon as something came up [4].

He posted the rules on a wall outside the control room. TIME reproduces them [4]:

1. "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."
2. "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank."
3. "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24–48 hours."

The OIG report describes the same shift in other words. A "badgeless, titleless" culture was adopted: tasks were assigned by skill and availability, regardless of whether the person was employed by the government or by any of the contractors [1]. For the first time, the technical team and the policy team worked together. One of the engineers described it this way: "We had policy folks figuring out the error messages. What did the code do or not do? What was supposed to happen?" [1]. And the agency hired a technical systems integrator, someone whose job was to make all the pieces talk to each other. According to the report, that improved coordination "by all accounts" [1].

There is a line from a CMS official, recorded by the auditors, that sums up the case better than any analysis: "CMS didn't need a technological surge, we needed an organizational surge" [1].

And the pressure to add more people? It was still there. Brill tells it this way: Zients had "carte blanche to add resources," but putting ten people on a fix that would take one programmer ten days does not turn it into a one-day project. "Coding doesn't work that way" [4].

The November clock

On October 25, Zients publicly set a goal: the site would work smoothly for the vast majority of users by the end of November [1]. The room's technical leaders later told the auditors that they had not been consulted about that date [1].

The road to that goal was hard. In late October, two failures in the infrastructure provider's equipment left the site down for 24 and 36 hours [1]. Some slept in nearby hotels and worked 24-hour shifts [1]. "Those were dark days," one official said [1].

But the numbers started to move. According to CMS's December 1 report, cited by the OIG, the share of time the site was up went from 42% in early November to more than 90% by the end of November [1]. (TIME reported 43% and 95% [4]; I use the official report's figure.) By December 1, the site could handle more than 35,000 simultaneous users without going down, according to the OIG; TIME spoke of at least 50,000 [1][4]. There were now system-wide performance dashboards and, when load overflowed, a "waiting room" that asked users to wait or leave their email address [1].

The deadline for coverage starting January 1 was extended from December 15 to December 24. On the last day, the site received 49,156 simultaneous users, according to the OIG [1]; TIME reported a higher peak, 83,000, on December 23 [4]. By December 28, about 1.2 million people had selected a plan through HealthCare.gov [1]. By the end of the first open enrollment period, 5.4 million people had selected a plan in the federal marketplace [1].

From six people on day one to more than a million in December.

What it cost

The rescue did not erase the bill. The OIG reports that six key contracts were initially estimated at 464 million dollars and, by early 2014, were already projected at 824 million; the one for the core system went from 58 to 207 million [1]. The Government Accountability Office (GAO) found that the site was launched without verifying that it met its performance requirements, and that obligations under the core system's contract grew from 56 to more than 209 million between September 2011 and February 2014 [2].

In January 2014, CMS changed its prime contractor. The outgoing contractor, the OIG says, found out from the Washington Post days before the new one arrived [1].

What this case teaches

1. Adding people is not enough, and sometimes it gets in the way

There were two waves of people. The August 2013 wave, 60 more people on a project without clear leadership, did not prevent the disaster [1]. The October wave, which also added staff from the agency, the contractors and the private sector, most of it within three weeks [1], was part of the recovery, because it came with leadership, metrics, integration and clear priorities [1]. The case does not allow the result to be attributed to a single factor.

Let us be honest about the facts: the rescue was not done with fewer people. It was done with different people, organized differently. The difference was not in the number, but in who arrived, how they were integrated and what instruments they worked with. Brooks was right in what he warned about, but the case does not say "never add people." It says: do not expect headcount to fix what the organization does not resolve.

2. A small, experienced, integrated core

The core that arrived in mid-October was small: "several" engineers, according to the OIG; "a handful" of fellows, according to the official announcement [1][3]. They did not come to do everyone's work. They came with experience in sites that do not go down, sat down with the contractors' engineers, who already knew the code, and directed them. TIME speaks of "dozens" of engineers, willing and even eager, whom that core coordinated [4].

That core was one piece, not the whole explanation. Among the key factors in the recovery, the OIG highlights that all CMS and contractor staff worked as a single team, badgeless, and a practice of "ruthless prioritization" that aligned the work with the most important and achievable goals [1].

The key word is integrated. Badgeless and titleless: whoever knows, speaks; whoever can, takes the task. An outside engineer who comes in to "audit" from the outside adds one more layer. One who sits in the same room, under the same rules, removes one.

3. Measure before you fix

The first thing was a dashboard built in one night [4]. Before that, the best source for knowing whether the site was up was television [1]. Without metrics, every new person has an opinion; with metrics, every new person goes after what the dashboard points to.

4. A single table, with rules

Not one more daily meeting, but the meeting: a single room, two check-ins a day, 45 minutes, a permanently open line and three rules on the wall [4]. The discussion is about what hurts in the next 24 to 48 hours, whoever knows does the talking, and nobody looks for someone to blame. It is the opposite of several command centers that cannot see each other [1].

5. Someone who owns making everything talk

Before the launch, no one was serving as systems integrator and each contractor looked at its own piece [1]. During the rescue, the agency hired one [1]. When a system is made of many pieces and many vendors, someone has to own the seams. If nobody does, that is exactly where the problem lives.

6. The date nobody moves deserves a question

The October 1 date was not set by law: it came from a regulation and was held in place by a political commitment, and no one formally discussed moving it [1]. Before adding people to hit a date, it is worth asking who set it, what happens if it moves, and what the minimum is that has to work on that day.

Applying it: two key engineers gone before Buen Fin

Scenario (illustrative, no client and no real case). I apply the lessons of HealthCare.gov to a situation many executives in Mexico will recognize.

It is September. Your e-commerce operation is getting ready for Buen Fin, the November weekend that concentrates a large share of the year's sales. Within the same two weeks, the two engineers who know the integration between the store, inventory and payments best both resign. Seven weeks remain.

The natural reaction is the August 2013 one: "let's hire five." The lessons of the case suggest a different order:

1. First, measure. Before they leave, ask them for a session to bring into view what only they could see: which alerts they checked, which dashboards, which processes they restarted by hand. If there is no dashboard for the full purchase flow (from the store to payment and inventory), that is the first deliverable, not the last.
2. Then, add few people, and experienced ones. Not five generic profiles who need training seven weeks before peak season. One or two engineers who have already worked on similar integrations and through high-volume seasons, whom your team can interview before they start.
3. Integrate them, do not just assign them. Have them come in with your company's accounts, in your repositories, on your schedule and to your meetings. No "vendor team" and "in-house team": a single table.
4. A short meeting, with rules. Two check-ins a day in the weeks leading up, focused on whatever could take down sales in the next 48 hours.
5. Cut ruthlessly. Anything that is not essential to selling, collecting payment and fulfilling orders that weekend gets postponed.
6. Review the date of each change, not the date of Buen Fin. That one does not move. But the date of the cart redesign, the new payment gateway or the catalog migration might.

Framed this way, Buen Fin is not won with more people. It is won with the right people, in time and sitting at the same table.

Five questions for your next committee

1. If the two engineers who know our integrations best left tomorrow, what would stop working, and does anyone else know it?
2. Do we have a dashboard of the full flow that matters to the business, or does each area watch only its own piece?
3. Who owns the seams between our systems and our vendors?
4. When we add people to a late project, how long do they take to become useful, and who integrates them?
5. Are the dates that "cannot be moved" in a contract, in a law or in a slide deck?

If two or more answers are "I don't know," that may be where your next October is.

What Hábil does when critical knowledge is missing

Hábil is a Mexican engineering consultancy, founded in 2006, that adds senior engineers to company teams in Mexico and the United States. We do not send "more people": we present profiles with experience in integration, cloud and regulated systems that you interview and validate before they start, and who work inside your controls: with your company's identity, in your repositories and within a daily window of working hours shared with your team.

You can start with one or two engineers who join your team, with a cell that we coordinate, or with a fixed-scope project. See the page Add engineers to your team.

Request your first conversation, at no cost. It starts from what you need to sustain or build, which profile fits and under which way of working. Write to us on WhatsApp.

Frequently asked questions

What was the HealthCare.gov "tech surge"?

It was the name the U.S. Department of Health gave, on October 22, 2013, to the reinforcement brought in to repair the site: a management adviser, Jeff Zients, and experts from government, the contractors and technology companies [3]. The technical core was recruited by the White House from engineers who had not built the site [1].

How long did the rescue take?

The OIG concludes that CMS recovered the site for high use in about two months: the improvement goal was met on December 1, 2013 [1].

What does Brooks's law say?

"Adding manpower to a late software project makes it later." It comes from the book The Mythical Man-Month, by Frederick P. Brooks Jr., published in 1975 [5]. It is a warning, not a prohibition: the cost lies in the time it takes to integrate each person and in the communication that multiplies.

So is it never a good idea to add engineers to a late project?

The case does not say that. In October 2013 people were added and the site recovered, but those people came with experience, were integrated at a single table regardless of employer, and worked with metrics and trimmed priorities [1][4]. The case does not allow the result to be attributed to a single factor. What was not enough was adding people without leadership or visibility [1].

Did Hábil take part in the HealthCare.gov case?

No. It is a public case, documented by the U.S. government and by the press. We use it because its lessons apply to any team that sustains a system that cannot fail.

How long does it take a Hábil engineer to integrate into my team?

It depends on three things: the profile you need, the interviews your team uses to validate the engineer, and how long your company takes to grant accounts and access to your repositories. In the first conversation we define the profile and tell you how soon they can start in your context.

Quotes are given in their original English. The figures are those reported by each source; when two sources disagree, the text says so and uses the official one.