Knight Capital: 45 minutes, one server nobody updated and more than $460 million

On August 1, 2012, Knight Capital lost more than $460 million in about 45 minutes. According to the regulator, it all began with a software change that was installed on seven of eight servers. The eighth kept old code that was no longer in use, and that code woke up when the market opened. What followed was a chain of controls that were not where they were needed.

The case is widely documented in public records. The U.S. Securities and Exchange Commission (the SEC, the country's market regulator) investigated it and published its findings in an 18-page order [1]. Almost everything below comes from that order. One important caveat: Knight accepted the settlement with the SEC without admitting or denying those findings [1, section II].

This article is for people who run a company or a technology area. It covers four things:

1. What happened, hour by hour.
2. Why it was not bad luck, but a chain of small decisions.
3. What a director can learn, in business terms.
4. What to ask at your next committee meeting and where to start this week.

One clarification before we start: Hábil did not take part in this case. We tell it from what the SEC and Knight itself reported publicly.

First, who Knight was

Knight Capital was one of the largest firms on Wall Street in executing stock orders. According to the SEC, in 2011 and 2012 its trading accounted, in general, for about 10% of all shares traded on U.S. listed markets [1]. When a small investor bought shares through a brokerage, the order often passed through Knight.

To do that, Knight used a system called SMARS: an automated, high-speed order router. Its job is simple to explain. It receives a buy or sell order (the "parent" order) and, depending on what is available in the market, sends one or more derived orders (the "child" orders) to different trading venues (exchanges and other markets where shares are bought and sold) to fill it [1, para. 12]. The logical expectation is that once the parent order is complete, the system stops sending children.

That last idea is the one that matters. Hold on to it.

The timeline, hour by hour

2003: a function nobody switches off

Knight stops using a SMARS function called "Power Peg." It stops using it, but it does not delete it. The code stays inside the system, present and ready to be activated [1, paras. 13–14].

That code held one key piece: a counter that added up how many shares of the parent order had already been executed. That counter was the brake. It told the system: "this order is complete, stop sending."

2005: a change nobody tests again

Two years later, the team moves that counter to another part of SMARS for use in a different function. No one tests Power Peg again to see whether it still works without the counter [1, para. 14]. Why would they, if nobody uses it anymore?

From that day on, Power Peg is a car with no brakes parked in the garage. Nobody drives it. Nobody checks it. The keys are in the ignition.

July 2012: eight servers, seven copies

The New York Stock Exchange (NYSE) is about to launch a program for retail investor orders, the Retail Liquidity Program, which starts on August 1 [1, para. 12]. Knight wants its clients to take part from day one, so it writes new code for SMARS.

Two technical decisions set the trap:

• The new code replaces the old Power Peg in that part of the system.
• To activate, it reuses the same flag (a switch inside each order) that years earlier turned Power Peg on [1, para. 13]. The idea was to delete Power Peg, so that the flag could only turn on the new code.

Starting on July 27, a technician installs the new code in stages, server by server, over successive days. SMARS runs on eight servers. On one of them, the new code is not copied [1, para. 15].

Nobody notices. According to the SEC, Knight did not require a second technician to review the installation, and it had no written procedures that would have required one [1, paras. 15 and 26]. The eighth server is left with Power Peg inside and without the new code.

Wednesday, August 1, 8:01 a.m. (New York time): 97 emails nobody reads

The market opens in almost an hour and a half. Orders from the new program arrive, flagged to trade before the open. Around 8:01, an internal Knight system starts sending automatic emails to a group of staff. They mention SMARS and flag an error: "Power Peg disabled" [1, para. 19].

Before 9:30, 97 of those emails arrive. No one acts on them. This is not one person's negligence: those messages were not designed as alerts, and staff, in general, did not review them when they arrived [1, para. 19]. The SEC puts it this way: they were real-time messages, caused by the failed installation, and they gave a potential opportunity to identify and fix the problem before the open [1, para. 19].

The system gave a warning. The warning went to a mailbox nobody was watching.

9:30: the market opens

The bell rings. Seven servers process the new program's orders correctly. The eighth does not.

Every order carrying the reused flag that reaches that server wakes up Power Peg. And Power Peg no longer has its counter: it cannot tell when the parent order is complete. So it sends child orders without stopping, one after another, at full speed, for every order it receives [1, para. 16].

The most unsettling part: another part of Knight's system did know the orders had already been completed. But that information did not reach SMARS [1, para. 16].

Remember the idea I asked you to hold on to. The data that would have served as the brake existed somewhere else in the system, not in the part that was sending.

The next minutes: searching blind

From the open, senior Knight staff see on their monitoring tool that a huge position is building up in an internal account, "Account 33" [1, paras. 24–25]: shares bought or sold that Knight now carries, with all their risk. They see the symptom. They do not see the cause.

The SEC describes why it took them so long to understand what was happening:

• The monitoring tool depended entirely on a person watching it: it did not generate automatic alerts about the firm's exposure, that is, the money it had at risk [1, para. 25].
• It did not display the limits: whoever watched it had to know each account's limit by heart to realize it had been exceeded [1, para. 25].
• With that morning's volume, it lagged, and its reports came out inaccurate [1, para. 25].
• Account 33 had a $2 million limit, but it was not tied to automatic controls over Knight's total exposure; and the monitoring tool did not stop the sending of orders on its own either [1, paras. 23–25].
• And there were no procedures to guide staff through a serious incident; the SEC concluded that the technical team needed clear guidance on when to disconnect a failing system [1, paras. 27 and 42].

Meanwhile, Knight stays connected to the market. Its technology team tries to find the problem live, with the system sending millions of orders [1, paras. 10 and 27].

The remedy that makes everything worse

In one of its attempts, the team does what many of us would do: go back to the previous state. It uninstalls the new code from the seven servers where it was correctly installed [1, para. 27].

But on those seven servers, underneath the new code, Power Peg was still there. When the new code comes out, the flag turns the old code back on. Now eight servers are sending orders with no brake [1, para. 27].

The rollback, which was supposed to be the safety net, was the final push, because nobody knew for certain what was underneath.

About 45 minutes later

When Knight finally stops sending orders, this is the SEC's count [1, paras. 1 and 17]:

• 212 retail orders (from small investors) set it all off.
• SMARS sent millions of child orders, with more than 4 million executions in 154 stocks and more than 397 million shares.
• Knight ended up, without intending to, with a net long position of about $3.5 billion in 80 stocks and a net short position of about $3.15 billion in 74 stocks.

The damage spread beyond Knight. In 75 of those stocks, its trades were more than 20% of the volume and contributed to price moves of more than 5%; in 37, the price moved more than 10% and Knight was more than half of the volume [1, para. 18]. Those executions affected other market participants: some received less favorable prices and others more favorable ones than they would have received without them.

The following days

• Thursday, August 2. Knight reports a pre-tax loss of approximately $440 million [3]. That morning, according to the Associated Press, its stock loses half its value early in the session [5].
• Monday, August 6. A group of investors agrees to put $400 million into Knight in exchange for convertible preferred stock (shares with special rights that can be exchanged for common shares) [4]. In its quarterly report, Knight itself acknowledges “substantial doubt about the Company’s ability to continue as a going concern” [6].
• July 1, 2013. Knight Capital Group and GETCO combine to form KCG Holdings; Knight stops being an independent company [1, para. 11].
• October 16, 2013. The SEC imposes a $12 million fine. It is the SEC's first enforcement action under its market access rule, adopted in 2010: the rule that requires firms to have risk controls before sending orders to the market [1][2]. The final loss figure the SEC sets is more than $460 million [1, paras. 1 and 10].

A note on the figures: the $440 million is the estimate Knight published the next day [3]; the more than $460 million is what the SEC established at the end of its investigation [1]. Here we use the SEC's figure.

It was not bad luck: it was a chain

It is tempting to tell this story as "a technician forgot to copy a file." The SEC does not tell it that way. Its order points to failures of control, not of one person [1, paras. 9 and 40–44]. Any one of these links, on its own, would have broken the chain:

Read the third row again. According to the regulator, a simple double-check of the installation “could have identified that a server had been missed and averted the events of August 1” [1, para. 41].

And there is a detail that should make any committee uncomfortable: Knight did have controls. Its client interface, its internal order system and its internal execution system had controls against erroneous orders [1, para. 20]. The problem is that they sat before SMARS. In the last stretch, the one that touched the market, there was no sufficient control [1, para. 21].

The warning that came a year earlier

In October 2011, Knight ran a disaster recovery test over a weekend, using test data. On Monday, one of its systems kept using that test data to quote in the real market. It lost almost $7.5 million [1, para. 33].

Knight fixed that problem: it limited the system's hours, changed a control and added a step to a checklist. What it did not do, says the SEC, was ask whether it had sufficient controls against erroneous orders no matter which system sent them or why [1, paras. 33 and 44]. It reacted to the case, not to the cause.

The signature that said something else

In March 2012, Knight's chief executive signed the annual statement required by the market access rule. The text did not say the controls complied with the rule; it said Knight had “processes” in place to comply with it. It was an unintentional drafting error, the CEO did not notice it and believed he was signing that everything complied [1, para. 34]. For the SEC, stating that “processes” exist is not equivalent to stating that the controls and procedures are reasonably designed and comply with the rule [1, para. 45].

In a boardroom it sounds like a technicality. That difference does not by itself explain what happened on August 1, but it shows oversight that took for granted that the controls were enough, without checking.

What a director can learn

You do not operate a stock router. But if your company bills, invoices, approves credit, issues policies or sells online, you have systems that act on their own, and fast. These lessons are not about Wall Street; they are about any operation that cannot be stopped.

1. The costliest risk is not in the new code: it is in the code nobody remembers

Power Peg had gone unused for nine years. Nobody saw it as a risk because it "did nothing." In systems that have been around for years, it is common for switched-off functions, reused flags and integrations with no clear owner to stay behind. What is not used and not deleted is a debt that collects interest on the worst day.

2. A change installed by hand is not a controlled change

The problem was not that a person made a mistake: people make mistakes. The problem was that nothing was designed to detect the error. If installing on eight servers depends on someone repeating the same steps eight times without failing, sooner or later one of those repetitions will go wrong. An automated release path (the process that takes each change to the servers with no manual steps) reduces that variation: it installs the same thing everywhere and verifies that it came out the same. It does not eliminate the risk, but it makes it visible.

3. An alert nobody reads is a record, not a control

The 97 emails are the scene that should keep a director up at night. The system did detect the problem, an hour and a half ahead of time. But the warning was not designed for anyone to act on it. Ask yourself how many automatic warnings from your operation land today in a shared mailbox that nobody checks.

4. The brake has to sit where the risk is, and it has to be automatic

Knight had controls at the start of the path and monitoring after the fact. It lacked a brake right before the money went out, and it lacked limits that would stop the sending on their own. A limit that depends on a person remembering it while watching a slow screen is not a limit.

5. Going back is also a change, and it also has to be rehearsed

The attempt to return to the previous version made everything worse, because nobody knew what was underneath. A safe rollback is not "uninstall and pray": it is a procedure tested beforehand that takes you to a known state.

6. In an incident, the first decision is a business decision: do we disconnect?

For 45 minutes, the technical team searched for the fault with the system connected. The SEC notes that Knight needed clear guidance on when to disconnect [1, para. 42]. That decision should not be improvised in the middle of the fire or depend on whoever has the keyboard: it is agreed beforehand, with a name attached and a clear threshold.

7. Fixing the symptom is not fixing the cause

The $7.5 million loss in 2011 was a warning. Knight fixed that specific case and carried on as before. Every small incident is the cheapest chance you will get to find the big one.

Why this keeps happening

More than fourteen years have passed and the lesson still holds, for an uncomfortable reason: none of these controls requires exotic technology. A double-check, a well-routed alert, an automatic limit, a procedure for disconnecting. They combine process, technology and governance, and their real difficulty lies elsewhere: they compete with the rush. Knight had a fixed date, the launch of the Exchange's program, and wanted to be ready that day [1, para. 12].

When the calendar rules, the first thing to be cut is whatever "can't be seen": the review, the testing of the old code, the rehearsal of the rollback. Until it can be seen.

Five questions for your next committee meeting

1. How does a change reach our critical systems? Through an automated path that installs the same thing everywhere and verifies it, or because someone copies it by hand?
2. What code or switched-off functions are still in our systems, and who would answer if someone turned them on by accident?
3. What automatic alerts do our systems generate today, who receives them and who is obligated to act when they appear?
4. What limits on money, volume or quantity stop an abnormal operation on their own, and at what exact point in the path do they sit?
5. If a system starts doing damage tomorrow, who decides to disconnect it, at what threshold, and when did we last rehearse the rollback?

If any answer is "I don't know" or "it depends on who's there," turn it into a risk with an owner and a date. For each one, ask for five pieces of information: the affected system, its owner, the evidence that exists today, the impact if it fails and the next control that is needed. With that, your committee can prioritize a first system and decide the next step.

Where to start this week

Pick one system that collects, pays or moves money. Ask whoever releases it to walk you through, step by step, how the last change went out: who installed it, on how many servers, how they confirmed it came out the same on all of them, and what would have happened if it had failed. If the answer includes a "usually" or a "he knows the procedure," you already know where to start.

What Hábil does

Hábil is a Mexican engineering consultancy, founded in 2006, that builds systems for operations that cannot be stopped. One of the problems we solve is exactly the one in this story: making it stop being scary to release a change.

We design and implement automated release paths in the cloud, on your own servers or on both: every change goes out the same way, with testing and security review before it reaches your customers, test and live environments that resemble one another, an agreed rollback and an audit trail of who approved what. We start with the tools your team already uses and with the system that hurts the most, while operations keep running. See the page Get to production fast, with no surprises and, if you want the technical detail, the article CI/CD DevSecOps on your own infrastructure.

If you want to test these five questions against a specific system, let's start there. Request your first conversation, at no cost. We review how one of your priority systems is released today, and you leave with a first map of your risks and your next decision. Write to us on WhatsApp.

Frequently asked questions

Did Knight Capital go under?

Not exactly. It survived thanks to a $400 million injection from a group of investors, agreed five days later [4], but it lost its independence: in July 2013 it combined with GETCO to form KCG Holdings [1, para. 11].

Was it the fault of the technician who did not copy the code?

The SEC does not frame it that way. Its order points out that Knight had no written procedures or second review for installing code on that system, and no controls to keep unused code from being activated [1, paras. 9-D, 15 and 26]. A human error is foreseeable; what was missing was a system that would catch it.

Why do people talk about $440 million and also about $460 million?

The $440 million is the pre-tax loss Knight estimated on August 2, 2012 [3]. The SEC, closing its investigation in October 2013, established a loss of more than $460 million [1].

Does this apply only to financial companies?

No. Any company with systems that act on their own (collections, invoicing, payments, inventory, online orders) has the same exposure: a badly installed change, forgotten code and warnings nobody attends to. What changes is the speed at which the error turns into money.

Does automating releases eliminate the risk?

No; it reduces it and makes it visible. An automated path installs the same thing everywhere, verifies the result and leaves a record, but it still needs testing, automatic limits, alerts that someone acts on and a rehearsed rollback.

Where does a company that releases by hand today begin?

With a single system, the one that moves the most money or hurts the most, measuring first how it is released today. From there, automate in stages, without stopping operations.

The figures and facts of the case come from public documents consulted on October 20, 2026. Quotations are taken from the English originals.