Five backup techniques and none of them reliable: the night GitLab deleted its database

Having backups is not the same as being able to restore. The difference only shows on the day you need it, and by then it is too late to discover it. In January 2017, a cloud service for hosting code discovered it live, with thousands of people watching.

One clarification before we start: Hábil did not take part in this case. We tell it from what GitLab published in its public postmortem and what the specialized press reported.

Tuesday, a little after eleven at night

It is January 31, 2017. GitLab.com, the cloud service where software teams keep their code, their to-dos and their reviews, has been having trouble for hours. Since the afternoon, the main database has been under abnormal load. The team suspects spam. Users cannot post comments [1][2].

Behind that load lies a coincidence nobody had foreseen. On top of the spam, a process to delete the data of one of the company's own employees had been scheduled by mistake, triggered by an abuse report [1]. Together, the two squeeze the database.

Around eleven at night (UTC), the pressure leaves a more serious consequence. The main database has a replica: a second copy kept up to date by receiving every change, ready to take the place of the first if it fails. Under the load, the replica falls behind. And when it tries to catch up, the changes it needed are gone: the main database discarded them before the replica received them [1].

There is no automatic way out. The replica has to be rebuilt by hand [1].

An engineer gets to work. He deletes the replica's data directory in order to copy it again from the main database. He launches the tool that makes that copy, and the tool does not respond. It seems frozen. He adjusts parameters. The database refuses to restart. He adjusts others [1].

What nobody knows at that moment is that the tool was not frozen: it was silently waiting for the main database to start sending data, something that can take several minutes. That behavior was not in the team's procedures or in the official documentation [1][2].

It is night in the Netherlands, where the engineer works [4]. The problem has been going on for hours. He believes the failed copy left junk in the replica's directory and decides to clean it out again.

He types the delete command. He runs it.

A second or two later he realizes: he is not on the replica. He is on the main database [1][2].

He stops it. But the delete command does not ask and cannot be undone. Of some 300 GB of data, about 4.5 GB remain [2][4].

The search

At that moment, everyone is asking the same question: where do we restore from?

GitLab was not a makeshift company. On paper, it had several layers of protection. The team starts checking them one by one, and writes them down in a Google document it makes public so anyone can follow the progress [1][3].

The daily backup to the cloud. Every 24 hours, a scheduled job was supposed to take a full dump of the database and store it in Amazon storage (S3). The team opens the storage. It is empty [1][2][4].

Why? The job ran version 9.2 of the dump tool against a database on version 9.6. The tool picked its version based on the data directory of the server where it ran, and that job ran on an application server, which had no data directory. So it took the old version and failed [1].

It failed every day. And nobody noticed? The job did send an email when it failed. But those emails were rejected: the receiving server bounced them because DMARC was not enabled for the scheduled-job emails, a mechanism mail servers use to verify that a message is legitimate [1][5]. The alarm existed. It never reached anyone.

The cloud disk snapshots. GitLab ran at the time on Microsoft Azure, which can take periodic pictures of an entire disk. They were enabled for the file servers, but not for the database servers: the team assumed the other backups were enough [1].

The replica. It is the first thing anyone would think of. But a replica is not a backup: it copies everything, mistakes included, and it serves to take over from a server that went down, not to go back in time. Besides, in this case it no longer existed: its directory had been deleted minutes earlier, as part of the rebuild [1].

The server snapshots. One last layer remained: disk pictures taken with LVM, the system's logical volume manager. They were taken every 24 hours, and not as a backup, but to copy real data to a test environment (staging). The most recent scheduled one was almost a day old [1].

That early morning, the public document, which counted the daily dumps and their copy in S3 separately, summed up the situation in a sentence that went around the internet:

"Out of five backup/replication techniques deployed none are working reliably or set up in the first place." [2][4]

Six hours of luck

And then the only good news of the night appears, and it is almost an accident.

That same day, around 17:20 UTC, an engineer had manually taken a snapshot of the database to load it into the test environment. He took it to have recent data for testing a tool to spread the database's load [1]. That copy, made for something else, was now the only one that was six hours old, not twenty-four.

The team chooses that copy. The alternative was losing almost a full day [1].

It still has to be brought back. And here another decision appears, one someone made a while back for good reasons. The test environment ran on low-cost Azure disks, without high-performance storage, to save costs. Those disks were limited to about 60 Mbps. Copying the data from the test environment to the main server took around 18 hours. The bottleneck was not the network or the processor: it was the disks [1].

Eighteen hours of copying, with GitLab.com out of service.

The recovery, live

What GitLab did during those hours is the other half of this story, and it is what sets it apart.

Instead of closing the door and publishing a terse statement at the end, it reported in public what was happening while it recovered. Its status account on Twitter said it bluntly: "We accidentally deleted production data and might have to restore from backup" [1][2]. The working document, with the team's notes and the list of what had failed, remained in plain sight [1][3]. And the recovery was streamed on YouTube. It reached about 5,000 simultaneous viewers and was, for several hours, the second most watched live stream on the platform [1].

On February 1, around 17:00 UTC, the database was restored, still without the webhooks (the automatic notices GitLab sends to other systems when something changes). Around six in the evening, the service was back, with the final steps completed [1][2]. In total, GitLab.com was down for around 18 hours [1].

The team took care of details that only someone who has been through this before notices. It recovered the webhooks by bringing up a separate database from the same snapshot, without the step that strips them when data is copied to testing, and moved them from there to the main database. It moved all the database's identifier counters forward by 100,000, so that no number used during the lost hours would be assigned again to something else. And since the repositories' code does not live in the database but separately, on file servers that were not wiped, it rebuilt from there, in basic form, the projects created after 17:20, although without their to-dos or their reviews [1].

How much was lost

The changes made to the database during about six hours were lost: between 17:20 UTC and midnight on January 31, according to the postmortem, which elsewhere places the end around 23:30 [1][2]. Its best estimate: about 5,000 projects, 5,000 comments and 700 new user accounts affected [1].

What was not lost matters too. The code and wikis stored on GitLab.com were out of service during the outage, but they were not lost. Companies that had GitLab installed on their own servers were not affected [1][2][4].

Ten days later, on February 10, GitLab published a full postmortem: the hour-by-hour timeline, every mechanism that failed and why, and a public list of tasks to correct it, each with its own tracking number [1]. It opened with a direct apology: "To the GitLab.com users whose data we lost and to the people affected by the outage: we're sorry" [1].

What this case teaches

It is easy to read this story as that of an engineer who was in the wrong window. That would be the wrong reading. GitLab's own postmortem does not stop there: it asks why a single mistake could cost so much, and the answer has little to do with that person.

1. A backup nobody has restored is an assumption

GitLab had backup jobs, cloud storage, snapshots and a replica. On an architecture diagram, that looks solid. What it did not have was proof that any of those mechanisms would give back a working database.

The postmortem says it in a single line, without embellishment: "Because there was no ownership, as a result nobody was responsible for testing this procedure" [1].

Note what that means for a leadership team. Nobody turned off the backups. Nobody decided to take risks. The daily backup failed silently because of a version mismatch nobody could see, and it raised the alarm through a channel nobody received. The only way to discover it earlier was to try to restore. And since that task belonged to no one, no one did it.

2. A replica protects against a crash, not against a mistake

This point confuses even experienced teams. A replica is meant to keep the service running if a server goes down. Depending on how it is configured, it may lag behind, like GitLab's that night. And if someone deletes data, or an application writes junk, the replica copies the deletion and the junk with the same diligence. To go back to an earlier moment you need something else: a backup that keeps the history, or a log of changes that lets you rebuild the database up to a given minute.

The postmortem draws the line clearly: the replica was there to take over from the server, not to recover from a disaster [1].

3. What is done by hand, after hours, has no safety net

Rebuilding the replica was neither automated nor well documented. GitLab acknowledges it: the procedure was manual, it relied on improvised scripts and the documentation was scarce [1][2]. An engineer in the middle of the night, after hours of incident, had to remember every step, tell apart two servers with almost identical names (db1 and db2) and guess whether a silent tool was frozen or working.

Under those conditions, the risk of making a mistake grows a lot. The first task on GitLab's list was, in fact, to make it more evident which server and which environment each person is on when typing a command [1].

4. Cost-saving decisions come due on the day of the disaster

The Azure snapshots were not active for the databases because the other backups were thought to be enough. The test environment had slow disks because that cost less [1]. Each decision was reasonable on its own. Together, and added to the lost replica, the daily backup that kept failing and the rejected alerts, they contributed to a two-second mistake ending in a recovery of about 18 hours.

There is nothing wrong with saving money. What is missing, almost always, is someone writing down the consequence next to the decision: "with these disks, restoring takes this many hours; do we accept that?" If that question is asked beforehand, it is a decision. If it is discovered on the day of the disaster, it is only suffered.

5. What GitLab did well is also part of the lesson

It would be unfair to tell this story without saying it plainly: GitLab handled the crisis with a transparency few companies dare to have. It reported in public during the recovery, opened its working notes, streamed the recovery, apologized without hedging and published an analysis that names each of its own failures [1].

And it learned from its own transparency. The public document ended up showing the name of the engineer involved; in the postmortem, GitLab announced that in the future it would omit names, because other engineers might not be comfortable having theirs published [1]. The analysis points at the processes, not at the person.

That attitude is not just good image. A company that can say in public "this failed, and this is why" is a company that can fix it. Transparency does not eliminate risk, but it makes it possible to examine failures and assign concrete improvements.

What they planned and what they had already changed

The postmortem published a list of open tasks, each with its public number. Among them [1]:

• Make it evident on the command line which server and which environment you are working on.
• Monitor backups, so that a failing backup does not go unnoticed.
• Take database snapshots every hour instead of every day, and enable disk snapshots for the databases.
• Investigate point-in-time recovery and continuous archiving of the database's changes.
• Test, in an automated way, that backups can be restored.
• Improve the replication documentation and procedures.
• Name an owner of data durability (someone accountable for keeping data intact and recoverable).
• Prepare disaster recovery for everything that is not the database as well.
• Make deleting a user for spam not remove them permanently.

By the time the postmortem was published, GitLab was already taking LVM snapshots every hour and had started building a public dashboard of backup status; Azure disk snapshots for the databases were still under evaluation [1]. According to InfoQ, in the two weeks after the incident GitLab began archiving the database's changes in real time to Amazon storage, and a test point-in-time restore took just under two hours [5]. Months later, the team was holding days dedicated to restoring its backups (the main database, other databases and a disk snapshot) to check that they really backed things up [6][7]. In June 2017 those restores were still manual and automation was on the to-do list [6]. Changing a habit takes time, even after a scare like that.

The postmortem sums up the spirit of the change in a sentence that applies to any operation: "An ideal environment is one in which you can make mistakes but easily and quickly recover from them with minimal to no impact" [1]. It is not about forbidding commands. It is about making sure a mistake does not cost 18 hours.

What this means for your company

You probably do not operate a global platform like GitLab.com. But the pattern in this story shows up in companies of any size, and almost always with the same ingredients:

• Infrastructure built by hand. Servers, networks and scheduled jobs someone configured once, that work and that nobody dares to touch. Nobody knows for sure what they contain, because it is not written down anywhere.
• Backups that are scheduled, but never restored. There is a job that "does the backup" and a report that says it ran. Nobody has tried to bring up a complete database from that file.
• Alerts that reach no one. Emails to mailboxes nobody reads, or that get rejected, as at GitLab.
• Procedures that live in one person's memory. They work as long as that person is around, rested and available.
• Savings without the consequence written down. Slower disks, less frequent copies, fewer environments, all without anyone having calculated how long recovery would take with those decisions.

The difference between a night like GitLab's and a one-hour scare is not about having more tools. It lies in three practices that can be reviewed:

Infrastructure written as code. When every server, network, permission and scheduled job is described in files that are stored and reviewed like any software change, configurations become visible and reviewable. If a backup job runs on the wrong server, like GitLab's, the error can be seen in a review, instead of staying hidden on a machine. And when it comes with review and testing, that description makes it easier to rebuild whatever is lost consistently.

Evidence of every change. Who changed what, when and why, recorded without depending on anyone's memory. When something breaks, that history is the first thing you check.

Rehearsed restores. A backup is worth what its last successful restore is worth. It is rehearsed on a schedule, you measure how long it takes and compare it against what the business agreed it could lose and wait. If nobody has restored in months, what you have is not a proven backup: it is an assumption.

Five questions for your next committee

1. When was the last time we restored a complete database from a backup, in a separate environment, and it worked? If the answer is "never" or "I don't know," that is your first risk.
2. How long would that restore take, measured and not estimated, and how many hours of data would we lose?
3. Who owns making sure the backups work? A name, not a department.
4. If the backup job fails tomorrow, who finds out and how? Has that alert been tested?
5. Could we bring our infrastructure back up exactly as it was if we lost the server, or does it depend on what one person remembers?

If two or more answers are "I don't know," your backups are not proven.

Where to start this week

You do not need a project to take the first step. Choose your most important database and ask your team for one single thing: restore the most recent backup in a separate environment, measure how long it takes and check that the application works on top of it. Write down the result, the date and who did it. If it works, you already have your first piece of evidence. If it does not, you have just discovered on a quiet Tuesday what GitLab discovered at midnight.

Frequently asked questions

Does a database replica count as a backup?

Not necessarily. A replica can help keep the service running if a server goes down, but it also copies deletions and mistakes. To go back to an earlier moment you need a backup that keeps the history or a point-in-time recovery mechanism.

How often should we rehearse a restore?

It depends on how much your data changes and on what your business can tolerate. What matters is that it has a date on the calendar and an owner, and that it is repeated after major changes too, such as a database version upgrade. At GitLab, a version mismatch was precisely what broke the daily backup [1].

What is "infrastructure as code" and why does it help with backups?

It means describing servers, networks, permissions and scheduled jobs in files that are stored and reviewed like any software, instead of configuring them by hand. It helps because what is written down can be reviewed, compared and brought back up the same way. The data does not travel in that code: it travels with your backups, which are rehearsed separately.

Would more backup copies have protected us?

Not necessarily. GitLab had several layers and none of them was working reliably that night; the database was saved by a copy made by hand for something else [1][2]. Regular restore rehearsals would have helped detect earlier that some layers were not recoverable and that the alerts were reaching no one.

What happened to the engineer?

The postmortem does not blame him. It analyzes why a human error, which was bound to happen at some point, could have that cost, and it announces that in future incidents it will omit the names of the people involved [1].

How much data did GitLab lose?

About six hours of database changes, according to the postmortem itself. The code and wikis were not lost [1].

What Hábil does

Hábil is a Mexican engineering consultancy, founded in 2006. In Infrastructure provisioning we design with you the foundation your systems run on, we build it and hand it over to you working, documented and written as code, in your cloud, in your data center or in both. It includes, when your case calls for it, backups that restore and a recovery plan with agreed and tested times. Your team receives the code and the documentation to operate it, and to bring it back up exactly the same, without depending on us. See the page Infrastructure provisioning.

The first conversation is at no cost: it starts from what you have set up today, how it is backed up and when someone last restored it. Write to us on WhatsApp.

The facts, times and figures are those published by GitLab and the press cited; they are not our own measurements. Quotes are given in their original English. Times in UTC, as GitLab publishes them. Sources consulted on October 20, 2026.