The test suite that ate the release: when raising the time limit hides the cause

One Friday at 7:05 p.m., the new version of the corporate website did not go out: the build had been running for 99 minutes against a 100-minute limit, and at minute 100 the system cut it off.

The build is the delivery pipeline: it compiles, runs the automated test suite and gets the version ready. No test failed. They simply did not finish in time, and the delivery chain did what it was supposed to do: stop. The version carried a text change and a fix to the contact form, and marketing needed it that day: a campaign was launching on Monday.

In the team channel, someone wrote what everyone was thinking: "What if we raise it to 120?"

The question for you is a different one: how many times did your team raise a time limit this year, and in how many of them did someone ask why it was needed?

A clarification before we start: this is a hypothetical, illustrative scenario. The team, the site and the figures are invented and do not describe any Hábil client. The pattern shows up in many teams and tends to start the same way: a design defect covered by a patch that sounds reasonable. What comes from published sources carries its reference number; when something is our own reading, we say so.

In 1 minute

• The risk: treating test duration as a configuration problem: every time delivery gets close to the limit, someone raises the limit and the cause keeps growing.
• The case: in five weeks, the build limit went from 30 to 100 minutes. The cause was there from day one: every test file rebuilt the entire site before checking its own piece. No patch touched it.
• The cost of not acting: a release stalled on the day the business needed it, a campaign without its page, and a team that stops trusting its own alarms. For the validation that runs with every change, Fowler cites a guideline of about ten minutes [1] and DORA, a limit of about 10 minutes for tests [2]; on this team, the test stage ended up at about 58 minutes and the limit at one hundred.
• What you need to decide on Monday: that test duration gets a budget and is compared version against version, and that no patch is accepted unless it explains the cause, whether a person or an artificial intelligence assistant proposes it.

In this article

1. What happened on the Friday the release did not go out?
2. How did it get to one hundred minutes in five weeks?
3. What was the real cause?
4. Why did no patch solve the problem?
5. What does industry practice say about how long a release should take?
6. Which three rules change the story?
7. What role did artificial intelligence play?
8. What does the boring version of this story look like?
9. Key takeaways
10. What does Hábil do?
11. Frequently asked questions

What happened on the Friday the release did not go out?

The release did not stop because of an error in the code: it stopped because the tests took longer than the time they were allowed.

It pays to separate two clocks from the start, because in this story they are easily confused:

The limit is a permission. The duration is a fact. Raising the permission does not change the fact; it only changes when you find out about it.

The site was the corporate website of a mid-sized company: a few pages in Spanish and English, a form and a news section. Nothing mission-critical. That is why the case is useful: the same pattern can affect more critical systems, with an impact that depends on their architecture, the available capacity and the design of their tests.

In week 5, the test stage was running at around 58 minutes under normal conditions; the rest of the full build was added on top of that figure. That Friday, in addition, another pipeline was using the same integration machine: the full build, with all its stages and the wait for resources, reached the 100-minute limit and stopped. The shared machine was the push, not the cause: a healthy suite would have taken longer, but far from the limit. This one already lived so close to the limit that any extra load was enough to cross it.

How did it get to one hundred minutes in five weeks?

Through five decisions that, one by one, made sense, and none of which asked why it took as long as it did.

The team used artificial intelligence assistants to generate a good part of the code and the tests, and every new feature arrived with its test. That looked like discipline. Every row of the table had an explanation that sounded right: "the suite grew", "big files are slow", "the machine is choking". And nobody asked the only question that mattered: why does it take as long as it does?

What was the real cause?

Every test file rebuilt the entire site before checking its own piece: a cost that should have been paid once was paid two hundred and thirty times.

The clue was in the table the whole time. Look at week 5: almost the same tests, about 18 more minutes. Since the scope of the tests changed little, the reasonable hypothesis was that those minutes came from how they were run, not from what was being tested. A hypothesis is confirmed by measuring: separating preparation time, testing time and the wait for resources. In the scenario, that measurement shows that almost all of it was preparation.

To check that a button said "Send" in the English version, the test file did all of this: compile the site, generate all its pages in both languages, get it running and, only then, look at the button. The next file, the one for the same button in Spanish, repeated everything from scratch. And the next one. About two hundred and thirty times.

It is like building the building again before checking each light bulb.

With 60 files, that repeated cost did not hurt, and since nobody measured how much went into preparing and how much into testing, nobody saw it. With every new file, it grew in proportion. That is the birth defect: it was not a week 5 problem; it was a design that, from week 1, built in a repeated cost that would grow with every new file.

With that in hand, every patch reads differently:

• Splitting 4 files into 13 turned 4 rebuilds of the site into 13.
• Lowering parallelism made those rebuilds run more in a queue and less at the same time.
• Raising the limit gave more rope to everything above.

The "no reason" failures of week 3 also have another reading. In the scenario, they were consistent with a saturated machine. Google documented in 2016 that, among several possible causes of tests that sometimes pass and sometimes fail with the same code, are concurrency (several tests competing for the same thing at the same time) and infrastructure problems [3]; confirming it in a specific case requires execution metrics and logs. Giving them more time without measuring did not cure them: it took away their voice.

Why did no patch solve the problem?

Because all of them addressed the symptom (the clock) and none addressed the work that was being repeated.

No patch was wrong in itself; what all of them lacked was measuring before deciding.

A time limit works like a smoke detector. If you raise it every time it goes off, there is no less fire: there is less warning.

Google describes it with numbers from its own operation: about 1.5% of all its test runs reported an unstable result, and almost 16% of its tests had some level of instability [3]. Most relevant for you is the consequence they point out: with so many false positives, it is common for legitimate failures to be ignored [3]. And they warn that automatically setting aside unstable tests can easily mask a real error in the code [3]. Tolerating an alarm has a price, even if it does not appear on any invoice.

That is why the moral of this story is not "the suite grew until it broke". It is more uncomfortable: it was born with the defect, and every patch that did not measure the cause made it slower.

What does industry practice say about how long a release should take?

That the validation that runs with every change should be around ten minutes, and that when it starts to grow you act immediately, not when it no longer fits.

Martin Fowler, in his article on continuous integration, picks up the extreme programming (XP) guideline of a ten-minute build and says that, for most projects, it is perfectly reasonable [1]. And he asks teams to watch build times and act as soon as they start going over the ten-minute rule [1]. He also observes that, at least in enterprise applications, the usual bottleneck is testing [1].

DORA describes continuous integration in similar terms: tests should run in a few minutes, with an upper limit of about 10 minutes [2]. And it sets a team discipline: if the automated tests fail, the team stops what it is doing to fix them immediately; if the build cannot be fixed in a few minutes, the change that broke it is identified and reverted [2]. DORA also asks for tests to be reliable and fast, to give feedback as early as possible, and proposes moving the longer tests to a separate build [2]. Note the difference: deciding which tests run with every change and which run at another time is a design decision made in advance; moving them out because they no longer fit is just another limit being raised.

Placed next to the scenario:

Our reading: ten minutes is not a magic number, but the practice makes it clear that duration is watched and defended as a property of the system, not as a box that gets adjusted when it gets in the way.

Which three rules change the story?

None of them is a line of configuration: all three are judgment, and they fit in a one-page engineering policy.

Rule 1. What is expensive is prepared once and reused as far as it is safe

If the tests need the built site, a database with sample data or a running service, identify what can be prepared once and shared without breaking the isolation between tests.

What does not change, like an already built site, is used by all the tests; what does change, like a database, is isolated or reset in a controlled way. Nobody rebuilds the world to look at a button.

It is not a new idea. Modern test frameworks come with a place for global preparation: a function that runs once before all the tests [4][5], or resources whose scope is the entire test session and not each individual test [6]. The documentation of one of them notes that resources that require network access are usually expensive to create [6]. The mechanism changes from one tool to another: another of those frameworks recommends declaring the preparation as a project the others depend on, instead of its global function, because it integrates better with its test runner [4].

And what is prepared is not necessarily available in the same way to every test: one of those frameworks clarifies that global variables defined in its preparation cannot be read from the tests [5]. Your team should confirm this before considering the fix done.

Rule 2. The suite has a budget, and it is compared version against version

Just as a project has a money budget, the test stage has a duration budget, and it is watched in every version.

It is not enough for it to "fit within the limit". The question is whether it takes the same as the previous version and, if it takes longer, whether anyone expected it. For that, the delivery chain carries a gate: an automatic rule that stops the release when the duration exceeds the allowed variation. For example, if the stage took 12 minutes and the new version takes 16 without anyone having anticipated it, that is seen and explained before moving on.

It is a gate, not a statistic someone checks when they remember. With it, this team could have detected the deviation in week 2, and not in week 5, when it cost a campaign. It is Fowler's instruction to act as soon as the build starts going over [1], turned into an automatic control.

Rule 3. Raising the limit or splitting tests does not count as a fix

When the suite hits its budget, the first question is not "how much more time do we give it?" but "what is being repeated that should not be?"

Raising the limit can be a valid temporary measure for an urgent release, but it is recorded as what it is, debt with a date and an owner, and the investigation of the cause is opened that same day. If the limit goes up and nobody opens the investigation, it was not a technical decision: it was sweeping it under the rug.

What role did artificial intelligence play?

None that makes it guilty or innocent: the proposals answered the question the team was asking, and nobody asked the one about the cause.

Blaming the tool or absolving it does not help management. The question the team was asking was "how do I make it pass?", and the proposals it got answered that: more time, more pieces, fewer processes. Plausible answers to the wrong question.

The problem was something else: the team accepted every patch without asking why. When a person with little experience proposes raising a time limit, someone with more craft usually asks them "and why does it take that long?" When the proposal arrives instantly and written with confidence, that question is skipped more easily.

The speed of assistants does not remove the need for judgment: it increases it, because now errors are also produced faster. The rule this story leaves is simple to write and demanding to follow: a patch that does not explain the cause is not accepted, whether it comes from a person or from a tool. Change review demands evidence of the cause, of the expected impact and of the measurement afterwards, regardless of where the proposal came from.

What does the boring version of this story look like?

It is the same week 2, with a gate that sounds the alarm in time and someone who asks why.

In that version, on the Tuesday of week 2 the gate flags that the stage went from 12 to 16 minutes with no explanation. The version is held for a few hours. A person on the team measures where the time goes and finds that almost all of it is preparation: every file builds the site from scratch.

That afternoon, the site is built once at the start and all the tests use it; the sample database is reset in a controlled way between tests. After the change is validated, the stage can return to the agreed budget and the limit stays where it was.

Three weeks later, on the Friday of the campaign, another pipeline is using the same machine and the build takes a little longer. With enough margin for that extra load, the risk of the suite stopping the release is much lower.

That is the boring version. And in a delivery chain, boring is the goal.

Key takeaways

Four questions for your next technology committee:

1. How long does the test stage of our main systems take today, how long did it take three months ago, and who watches that difference?
2. How many times did we raise a time limit this year, and in how many of those times was an investigation of the cause opened, with a date and an owner?
3. Do we know what part of our test duration is preparation and what part is testing?
4. When an artificial intelligence assistant proposes a patch, does our review process require it to explain the cause, or only that the release passes?

What does Hábil do?

Hábil helps make your delivery chain fast and reliable, so that releasing a change stops being a gamble.

A test suite with growing duration can delay releases and generate operating costs nobody sees coming. We work with your team on four fronts:

• Tests that measure what matters. Every test checks something about the business, and what is expensive is prepared once, not two hundred and thirty times.
• A watched duration budget. Test duration is controlled like any other budget, with a gate that compares it version against version.
• Every alarm with an owner and a cause. Intermittent tests and time limits are treated as warnings to investigate, not as noise to silence.
• AI assistants without losing judgment. Your team takes advantage of the speed of assistants with a review that requires every change to explain its why.

We integrate testing and security into the delivery path. See Cloud and DevSecOps.

If your test stage has been growing for weeks and the limit has already been raised more than once, let's start there. Let's talk about your case. The first conversation is at no cost.

Frequently asked questions

Is raising the build time limit wrong?

Not necessarily. It can be a valid temporary measure to get an urgent release out. The problem appears when it is used as a fix: if the limit goes up and nobody measures where the time goes, the cause keeps growing. That is why every increase is recorded as debt with a date and an owner.

How do I know if my team has this same defect?

A clear sign is the one from week 5: duration goes up much more than the number of tests. Another, that splitting slow files does not improve the total time. The underlying check is to measure, per file, how much goes into preparing the environment and how much into testing. If preparation dominates and is repeated in every file, that is where the cost is.

Sources consulted on November 6, 2026. Quotations from English-language documents are paraphrased.

Dorian Chávez tells a narrated version of this pattern, in Spanish, on his Substack, "En construcción": https://enconstruccionmx.substack.com/p/la-suite-que-se-comio-la-entrega