The film that played for 45 seconds

The report

We uploaded an hour-long event film for a customer. Its streaming versions were still encoding, but the player already had a quality menu, so we opened it to check. At 1080p the film ran for 45 seconds and stopped. At 720p it managed 5 minutes.

The obvious reading was "it's still encoding, the rest will appear". That was wrong. Nothing in our player shows a half-finished encode: the stream from Part 6 is only published once every quality is done. So something else was playing, and something else had stopped early.

It took an afternoon and most of the night to untangle, and the next morning the cloud bill showed what it had cost. There turned out to be four separate bugs, one machine doing too much at once, two red herrings, a few mistakes of our own, and one expensive day. This post goes through all of them, and what we changed for each.

Step one: find out what is actually playing

When a stream isn't ready yet, the player falls back to plain MP4 files. For each uploaded video, a background task makes two smaller copies, 1080p and 720p, called proxies, so there's something quick to play while the real stream is prepared. Those proxies were what we were watching.

Listing the film's files in storage made the problem obvious:

A full hour of 1080p is well over a gigabyte. These were a few percent of that. But the timestamps were the real clue: each proxy finished exactly 15 minutes after the previous task ended. Bugs that stop at a round number usually hit a limit somebody set.

Bug 1: the link that expired mid-encode

Our storage bucket is private, so ffmpeg reads the original through a signed URL, a link that works only until it expires. The proxy code reused the lifetime set for thumbnails, where ffmpeg grabs one frame in a few seconds:

private static final Duration VIDEO_SOURCE_URL_TTL = Duration.ofMinutes(15);

Fifteen minutes is plenty for a thumbnail. A proxy of an hour-long film is a different job, and ours ran on a background thread, which on Cloud Run gets almost no CPU once the request that started it has returned (we hit that in Part 8). At that speed, 15 minutes produced 45 seconds of 1080p and 5 minutes of 720p.

Then the link expired. ffmpeg doesn't read a big file in one go: it asks storage for pieces as it needs them. The next request after the expiry was refused, and ffmpeg treated the failed read as the end of the film. It finished the file neatly and exited with success. Our code checked the exit code, saw zero, and saved a 45-second file as the 1080p version of a one-hour film.

Fix. Three changes, because one would only have moved the problem:

• The link outlives the encode. A proxy's link now lasts longer than the encode's own 30-minute time limit, so it can't expire first.
• Check the result, not the exit code. Before saving, we compare the proxy's duration with the original's. Anything shorter than 98% is thrown away. The widget's "Check length before saving" button shows what that buys.
• Long films don't get proxies. Even with a working link, a background thread can't encode an hour of video in 30 minutes, and the output is held in /tmp, which on Cloud Run is RAM. Anything over 20 minutes now skips proxies and plays through the adaptive stream. (A day later we stopped making proxies altogether; the bill below explains why.)

Lesson. A zero exit code means the program didn't crash. It doesn't mean the program did what you wanted. When the output has a property you can measure, like a video's length, measure it.

Bug 2: the queue that doubled itself

With the proxies explained, the next question was why the real stream still wasn't ready hours later. The job queue answered it:

gcloud tasks list --queue=therare-media-jobs --location=asia-southeast1

About 170 jobs were waiting, and almost all of them showed zero attempts. A large batch of raw footage had been uploaded that morning, and our queue runs one encode at a time, on purpose, so encodes never compete with the website for CPU. The film was at the back.

Worse, the film's jobs were in the queue twice, queued exactly two hours apart. Here's why. To stop two servers queuing the same encode, the first one claims the video in the database. If a job gets lost, the claim would block the video forever, so a claim older than two hours counts as dead and anyone may claim it again:

public static final Duration CLAIM_STALE_AFTER = Duration.ofHours(2);

The two hours count from when the job was queued, not from when it started. In a long queue, a job can wait more than two hours without ever starting. When someone opened the film after that, the code decided the job was lost and queued all four qualities again. Every two hours it would have added four more hour-long encodes to a queue that was already hours behind.

Fix. Cloud Tasks can refuse duplicates for us. Each task now gets a fixed name built from what it does:

String key = job.type() + "|" + job.kind() + "|" + job.assetId() + "|" + job.flavour()
        + "|" + job.rendition() + "|" + job.objectKey();
// SHA-256 of the key, first 16 bytes as hex, becomes the task name

Queuing the same job twice now gets a 409 ALREADY_EXISTS, which we treat as "already queued" and move on. One catch: Cloud Tasks keeps a name reserved for a while after the task has run. So on a 409 we ask whether the task is still waiting. If it isn't, this is a deliberate re-run, like an admin pressing Rebuild, and it's queued under a fresh name. That check needs the cloudtasks.viewer role, which the service account didn't have. If the check fails, the code falls back to the old behaviour rather than blocking the encode.

Lesson. A timeout meant to detect lost work has to measure the work, not the wait. And if a queue can deduplicate, let it: the queue knows what it holds better than a database column does.

Bug 3: all or nothing, against a 30-minute wall

The film's stream has four qualities: 1080p, 720p, 480p and 360p. Each one is its own job, so each job has to finish within Cloud Tasks' 30-minute limit. That's fine for clips, but an hour of 1080p from a 13 Mbps source has to be encoded at about twice real time, while also decoding the source. On two CPUs that's unlikely.

Three details turned "1080p might fail" into "the film never plays":

1. The stream was all or nothing. It was published only when every quality had finished. If 1080p failed, the three qualities that did finish were never shown.
2. ffmpeg's time limit was exactly the request's. Both were 30 minutes, so Cloud Run killed the request before our code could notice the timeout. Nothing was recorded, the film sat on "encoding", and two hours later the claim went stale and the whole thing started again.
3. A timeout was retried like any other error. Three attempts of half an hour each, all failing the same way, while the one-at-a-time queue waited behind them.

Fix.

• Publish what finished. A quality that fails for good leaves a small .failed marker next to the stream. Once every quality has either finished or failed, the stream goes out with the ones that worked, so the film plays in 720p, 480p and 360p. It's only marked failed if every quality failed.
• Time out before the deadline. ffmpeg now stops at 28 minutes, which leaves two minutes to record the failure and publish the stream.
• Don't retry a timeout. The same file on the same hardware won't be faster next time. Other errors, like a dropped connection, still retry.

Lesson. Decide what "partly done" should look like before production decides for you. For a video, three qualities out of four is a working product. Zero out of four is an outage.

Bug 4: a 720p that was "ready" at 12 minutes

With partial publishing in place, the obvious shortcut for this film was to publish by hand the qualities that had already finished: 720p and 360p. Before doing that, we listed the files:

720p should be the bigger file, not less than half of 360p. A playlist has one entry per 4-second piece, so its size tracks the video's length. The 720p playlist was about a sixth of the 360p one: roughly 12 minutes of a one-hour film, marked "ready".

It was Bug 1 again, in a different place. The stream encoder's link to the original lasts an hour, so it didn't expire. The most likely cause is that the connection to storage dropped part way through a long read; we can't prove it from the logs, because ffmpeg reported nothing wrong. ffmpeg treated the failed read as the end of the file, exited with success, and the code saved a 12-minute rendition as complete. Publishing it would have given the customer a film that stops 12 minutes in, in the best quality on offer.

Fix. The same two ideas as the proxies, applied to the stream:

• Reconnect instead of stopping. ffmpeg reads the source with -reconnect 1 -reconnect_on_network_error 1 -reconnect_delay_max 30, so a dropped connection resumes the read instead of ending the film.
• Check every rendition's length. After each encode, its duration is compared with the source's. A rendition more than 2% short fails the attempt and is retried, instead of being published.

Lesson. When you fix a bug, look for the same pattern elsewhere in the code. We'd fixed "ffmpeg's success doesn't mean the output is complete" for the proxies an hour earlier, and the stream encoder had exactly the same blind spot.

The real bottleneck: sharing a machine

The logs held one more surprise. The quality that failed for good wasn't 1080p. It was 480p, the second-smallest, which timed out while 720p had finished comfortably earlier. A smaller output taking longer made no sense until we lined up the timestamps: 1080p and 480p kept failing within a minute or two of each other. They were running at the same time, on the same instance, together with the duplicate jobs from Bug 2, and splitting its CPU. The 720p that finished had had the machine to itself.

Cloud Run sends several requests to one instance by default, which is exactly right for a website and exactly wrong for video encoding. Our encodes were also sharing that instance with the website itself.

Fix. A separate Cloud Run service for encoding, running the same image as the backend:

• One request per instance (containerConcurrency: 1), so every encode gets a whole machine.
• 4 CPUs and 8 GiB, since /tmp is RAM and holds a whole rendition (an hour of 1080p is about 2.5 GB).
• Up to 3 instances, matching the queue's concurrency, so three encodes run side by side without touching each other.
• Scales to zero. With request-based billing it costs nothing while idle, and about $0.42 per hour of encoding at list prices. Encoding costs roughly the same CPU-seconds on a big machine as a small one; the big one just finishes sooner.

The backend only needed one setting to use it: the URL its job queue sends work to. A script copies the backend's configuration (image, environment variables, secret references) into the encoder, so the two can't drift apart, as long as someone runs it.

Lesson. Before concluding a job is too big, check whether it's alone. A timeout on a shared machine tells you about the machine's load as much as about the job.

Rolling it out: four more slips

Changing a live system mid-incident produced its own small problems. None did damage, but each one is easy to repeat:

1. Old jobs remember where they're going. Cloud Tasks stores the destination URL inside each task when it's created. We raised the queue to three encodes at once right after creating the encoder, but the 170 jobs already waiting still pointed at the backend. Three encodes at once on the website's machine was exactly what we were trying to stop. We put the queue back to one until the old jobs drained.
2. The task list hides the task body. Our loop to find and delete the film's old jobs matched nothing. gcloud tasks describe returns a basic view that leaves out the request body, which is where the job description lives. It needs --response-view=full.
3. A deploy interrupts running encodes. Deploying the fix stopped the two encodes that were running at that moment. One was on its last attempt, so it was recorded as failed for good with InterruptedException. Deploy between long encodes, not during them.
4. The encoder is a copy, not a link. Because it's a separate service, deploying the backend doesn't update it. Our runbook now says, in bold, to re-run the sync script after every backend deploy, and to compare the two services' image digests when in doubt.

The small one: a missing line

While reading logs we found an unrelated failure on a raw clip: No STREAM-INF for 720p. ffmpeg writes a one-line summary of each quality (bitrate, resolution, codecs) and we copy it into the stream's master playlist. For one file it didn't write the line, even though the encode worked.

We never pinned down which property of that file caused it. The fix doesn't depend on knowing: if the line is missing, we build it ourselves from the quality's bitrate cap and a quick ffprobe of the encoded file. While there, the encoder now also skips cover images that some cameras store as the first "video" track, so it always encodes the actual footage.

Two red herrings

• "Broken pipe", every few minutes. These were loud and filled the error log. All of them came from the database driver opening new connections to our serverless Postgres, which pauses itself when idle. The connection pool retries on its own. It was noise, not the cause.
• Encode jobs failing with 503. It looked like Cloud Run falling over. It was our own code: when an encode job throws an error, the endpoint deliberately returns 503, because that's how you tell Cloud Tasks "retry this later". The real error was a warning two lines earlier in the log. Filtering on severity>=ERROR hid it.

Lesson. Before you chase an error, check who wrote it. Half of today's scary log lines were our own code working as designed.

Our mistake: deleting the wrong video's files

With the cause found, the plan was to delete the short proxies and let them be rebuilt properly. We'd seen that the film's jobs were queued at about the right time, assumed which database record was the film, and deleted that record's proxies.

It was a different long film from the same event. Its original was untouched, since we'd only removed derived copies, and those can always be rebuilt from the original, so nothing was lost. But we did delete files we hadn't meant to, and it was avoidable. The right record was one query away:

SELECT id, object_key, proxy_1080_object_key, proxy_720_object_key
FROM media_assets WHERE object_key ILIKE '%<file name>%';

Lesson. Before anything destructive, find the target by name, list it, check that the sizes and dates match what you expect, and only then delete it. And change storage and database together: a deleted file whose path is still in the database is a broken link in the player.

The bill: ₹685 in one day

The next morning, Cloud Run's line on the billing report for that single day was ₹685. On a normal day it's a small fraction of that. Our estimate for all video encoding had been ₹1,500–3,000 a month.

First question: is something billing while idle? On Cloud Run that's the classic surprise, and it comes from two settings: a minimum number of instances, which keeps machines running around the clock, or "CPU always allocated", which bills CPU even between requests. One command answers it for every service:

gcloud run services describe therare-backend --region=asia-southeast1 \
  --format='value(spec.template.metadata.annotations["autoscaling.knative.dev/minScale"],
                  spec.template.metadata.annotations["run.googleapis.com/cpu-throttling"],
                  spec.template.spec.containers[0].resources.limits)'

All three services scale to zero, and CPU is billed only during requests. No leak. So the ₹685 was real work.

Second question: where did the time go? Cloud Monitoring has a metric for exactly this: billable container instance time. Charted hourly, each value is the average number of machines being paid for during that hour:

About 20 machine-hours in total, at roughly ₹35 each (a mix of 2-CPU backend and 4-CPU encoder machines). The shape tells the story. For six hours, one machine worked through the one-at-a-time queue. Then for three hours, three or four machines ran side by side, and those three hours were half the bill. That's the window when the queue was at three while the old jobs still pointed at the backend, the duplicate film jobs were running, and the new encoder was starting up.

The same chart had one more surprise: five days earlier there was an identical plateau, one machine for 16 hours straight, about 13 machine-hours. Another booking's raw footage going through the queue. Nobody had noticed, because a single day costing a few hundred rupees doesn't stand out on a monthly invoice. So this wasn't only a bad day; it was also the normal cost of a big upload.

Here's what those machines were doing:

Most of the spike was waste the earlier fixes had already stopped: timeouts are no longer retried, duplicates are refused by the queue, and short outputs aren't published and then re-encoded. The baseline, one machine for most of a day per big upload, needed four more changes.

Fix.

• Stop making proxies. Once every video has an HLS stream, the 1080p and 720p MP4 copies duplicate its work. We removed them: roughly 40% less encoding per video. Videos that already have them keep playing them.
• Fewer qualities for raw footage. Raw clips exist to choose takes; anyone who needs full quality downloads the original. Their streams dropped from 720p, 480p and 360p to just 480p and 360p, roughly halving raw-clip encoding again. Delivered films keep all four qualities.
• Right-size the machines. The encoder went from 4 CPUs / 8 GiB to 2 CPUs / 4 GiB with at most two running. With encodes no longer fighting over one machine, that's enough, at about half the hourly price. The backend goes back to one CPU once the old queue drains, since it only serves the website now.
• A budget alert. A monthly budget with email alerts at 50% and 90%, so the next day like this one is noticed that day, not a week later.

Lesson. On pay-per-request infrastructure, a timeout is a bill. Every failed 30-minute attempt was a machine rented for 30 minutes to produce nothing. And when a bill looks wrong, look at machine-time, not rupees: "three to four machines for three hours" pointed straight at parallel work, where "₹685" alone pointed nowhere. The same chart also exposed a cost we'd been paying quietly every time a booking was uploaded.

What's still left

These fixes stop the bad outcomes: short copies, duplicate jobs, films that never play, encodes fighting over one machine, and paying for all of it. A large raw-footage upload that cost ₹300–450 in encoding should now cost around ₹150–250, and we'll check that against the same chart after the next one. A dedicated 4-CPU machine should get even an hour of 1080p inside the 30-minute limit. If it doesn't, the film still publishes without it, and the next step is ready:

• Long films as Cloud Run Jobs, which can run for hours instead of 30 minutes.
• Or split the film into chunks, encoded side by side and joined at the end.

Takeaways

• Find out what is actually playing before you decide what's broken.
• A zero exit code is not proof of success. Measure the output.
• Credentials that expire must outlive the slowest job that uses them.
• Measure "stuck" from when work started, not from when it was queued.
• Let the queue deduplicate by giving tasks stable names.
• Decide what partial success looks like, and publish it.
• Keep your own timeouts inside the platform's, so you get to handle them.
• Confirm a target by name before deleting anything.
• When you fix a bug, search for the same pattern elsewhere.
• Check whether a slow job was alone before calling it too big.
• Queued work remembers where it was sent; moving the destination only affects new work.
• On pay-per-request platforms, failures cost money too. Don't retry what will fail the same way.
• Read a surprising bill as machine-time (Cloud Run's billable instance time), and check for anything billing while idle first.
• Don't do the same work twice. Retire the old path when the new one covers it.