Why write about bugs
Tutorials show code that works. Real engineering is mostly finding out why code that looks right doesn't. Each bug below cost us somewhere between an hour and a full day. None of them was clever; all of them were confusing. We keep every one of them written down in our deployment guide, so nobody loses that day twice. Here they are, so you don't lose it either.
1. The invisible newline in the password
Symptom. After storing the database password in Google's Secret Manager, the API couldn't connect. Postgres rejected the login with "Prohibited character". Later, email failed the same way with "535 Authentication Failed". The passwords were definitely right.
Cause. We created the secrets like this:
echo "the-password" | gcloud secrets create db-password --data-file=-echo adds a newline at the end. So the stored secret was the password plus an invisible line break, and that's what was sent to the database.
Fix. Write secrets without a trailing newline, for example printf '%s' "the-password", or from a file saved without one.
Lesson. When a value is "definitely right" but rejected, check the bytes, not the characters you can see. cat -A or xxd shows hidden whitespace in a second.
2. Emails that failed only after deploys
Symptom. Booking confirmation emails sometimes failed with "Connect timed out". Then they'd work for hours. It seemed random. A whole day went into suspects: the network connector, the email provider blocking us, port 587 versus 465, timeouts from 10 to 30 seconds.
What finally broke it open was noticing when it failed: right after a deploy. Two separate causes were stacking up.
- Cold starts. The first request to a freshly deployed Cloud Run container arrives while Spring is still starting up with limited CPU. Opening an email connection took longer than the timeout. Minutes later, it worked. We turned on Cloud Run's start-up CPU boost and raised the timeouts.
- Shared outgoing IP addresses. Each Cloud Run instance sends traffic from an address in a shared Google pool, and our email host blocked some of those addresses. Every deploy creates new instances with new addresses, so failures clustered after deploys. There was no fix on our side.
Fix. We stopped using SMTP. Email now goes out through an email provider's HTTPS API. Outgoing HTTPS on port 443 is never what gets blocked. The old SMTP code is still there behind a setting, but we don't use it.
Lesson. When a bug looks random, record exactly when it happens. "Random" failures usually line up with something: a deploy, a time of day, a particular user. The pattern is the clue.
A related trap from the same week: port 465 expects encryption from the very first byte, while port 587 starts in plain text and upgrades (STARTTLS). Configure one while using the other and the client waits forever for a greeting that never comes, which looks exactly like the cold-start timeout above.
3. A 528 MB upload for a small website
Symptom. Deploying the website took ages to even start.
Cause. gcloud builds submit uploads the entire folder to Google's build service unless told otherwise. That included node_modules: 528 MB of packages that the build installs again anyway.
Fix. A .gcloudignore file, which works like .gitignore, in both the frontend and backend folders. We added a note saying never to delete it.
Lesson. Know what your deploy command actually sends. Most tools have an ignore file; check it exists.
4. The sitemap that pointed at the wrong website
Symptom. Our pages weren't showing up in Google the way we expected.
Cause. The sitemap, the file that tells Google which pages exist, listed every page under an old placeholder domain instead of therarevisuals.in. The code read the domain from an environment variable with a fallback:
const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || "https://www.therare.in";In Next.js, variables starting with NEXT_PUBLIC_ are baked in when the app is built, not read when it runs. Our Docker build never passed that variable in, so every production build silently used the placeholder. The site worked perfectly for humans. Only crawlers saw the wrong domain.
Fix. Pass the variable as a Docker build argument, and make the fallback the real domain.
A second bug in the same file. Each page's "last modified" date was new Date(), evaluated on every request, so every page claimed it had changed at the exact moment Google looked. Google ignores last-modified dates that look unreliable, and "everything changed just now, every time" is as unreliable as it gets. Now the date is set once when the server starts, which is when the pages actually change.
Lesson. Some bugs are invisible to people and only visible to machines. Check what search engines, social previews and crawlers actually receive, not just what the page looks like.
5. The background job that barely ran
Symptom. Video encoding, started in a background thread after replying "queued", crawled along or stalled.
Cause. Cloud Run gives a container CPU while it's handling a request. After the response is sent, CPU is throttled to almost nothing. Our background thread was starving.
Fix. Every job became its own HTTP request, delivered by a Cloud Tasks queue (Part 4 has the details).
Lesson. Serverless platforms make assumptions about how you'll use them. Read the section of the docs about CPU and memory limits before designing background work.
6. A typo shouldn't take the business down
This one isn't a bug we hit; it's one we designed out after the email saga. The email transport is chosen by a setting. If someone mistypes it, Spring would normally fail to start, and a mistyped email setting would stop customers booking and uploading.
Now a fallback takes over: it logs a loud warning at start-up and fails only the email sends. Everything else keeps working.
Lesson. Ask of every configuration value: if this is wrong, what's the smallest thing that should break? Make that the only thing that breaks.
7. The regions don't all have the same features
We built in the Mumbai region, closest to our customers. Then we found that Cloud Run's custom-domain mapping isn't available there. Everything moved to Singapore.
Lesson. Before choosing a region, list the features you need and check each one is offered there.
How we debug now
The same habits solved almost every problem above:
- Find when it fails, not just that it fails. After a deploy? Only on phones? Only for big bookings?
- Check the actual bytes. Hidden newlines, wrong domains in a generated file, headers the browser really received.
- Question the platform, not just your code. CPU throttling, region features, shared IP addresses and build-time variables are all platform behaviour.
- Change one thing at a time, and write down what you tried.
- Write it down when it's fixed. Every lesson here is in our deployment guide, next to the command it affects.
Thank you for reading
That's the end of the series: the architecture, React from the ground up, pagination and memory, backend design, the media pipeline, our own video player, security, and the bugs. Everything we've described was built by a small team for a real business with real customers. You don't need a big company to do serious engineering. You need a real problem, curiosity about why things break, and the habit of fixing them properly.
Pick a problem you can see from where you are, and build something useful. If you do, we'd love to hear about it.