Building Shootlo: what broke while we built a same-day marketplace

Why this post

Shootlo is our second product: book a verified photographer near you, for the same day. We've written the design down in the Shootlo engineering notes. This post is the other half: the things that broke while we built it, in the order they hurt. Every one of them looked like a mystery at first. None of them was one once we found it.

If you're building a two-sided marketplace (customers on one side, people doing the work on the other), most of these will reach you eventually.

1. An error that appeared right after a harmless migration

Symptom. Minutes after a deploy, the photographer's jobs page started failing with 500s every few seconds. So did the payouts screen. Other pages were fine. Refreshing sometimes worked.

The log said:

ERROR: cached plan must not change result type
SQL [SELECT * FROM bookings WHERE photographer_id = ? AND status = 'PREVIEWS_SENT' ...]

Cause. The deploy included a migration that added five columns to bookings. Nothing in it was wrong. But the Postgres driver prepares queries it runs often: after a few runs it asks the server to remember the query plan, including the shape of the result. SELECT * on a table that just gained columns now returns a different shape, and the server refuses to reuse the old plan.

Normally a new deploy means new connections, and new connections have no old plans. Our database sits behind a connection pooler (PgBouncer style) that keeps server connections open across our deploys. The stale plans survived, and any request that happened to land on one of those connections failed. That's why it looked random.

Fix. Stop keeping server-side prepared statements, which is the standard setting behind a transaction pooler:

spring:
  datasource:
    hikari:
      data-source-properties:
        prepareThreshold: 0

Restarting the database compute cleared the stale plans straight away. The setting stops it coming back after the next migration.

Lesson. Anything that outlives a deploy (a pooler, a cache, a CDN) can hold on to the old version of your schema. When an error appears right after a migration and only on some requests, ask what didn't restart.

2. The scheduler that was locked out by our own security

Symptom. Offers stopped expiring and unpaid bookings stopped timing out. The scheduled endpoint that does that work returned 401 every minute.

Cause. Cloud Run throttles CPU between requests, so a background timer inside the app doesn't run when nobody is using it. We moved the timer out: Cloud Scheduler calls /internal/tick every minute, with a Google identity token so only it can reach the API.

But our API is a resource server that treats every Authorization: Bearer ... header as a Firebase sign-in token. The scheduler's Google token arrived, the Firebase check rejected it (wrong issuer), and Spring answered 401 before our own shared-secret check even ran.

Fix. Don't try to read that token as a user at all. The internal path is protected separately (Cloud Run IAM plus a shared secret), so the bearer resolver skips it:

var bearer = new DefaultBearerTokenResolver();
http.oauth2ResourceServer(o -> o
    .bearerTokenResolver(request ->
        request.getRequestURI().startsWith("/api/v1/internal/") ? null : bearer.resolve(request))
    .jwt(...));

Lesson. When two systems both use Authorization: Bearer, decide explicitly which one owns each path. "Any token is a user token" is fine until something that isn't a user calls you.

3. The AI reviewer that ran the API out of memory

Symptom. New photographer applications are pre-read by an AI assistant: it looks at their portfolio and kit photos and writes notes for the person who makes the decision. On a real application it failed with "something went wrong", and Cloud Run's logs said the container had been killed for using more than its 1 GiB.

Cause. Phones take 12 to 50 megapixel photos. We sent the originals to the model as base64, and base64 is a third bigger again. Several of them, repeated on each round of the conversation with the model, and a decoded image in Java takes about 4 bytes per pixel. One 48 MP photo is roughly 190 MB once decoded. Three of them and a 1 GiB container is gone.

Fix. The model doesn't need 8,000 pixels to judge focus, light and composition. We shrink every image to 1024 px on the long edge before sending it, and we decode it already subsampled, so the full-size picture never sits in memory:

int step = Math.max(1, Math.max(width, height) / (maxEdge * 2));
param.setSourceSubsampling(step, step, 0, 0);   // read every Nth pixel
var decoded = reader.read(0, param);
return jpeg(fit(decoded, maxEdge), 0.82f);

A 24 MP test photo now becomes a 20 KB JPEG and runs in a 256 MB heap. We also told the JVM how much of the container it may use (-XX:MaxRAMPercentage=70) and turned on headless mode, since there is no screen.

Lesson. Before sending anything to a model, ask how big it needs to be. It's usually much smaller than what the user uploaded.

4. Thirteen characters in a twelve-character column

Symptom. Sending an announcement to photographers failed with a 500. Sending one to customers worked.

Cause.

ERROR: value too long for type character varying(12)

The audience column was sized for CUSTOMERS (9 characters). PHOTOGRAPHERS is 13.

Fix. A migration widened the column. The fix took one line. Finding it took longer, because the UI only said "something went wrong".

Lesson. When a column stores an enum's name, size it from the longest name, or use TEXT with a check constraint. And make ops screens show the server's error message, not a generic apology: it would have turned half an hour into two minutes.

5. "Use my current location" did nothing on iPhones

Symptom. On iPhone Safari, tapping "Use my current location" did nothing. No permission prompt, no error.

Cause. Safari only shows the location prompt when the request comes directly from a tap. Our button did a little work first (awaited something, then asked for the location), which is enough for Safari to decide the user didn't really ask. Once location has been denied for a site, Safari also stops asking and fails silently.

Fix.

• Call navigator.geolocation.getCurrentPosition synchronously inside the tap handler, before any await.
• If a high-accuracy fix times out, retry once at low accuracy (indoors, GPS may never lock).
• If permission is denied, show step-by-step instructions to switch it back on, instead of nothing.

Lesson. Test permission flows on a real iPhone. Desktop browsers are much more forgiving about what counts as "the user asked".

6. The fake numbers we showed to customers

Symptom. Every new photographer appeared with ★ 5.0 and 80% accepted. Nobody had rated them yet.

Cause. Dispatch ranking needs a starting value for new photographers, or they'd never be offered anything. We gave them sensible defaults, and then the same field was shown to customers as if it were real.

Fix. Keep the defaults for ranking only. Customers see "New on Shootlo" until a photographer has 3 ratings, and the acceptance rate appears after 5 answered offers.

That created the next bug report within a day: "the rating didn't update after the customer rated". It had; it just wasn't visible yet. Now the photographer sees their own progress, "1 of 3 ratings · ★ 4.0 so far", while customers still see "New".

Lesson. A number used by the algorithm and a number shown to people are two different things, even if they're stored in one column. And if you hide something on purpose, say so, or it looks broken.

7. "Delivered" was a door that locked behind the photographer

Symptom. Once a photographer marked a shoot delivered, it disappeared from their screen. If they'd forgotten ten photos, there was no way back. Meanwhile, customers saw only the final files; the previews they'd been sent earlier had vanished.

Cause. We treated delivered as the end of the job. It isn't. The job ends when the customer agrees they have everything.

Fix. A second step, owned by the customer:

• After delivery, the customer is asked "Got all your photos?", with Yes, I got everything or Something's missing (and a note).
• Until they confirm, the photographer keeps a "Delivered · waiting for the customer" card and can add or change files.
• The payout follows the answer: confirmed means it's due now; no answer means it's due 48 hours after delivery; a reported problem holds it until ops marks it sorted. Both ops and the photographer are emailed.

We found a sibling bug while fixing this one: the photographer's screen said "Finish shoot, send previews later", but the API refused to finish a shoot without previews. The rule lived in two places and they had drifted apart.

Lesson. Model the end of a job from the customer's side. And keep each rule in exactly one place, with the UI reading it rather than restating it.

8. From one city to all of India

Symptom. Not a bug, a limit. We launched with one city and three rings around its centre (instant, later today, scheduled). People elsewhere couldn't book, and photographers who travelled for work got offers for the city they'd left.

Fix.

• Any point in India can be booked. "Now" bookings stay limited to the inner ring of cities with a local team; everywhere else is later today or another day. Customers pay only after a photographer accepts, so a booking nobody takes costs nothing.
• Photographers choose where they're working. For shoots later in the day, distance is measured from that place. For "now" shoots, it's measured from a GPS fix no older than 15 minutes. A photographer heading to Jaipur next week can set it before they go.
• Staff can be scoped to a state or a city, and every booking records both, so a city manager sees only their city.

Lesson. Hard-coding "the city" is the fastest way to launch and the first thing you'll undo. Store the city and state on every booking from day one, even when there's only one.

9. Two smaller ones worth knowing

• The lock file that only broke in the cloud. Running npm install on a Mac rewrote package-lock.json and quietly dropped some Linux-only optional packages. Everything worked locally; the build server's npm ci failed. We now regenerate the lock file on Linux (npm install --package-lock-only) and treat unexpected lock-file diffs as a red flag in review.
• Phone numbers stay private. Customers asked to call their photographer on the day. We deliberately hide both numbers: once a photographer has a customer's number, the next booking tends to happen off the platform. The plan is chat and push notifications during an active booking, then masked calling. Until then, both sides can see each other's live progress and call support.

What we'd tell ourselves at the start

1. Log the real error, and show it to your own team. Half of these took longer than they should have because a screen said "something went wrong".
2. Assume something survives your deploy. Poolers, caches, open tabs, old service workers.
3. Budget memory per request, not per server. One upload can be hundreds of megabytes once decoded.
4. One rule, one place. Every time the UI and the API each had their own copy of a rule, they disagreed eventually.
5. Don't show numbers you invented. Say "New" and let the real numbers arrive.

The Shootlo engineering notes keep the current design, what's still open, and a changelog, so you can follow how each of these turned out.