Personal finance aggregator: Keeping data fresh: manual sync, resilience, and real stats - Part 7

~10 min read

Everything up to this point assumed the nightly sync just works. It mostly does - right up until one bank connection has a problem, at which point the version of fetch() I'd had since the very first CLI prototype turned "one bank's data is temporarily unavailable" into "nobody's data updates tonight." This post is about the sync loop growing up: a manual trigger for when 3am is too long to wait, failure isolation so one bad connection can't take the rest down with it, error messages that actually say what went wrong, and turning the dashboard's stat tiles from hardcoded placeholder numbers into real SQL.

A button, not just a timer

The nightly systemd timer covers "keep data current," but not "I just linked a new bank and want to see its transactions right now" or "I suspect something's stale and want to check." Adding a manual Sync now button to the Accounts page meant the same fetch() function the timer calls needed to be safely callable from an HTTP request instead of only from a CLI process that's allowed to just exit.

That surfaced an actual bug immediately. The original fetch() bailed out with sys.exit(1) when no banks were linked - completely fine for a script, where sys.exit just ends the process. Called from inside a FastAPI route handler, sys.exit raises SystemExit, which isn't a normal Exception - it's the same exception class Python uses to actually terminate the interpreter, and it's not reliably something a web framework's error handling will catch cleanly. The fix was a proper exception the API layer could catch on purpose:

class NoLinkedBanks(Exception):
    """Raised by fetch() when no banks are linked yet - callers (the CLI,
    the manual-sync API route) decide how to surface that."""

The CLI's __main__ block catches it and calls sys.exit(1) itself, at the actual process boundary where that's the correct behaviour. The API route catches it and returns a 400 with a message the frontend can show. Same underlying condition, two completely different "this is how a caller should be told about it" - which is really the whole argument for not baking process-termination into a function you also want to call as a library.

One bank's problem shouldn't be every bank's problem

The bigger issue only showed up once I actually watched fetch() fail against real data: it looped over every linked bank, and any single requests.HTTPError - a bank whose consent had lapsed, an account TrueLayer returned a 403 for - propagated straight out of the whole function. Three linked banks, one having a bad day, and none of them got new transactions that run. That's a much worse failure mode than "this one bank is stale" - it makes reliable data quietly depend on the least reliable connection you have.

The fix was pushing try/except down to the smallest unit that can fail independently - token refresh, listing accounts, and each account's own transaction pull - and collecting failures instead of raising past them:

for kind, account_id, name in sources:
    try:
        txns = get(access, f"/{kind}/{account_id}/transactions", date_range, logger=logger)
    except requests.RequestException as exc:
        logger.error(f"{label} / {name}: could not fetch transactions - {exc}", exc_info=True)
        errors.append({"bank": label, "account": name, "error": f"could not fetch transactions ({exc})"})
        continue
    ...

continue instead of letting the exception bubble is the entire fix. One account failing means that account doesn't get new transactions this run - and every other account, on every other bank, still does. fetch() now returns a summary including whatever went wrong ({"transactions": N, "connections": N, "errors": [...]}), and the manual sync endpoint surfaces those as warnings alongside an otherwise-successful result, rather than the whole request just failing with one opaque error.

Knowing why, not just that, something failed

The error messages I was logging before this were nearly useless for actually diagnosing anything: "403 Client Error: Forbidden for url: ...transactions" and nothing else. TrueLayer's actual error body carries a specific code that explains what's going on - access_denied (consent's expired or been revoked, needs re-linking) versus sca_exceeded (some providers only allow pulling transaction history for a short window right after the user authenticates, and a 90-day pull outside that window gets rejected even with an otherwise-valid token) - and raise_for_status() throws that information away before I ever saw it.

if not resp.ok:
    # TrueLayer's error body carries a code (e.g. access_denied,
    # sca_exceeded) that the status alone doesn't - log it before
    # raise_for_status() reduces it to just "403 Client Error: ...".
    logger.error(f"TrueLayer {resp.status_code} on {path}: {resp.text}")
resp.raise_for_status()

One line, logged before the exception gets raised. (A few days later the detail stopped being log-only: a follow-up turned it into a real TrueLayerAPIError carrying the error code, so the UI could tell a revoked consent from sca_exceeded - that's in post 17.) It's a small change with an outsized effect on how it feels to operate this thing - the difference between "something's wrong with Monzo, no idea what" and "this specific account's consent expired, it needs re-linking" is the difference between a mystery and a two-minute fix.

Surfacing partial failure without crying wolf

Getting the backend to isolate failures was only half the problem - the sync button also had to show a partial failure honestly, without either hiding it or treating it as catastrophic. A sync where two banks updated fine and one hit a sca_exceeded error isn't a failed sync; it's a mostly-successful one with a specific, actionable warning attached, and the UI needed to say exactly that:

setSyncMessage(
  `Synced ${result.transactions_inserted} new transaction(s) across ${result.banks_synced} bank(s).`,
)
setSyncWarnings(result.errors)

A green success line for what worked, sitting directly above an amber warning list for what didn't, in the same result - not a single pass/fail toast that would have to pick one or the other and misrepresent whichever it didn't pick. The button itself walks through three states while this happens (Sync now → Syncing… → back to Sync now), each one covered by a test that stubs a slow-resolving promise and asserts the button's disabled and relabelled for exactly the right window - a small thing, but "did the button actually enter its loading state" turned out to be an easy regression to introduce silently while refactoring the surrounding component, and cheap to catch with a test that just waits on a manually controlled promise before letting it resolve.

A de-dup key that could move under me

Separately, and worth a short mention here because it's the same class of lesson - read what the vendor's docs actually promise, not just what the field looks like it means - I'd been de-duplicating transactions on TrueLayer's transaction_id since the very first prototype. Their docs describe that field as one that may change between requests, and recommend normalised_provider_transaction_id instead as the one that's stable. My primary key relied on exactly the property TrueLayer explicitly doesn't guarantee. The fix was a one-line change with a fallback for providers that don't return the normalised field:

"id": t.get("normalised_provider_transaction_id", t["transaction_id"]),

Nothing had gone wrong yet when I made this change - which is exactly why it was worth making before it did.

Real numbers instead of placeholders

The Home page's stat tiles started as hardcoded demo values, which was fine for getting the layout right but obviously not the point. Income and spending "this month" turned into an actual SQL aggregate over the transactions table:

SELECT
    COALESCE(SUM(CASE WHEN amount > 0 THEN amount END), 0) AS income,
    COALESCE(SUM(CASE WHEN amount < 0 THEN -amount END), 0) AS outgoings
FROM transactions
WHERE date >= :start AND date < :end

Run once for the current calendar month and once for the previous one, and the difference between the two gives the stat tiles their "up/down vs last month" delta - the same query, just two different date ranges, rather than two different code paths. (Calendar months didn't last: once categorisation meant totals had to exclude transfers, a later commit replaced this SQL with a Python summarize() that runs after categorisation, and another swapped calendar months for rolling 30/60/90-day windows. See post 12 and post 13.) Consent expiry got the same treatment on the Accounts page: TrueLayer's /me endpoint returns consent_expires_at for a connection, cached alongside everything else, and rendered as "Expires in N days" or a harder "Needs re-linking" once it's past.

Picking thresholds for the expiry badge

The consent expiry badge needed actual thresholds, not just a raw date, because "expires 2026-11-14" on its own doesn't tell me whether that's something to act on today or something to ignore for two months. I settled on three states: comfortably far away (no badge, nothing to see), inside fourteen days (an amber "Expires in N days"), and past expiry (a red "Needs re-linking"). Fourteen days came from wanting enough lead time to actually notice and act before a sync silently starts failing for that bank - shorter than that and there's a real chance I don't look at the Accounts page in the window between "worth mentioning" and "already broken." It's a judgement call, not a number TrueLayer prescribes, and one I'd tighten if a re-link ever turned out to need more notice in practice than two weeks gives.

What "fresh" actually means now

None of this changed what data the app pulls - it's the same TrueLayer calls as the very first prototype. What changed is what happens when one piece of that data has a bad day: it used to mean nothing updated and you found out from a stale dashboard; now it means everything else still updates, the one thing that failed says specifically why, and you can trigger a re-check whenever you want instead of waiting for 3am. That's the actual bar for "keeping data fresh" on a tool you're meant to trust every morning.

The last piece was making that trust more complete: showing what's actually in each account, not just what's been spent from it - and giving myself a clean way to disconnect a bank once I stopped needing it.

Previous: ← Auth, two-factor, and account settings · Next: Account balances and disconnecting banks →