Personal finance aggregator: Making the nightly sync boring - Part 17

Personal finance aggregator: Making the nightly sync boring - Part 17

Building Finance System

~9 min read

The goal of a nightly job is to be boring. Nothing about it should be worth reading about. By early October the sync was no longer a script I ran by hand (post 3) but a multi-user, multi-provider pipeline, and getting it to quiet took a run of unglamorous commits. This post gathers them, from systemd units to log retention to two small test fixes. If the earlier posts were about adding capabilities, this one is about the failure modes that appear once the capabilities are live.

Monzo and the 30-day window

Post 7 described sca_exceeded: some providers only allow transaction history to be pulled shortly after the user has authenticated. That's a particular problem for a job that asks for the last 90 days at 3am, when the last authentication was long ago. One commit (fixing HL systemd units, fix subsequent monzo sync's limiting to 30 days - the title also covers the Hargreaves Lansdown fix in post 14) changes the window, and the reasoning is written into a comment above two constants:

INITIAL_SYNC_DAYS = 90
SUBSEQUENT_SYNC_DAYS = 30

A bank's first pull after linking (or re-linking) asks for the full 90 days, because that's also when a provider is most likely to grant deep history - "per their SCA exemption rules." Every later sync asks for 30, since "by then most history is already stored locally, and a narrower ask is less likely to run into a provider's 'shortly after authentication' restriction than repeating the full 90 every time would."

How does fetch() know which kind of sync this is? A synced flag in that bank's stored token entry. It's absent for a bank that has never completed a fetch, and - usefully - it's also absent after re-linking, because the callback overwrites the stored token wholesale. So re-linking to recover from a problem automatically triggers a deep pull, with no extra bookkeeping. Overwriting a record you want to reset is a perfectly good way to reset it.

Turning an error string into data

Post 7 improved the log line for a failed request but left the error as a string. The same commit makes it structured. get() now raises a TrueLayerAPIError, a subclass of requests.HTTPError - so every existing except requests.RequestException still catches it - that carries TrueLayer's error code as a real attribute. The docstring says why: so callers "can branch on it structurally instead of substring-matching str(exc)."

And one caller does. When a transactions fetch fails with sca_exceeded, the connection is flagged needs_reauth:

if isinstance(exc, TrueLayerAPIError) and exc.code == "sca_exceeded":
    connection_entry["needs_reauth"] = True

This is deliberately separate from consent_expires_at. The comment is precise: the consent itself is "usually still valid for months, but this provider (Monzo, notably) still demands a fresh PSU authentication to pull transaction history", so the Accounts page flags it as its own state - or else it's "masked by an otherwise-healthy 'Expires in N days' badge." Two different problems, two different badges, the same fix for the user (re-link). The new column lands with the usual PRAGMA table_info guard, and save_connections() defaults a missing needs_reauth to 0 so old JSON files and test fixtures don't have to know about it.

Don't fail the whole unit because someone has nothing to sync

One fix nightly sync commit is two small edits that fix a real consequence of post 10. The nightly script loops over every user directory. A user who's been provisioned but hasn't yet linked a bank - or has unlinked their last one - makes fetch() raise NoLinkedBanks, which the CLI turned into sys.exit(1): exactly the same exit code as a real failure. The unit would report failure every night over something perfectly expected.

The fix gives "nothing to do" its own code:

# CLI exit status for NoLinkedBanks - distinct from 1 (a real failure) so
# deploy/fetch-all-users.sh can skip an unlinked user without failing the
# whole nightly unit.
EXIT_NO_LINKED_BANKS = 3

and the shell script treats 3 as a skip (continue) rather than a failure. It's the same principle as post 7's partial-failure reporting, one level up: a monitoring signal that cries wolf gets ignored, and then it's useless on the night it's right. Exit code 1 should mean something went wrong.

Logs that clean up after themselves

The add nightly logging commit addresses a problem that only shows after a few weeks: every sync writes a timestamped log file per user, and nothing ever deletes them. Retention is built into the sync itself rather than being a separate cleanup job. prune_old_sync_logs() runs right before a run opens its log file and deletes anything older than 30 days.

Two details are worth noting. Age is read from the timestamp encoded in the filename, not the file's modification time, because a backup, restore or copy can change an mtime; the name is the record. And the docs are candid about the tradeoff: because retention rides along with the sync, "a user whose syncs have stopped running keeps whatever was last written until they run again." That behaviour is acceptable, and it's stated rather than discovered.

The same commit exposes the logs in the browser: the Debug page from post 11 gains an "Account sync logs" section backed by /api/debug/sync-logs, sorted newest-first using that same filename timestamp - sorting by filename would always put hargreaves_lansdown_... ahead of true_layer_... regardless of which ran last. The log-file endpoint also uses the pattern from earlier: the requested name must match one that the directory listing itself returned, so no path is ever built from user input.

One module, run two ways

A second commit called fix nightly sync moves the storage CLI out of sqlite.py's if __name__ == "__main__" block and into storage/__main__.py. The module docstring has the reason, and it's a classic Python wrinkle: the storage package's __init__ already imports sqlite, so running python -m finance_system.storage.sqlite re-executes a module that's already been imported, which triggers a runpy RuntimeWarning. Moving the entry point to __main__.py means python -m finance_system.storage runs exactly once. The two fetch-*-all-users.sh scripts from post 10 and post 14 were updated to match, along with the docs - and the unit snippet in post 4, which that commit also touches.

Deploy and test hygiene

Two last categories of "boring" work. One commit hardens the deploy script. It now sets BatchMode=yes so a broken SSH key fails immediately instead of hanging on a password prompt Jenkins can't answer, and it prints where the script and the SSH session each actually land, so a mis-set host alias that loops "the target" back to the CI agent shows up in the log rather than as a deploy that silently does nothing. It also fixes remote-setup.sh resolving the unit files relative to the working directory - which, over ssh, is the login shell's home directory, not the script's.

And two test fixes. The first is a good one. Some frontend tests used a fixed fixture date of 2026-09-01, which stopped working when the real month rolled over, because the "this month so far" filter compares against the real clock. The fix uses today's date: the comment notes it's "always 'this month', so it's a safe default for every test that doesn't care about the exact date value." Tests that depend on the calendar rot on a schedule. The second updated a backend test's expectation after HL got a logo.

What boring costs

None of these is a feature, and few are visible. But the sync is the foundation under every page - budgets, savings and categories all read what it wrote - and it runs when nobody's looking. The pattern across all of them is the same: when something is expected (a user with no banks, a rolled-over month, a 30-day window), give it a name in the code, instead of letting it look like a failure.


Previous: ← Savings projections · Next: Cash, and an audit trail →