Profiling with cost:profile

#Profiling with cost:profile

cost:profile runs a catalogue of representative API calls against a live gateway, reads each response's cost receipt, and writes a versioned report. Paired with a checked-in baseline, it turns "did this change make requests more expensive?" into a command with an exit code.

#Prerequisites

• A running gateway (npm run gateway:dev locally, or point the runner at a staging instance)
• An account with the admin or super admin role — the runner creates a scratch collection to profile against

#The three commands

Command	What it does
npm run cost:profile	Runs every scenario (warm-up pass, then 5 samples each), reports medians and max, writes tests/reports/cost/<date>-<sha>.json and .md
npm run cost:diff	Compares the latest report to tests/cost/baseline.json. Budget violations fail with exit code 1; drift over 25% against the baseline warns
npm run cost:baseline	Promotes the latest report to the checked-in baseline

The runner shows live progress with baseline deltas inline, so you see a regression while the run is still going. Drift warnings apply to operation-count metrics only (rows read, query counts, KV reads) — timings jitter between runs and are never flagged.

#Authentication

The runner signs up or signs in as cost-profile@test.sveltesync.local. On instances without open sign-up, point it at an existing account:

$ COST_PROFILE_EMAIL=alex@example.com COST_PROFILE_PASSWORD=securePassword123 npm run cost:profile

The account needs admin or super admin to create the scratch collection. If it can't, the run fails with the exact promotion command to fix it.

#Adding a scenario

Scenarios live in tests/cost/scenarios.mjs — one entry per representative call. Append to the SCENARIOS array:

// tests/cost/scenarios.mjs
{
  id: 'records/list-by-category',            // unique, groups by prefix
  group: 'records',
  request: (ctx) => ({
    method: 'GET',
    path: `/api/v1/collections/${ctx.coll}/records?category._eq=alpha&limit=25`
  }),
  budget: { d1q: 25, rr: 900, dur: 2000 }    // hard caps; omitted keys are unchecked
},

• request(ctx) builds the call. ctx provides coll (the scratch collection name) and firstId (one seeded record ID).
• An optional prepare(ctx, api) runs before every sample, unrecorded — use it when a scenario needs fresh state, such as creating a record for an update or delete to target.
• After adding a scenario, run cost:profile then cost:baseline so the new scenario has a reference for future diffs.

#How budgets work

Each scenario's budget sets hard caps using receipt keys: rr (rows read), rw (rows written), d1q (D1 queries), kvr (KV reads), dur (duration in ms). Any metric over its cap fails the run.

Budgets start deliberately loose — around 3× the expected median. A loose budget that fires isn't noise; it's a genuine regression. Tighten budgets deliberately once a scenario has accumulated a history of stable baselines, not on day one.

#The release ritual

Profiling is manual by design — no CI gate. Before each release:

$ npm run cost:profile
$ npm run cost:diff

A clean diff means no budget exceeded and no unexplained drift. If you changed something that legitimately alters costs — a new index, an extra auth check — re-baseline so the next diff measures against the new normal:

$ npm run cost:baseline

To give a run a durable home, publish it:

$ npm run cost:profile -- --publish

Published runs appear in the Runs view of the Performance tab, where you can compare any two runs side by side.