Importing data

Bring data into a collection from a CSV file or a Directus instance — mapped, transformed, previewed, and reversible.

#Importing data

Every collection can ingest data through the import wizard at Data → Import — or straight from a collection page via its Import button, which pre-selects that collection as the target.

Two kinds of source are supported today:

  • A CSV file, dropped onto the wizard. The file is parsed in your browser, so you see your own columns and a preview before anything leaves your machine.
  • A Directus instance, connected with a read-only token. The wizard reads its schema, pairs each source collection with one of yours, and pulls the records server-side — your browser never has to hold them.

Either way, the pipeline behind the wizard is the same one: map → transform → validate → write → ledger. Every import is recorded row-by-row in an import ledger, which is what makes it precisely reversible.

#Importing a CSV

  1. Drop the file. Quoted commas, escaped quotes, embedded newlines and a leading byte-order mark are all handled — spreadsheet exports are messy and that is normal. Rows with a mismatched column count import with blanks, and the wizard tells you how many there are.
  2. Map the columns. Columns whose names match a field are pre-mapped; everything else is a dropdown. Unmapped columns are simply skipped.
  3. Add transforms where needed. Each column can carry a chain of transform chips — trim, case changes, split ("Surname, First"), join, replace, slug, truncate, number, yes/no, and date parsing with an explicit format (03/04/2024 is two different days depending on who wrote it, so the wizard never guesses). The ƒx button opens the same transform as a raw JSONata expression, pre-filled with what the chips already do.
  4. Watch the preview. The first rows are shown exactly as they will be written — computed by the same engine the import runs, not a lookalike. A row that would fail shows the field and the reason instead of its values.
  5. Choose the row identity. A unique column, a combination of columns, or an explicit "no identity — one-shot append" with its trade-offs stated.
  6. Test, then import. Test with first 100 writes real records through the real pipeline, capped. Check the result in the collection, then either roll it back or run the full import.

#Typed values fail loudly, not silently

A value that cannot become what the target field needs — "twelve" in a number column, "maybe" in a yes/no column, an unreadable date — fails its own row with the field named, and the rest of the batch still imports. Nothing is quietly written as null: a blank cell stays empty, but a broken value is an error you can see.

#Importing from Directus

Choose connect a Directus instance on the import page and paste the instance URL plus a read-only access token. The token is encrypted before it is stored and is never sent back to the browser; nothing is ever written to the source.

#Setting up the Directus token

Three things have to be true, and each one fails differently:

  1. Generate a static access token on the user, then save the user. Directus creates the token in your browser and only stores it when you press Save. An unsaved token fails with "Directus did not recognise the token (401)" — which is not a permissions problem, however good the role is.
  2. The user's status must be Active. A suspended, invited or draft user is rejected the same way.
  3. Its role needs read access to the data and to the schema. Grant read on the collections you want to migrate, plus the system collections directus_collections and directus_fields — those are how the instance is asked what exists. Missing them gives "Directus accepted the token but refused /collections (403)".

That third one is a genuine trade-off, so it is optional: if your role cannot read the schema, the wizard says so and asks you to name the collections instead, then learns their columns by reading a sample row. That path works without granting anything beyond the data itself, at the cost of one caveat — a column that is empty in every sampled row is invisible, and required/unique constraints are unknown.

The wizard then probes the instance and shows a compatibility report: every source collection, its row count, the collection here it would import into, how many columns line up, and what still needs a decision. Matching is deliberately conservative — a wrong automatic match writes plausible data into the wrong field and survives review, so anything uncertain is left for you rather than guessed at.

#System collections

Directus keeps some of your data in tables it calls its own, and the report leaves those out until you ask for them. Show system collections, at the top of the report, adds three:

Collection What it is Where it goes
directus_files Your media library File storage — see Importing files
directus_users Your user directory People — matched by email address
directus_folders The folder tree over your files Folds into files as a path on each one

Nothing else with a directus_ prefix is ever offered. Sessions, activity, revisions, presets, permissions and settings describe how Directus itself works rather than what you own, and there is nothing here for them to become.

The toggle re-reads the source rather than filtering what is already on screen, so it takes a moment on a large instance — those collections were not fetched while it was off.

Pick a collection and Set up import: the suggested mapping lands in the same editor used for CSVs, with a preview built from real sampled rows. The import itself runs server-side, page by page, using the source's own primary key as the row identity.

#When nothing has anywhere to land

A freshly migrated instance has none of the source's collections yet, so every row of the report reads "no matching collection". Use Create it here: the wizard proposes a collection built from the source's own shape — field names kept, types mapped (Directus string → text, text → richtext, integer → integer, dateTime → datetime, and so on), anything unrecognised falling back to text.

Nothing is created until you confirm, and the proposal is shown in full first. Three deliberate choices in it:

  • No field is marked required. A legacy blank would otherwise fail the row for the old system's reasons rather than yours.
  • Relations arrive as plain columns holding the source's key, not as relation fields. A relation needs a target collection that may not exist yet, and guessing wrong writes a foreign key into the wrong table. Convert them afterwards, once every collection is present.
  • The source's own key is kept as legacy_id, not discarded. See below.

#What happens to source ids

Every row arrives with a fresh EmuView id, and the source's key is kept alongside it. There are three things you can do with that key, and the wizard lets you pick:

When it is right
legacy_id column (default) Almost always, and always for counter-style integer keys. You keep every reconciliation benefit — redirects from old URLs, delta re-sync, joining back to the source, resolving the raw keys that imported relations still hold — without inheriting another system's id space.
Keep it as the row's id When the source key is opaque and globally unique (a UUID or ULID), the target collection is new and dedicated to this one source, and other things point at it — imported relations, deep links, an external index. Map the source key to id in the column list. A repeat run then reports "already exists" per row instead of duplicating, which is a much better kind of failure.
Drop it Only for genuinely disposable data that nothing references and you will never re-sync. Set the column to skip.

Two cautions on keeping the id. Rows whose id is already present fail as "already exists" rather than overwriting, so this suits an empty collection. And adopting sequential integer keys makes record ids guessable where they were previously opaque — another reason legacy_id is the default.

The import ledger records the source key for every row regardless, which is what rollback and error triage use; the difference is whether the collection itself can answer "which source row is this?".

Worth knowing up front:

  • Records only, for now. Users, permissions and flows migrate via the CLI toolset — same engine, wider reach. Files are copied storage-to-storage by a tool built for it: see Importing files.
  • Password hashes never travel over the Directus API. Users imported this way arrive without passwords (send reset links), unless you use the CLI against a database dump.
  • Private instances are refused, on purpose. A source on localhost or a private network cannot be reached from the platform and will not be tried — that is exactly the case the CLI exists for, since it runs on a machine that can see the instance.

#Saved connections are shared, not private

Once you connect a source, it is kept so you do not have to paste the token again. It is worth being clear about who "you" means here:
Who can see a saved connection Everyone in the project who can run imports
What they can do with it Probe the source, sample rows, and import from it
What they cannot do See the token. It is encrypted before storage and never returned to any browser

So a saved connection is a shared team resource, not a personal one. If a credential should not be usable by every admin in the project, do not save it — connect, import, and remove the connection afterwards.

The × on each connection forgets it. That deletes the stored credential and the history of imports made through it; the imported rows stay, but they can no longer be rolled back from the import history.

#Names get tidied up on the way in

Collections and fields here are lower case with underscores, starting with a letter. Most sources are not: Directus is happily lengthM, spreadsheets have Length (m) and 2024 total. Rather than refuse them, the wizard converts — and shows you every change before it creates anything.

Source name Becomes
lengthM length_m — the capital marks a word boundary, so it is kept
Length (m) length_m
2024 total f_2024_total — a name cannot start with a digit, and dropping the year would merge it with total
café cafe
Created At skipped — it lands on a column the platform sets itself
a collection called users users_import — users is a reserved name here

Two source columns can convert to the same name (lengthM and Length (m) both want length_m). The first keeps it and the next becomes length_m_2, decided by source order so a re-run produces exactly the same schema.

The conversion is one-way by design: length_m cannot tell you whether it came from lengthM or Length (m). The mapping records the original column explicitly, so nothing has to guess later.

#Not everything a source calls a field is a column

Sources carry interface furniture in the same list as real data: Directus has tabs, dividers and notices, and its one-to-many relationships appear as fields too. None of them holds a value, so none of them is switched on by default.

That is a suggestion, not a verdict. When you create a collection from a source, every source column appears in one table — the ones that will not be created are simply switched off, and you can switch any of them back on. You can also rename a column or change its type there, and a renamed column says what it was called in the source so nothing changes silently. This is the last point at which any of that is free: afterwards it is a schema edit.

One thing worth knowing before switching a greyed-out row back on: if the source keeps no value for it, the column is created but arrives empty, because there is nothing on the other side to fill it. The table says so on the row.

A handful of names cannot be used, because this platform sets those columns itself — id, created_at, updated_at, created_by, updated_by, deleted_at and _version. If a source column wants one of them, give it another name, such as legacy_created_at; the data imports normally under the new name.

What does travel besides the data: a field's description and its display order, because those are the author's own words and arrangement.

What deliberately does not:

Why
Column widths, groups, tabs This platform has its own form layout; importing another's produces one that matches neither
Hidden, readonly, conditions Interface rules about a different interface
Validation The target has its own; two overlapping rule sets is worse than one
Permissions Directus 11 is policy-based, this platform is role plus per-key scopes, and the mapping is not mechanical. A permissions import that is 90% right is a security problem, not a convenience — set access up here instead

#When a column sends more than one value

Some source fields hold a whole document rather than a single value — a JSON field, a geographic point, a list of related rows. A column that holds one value cannot take those, and the wizard now says so before the import, naming the column:

drive_to sends an object into drive_to — add keep as JSON text to store the whole thing, or take key to pull one value out of it.

Two transforms handle it:

  • keep as JSON text — stores the whole document as text. A plain value passes through unchanged.
  • take key — pulls one key out (defaulting to id, which is what a related row carries). If the value is already a plain key it is left alone, and a list of related rows becomes a comma-separated list of their keys.

If the target column is itself a JSON or geographic type, there is nothing to warn about — it can hold the document as it is.

#Map columns

A geographic column holds one shape, and each type says which: a point, a line, an area, or the multi-part form of any of those — several shapes under one name. Directus names its geographic columns the same way, so each maps straight across:

Directus Proposed here
geometry.Point point
geometry.LineString linestring
geometry.Polygon polygon
geometry.MultiPoint multipoint
geometry.MultiLineString multilinestring
geometry.MultiPolygon multipolygon

Only GeometryCollection has no equivalent — it mixes shape types inside one value, which nothing here can index — and a column of that kind is proposed as JSON instead. The document is kept exactly as it arrived and nothing is lost, but it is stored as a document rather than as something you can search by location.

Many sources emit a plural type for a single shape — one "multi-point" containing one point. That imports as-is and is perfectly valid. If you would rather have the plain type, change the type in the create-table step to point, linestring or polygon and add the single geometry transform to the column on the mapping screen; it converts a one-shape multi to the plain shape.

If a value genuinely holds several shapes, that transform leaves it alone rather than quietly keeping the first one and dropping the rest — so pick the multi type for it, which is what it is.

#Harder mappings: several columns into one

The transforms cover one column at a time. When you need several columns to become one value, press fx on the row and write the mapping out.

The thing to know — and it is not obvious from a screen laid out one row per source column — is that an expression sees the whole row, not just the column it sits on. $lookup($, "other_column") reads any column you like. So you attach the expression to any one of the columns involved, point it at the target field, and leave the others unmapped.

Everything below is a real expression, checked against the same engine the import runs.

#Six numeric columns into one JSON object

Say the source has as2156_track, as2156_gradient, as2156_signage, as2157_track and so on, and you want a single JSON column holding:

{ "as2156": { "track": 2, "gradient": 4 }, "as2157": { "track": 3 } }

Make the target column type JSON, pick any one of those source columns, set its target to that column, press fx, and write it out in full:

{
  "as2156": {
    "track": $toNumber($lookup($, "as2156_track")),
    "gradient": $toNumber($lookup($, "as2156_gradient")),
    "signage": $toNumber($lookup($, "as2156_signage"))
  },
  "as2157": {
    "track": $toNumber($lookup($, "as2157_track")),
    "gradient": $toNumber($lookup($, "as2157_gradient")),
    "signage": $toNumber($lookup($, "as2157_signage"))
  }
}

Long, but every key is visible and you can rename anything on the way through — "gradient" on the left is the name you want, "as2156_gradient" on the right is the name the source used. Leave a column out and it is simply not there.

#The same thing, without listing every column

If the source columns are named to a pattern, you can group them by their prefix instead. This one takes every column whose name starts as, splits the name at the first underscore, and groups what it finds:

[$each($, function($v, $k) { { "k": $k, "v": $v } })]
[$substring(k, 0, 2) = "as" and v != null]
{ $substringBefore(k, "_"): { $substringAfter(k, "_"): $toNumber(v) } }

Given as2156_track: "2", as2156_gradient: 4, as2157_track: 3 that produces {"as2156": {"track": 2, "gradient": 4}, "as2157": {"track": 3}} — and it keeps working when the source gains an as2158, which the long version does not.

Read the middle line as the guard: it decides which columns are in. Widening it to $contains(k, "_") takes every column with an underscore in its name, which is tidy right up until it sweeps in a length_m you never meant to include. The and v != null part is what keeps empty cells out of the object.

Whichever you use, the preview is the check — run it over a few rows and read what actually comes out before you start.

#Other things worth copying

You want Expression
Two columns joined into one line $lookup($, "first") & " " & $lookup($, "last")
The whole source row kept as a JSON record $
One value, falling back to another when it is empty $lookup($, "display_name") ? $lookup($, "display_name") : $lookup($, "name")
A number that might arrive as text $toNumber($lookup($, "length_m"))
A fixed value on every row "imported"
The time of the import $now
A list column flattened to comma-separated keys $pick($lookup($, "tags"), "id")

Four things that will save you a puzzled minute:

  • $now, not $now(). It is the moment the import ran, in epoch seconds, and it is the same value for every row in the run — which is what makes it useful for marking a batch. Written with brackets it fails.
  • Joining with & keeps the gap. If last is empty, first & " " & last gives you "Ada " with a trailing space. The join with transform drops blanks instead, so prefer it when either side might be missing.
  • A blank column reads as null, so a missing value gives you a key holding null rather than an error.
  • $toNumber refuses text rather than guessing. "12.5", "$1,200.50" and "1.200,50" all become numbers; "Ada" fails the row with "Ada" is not a number rather than importing a silent 0. That failure is visible in the preview before you start, and in What failed afterwards.

#Importing only some rows

A connected source gets a Which rows panel above the column mapping — the same filter conditions the collection views use. Only rows that match are imported.

The conditions are applied at the source, not after fetching, which matters in three ways: rows that do not match are never transferred, never counted against a pilot's cap, and never paid for on a metered source. The preview re-reads through the filter too, so what you check is what you get.

The filter is fixed for the life of a run, including a resume — a continued run cannot quietly widen its own scope and pull in rows you excluded.

Two things worth knowing:

  • The row count on the compatibility report is the collection's total, not the filtered count. The first preview after applying a filter is the honest check on whether it matches what you expected.
  • A file drop has no filter panel. The rows in the file are already the rows you chose; filter the export instead.

#Rows per batch

Under the import buttons is Rows per batch, folded away because the default suits most tables. It sets how many rows go in each write.

Your rows Try
Ordinary — names, dates, numbers, short text Leave it at 200
Wide — long article bodies, large JSON columns, many columns 25–50, so a single request stays small enough to accept
Failing and you cannot see why 1, so the failure names the exact row

It changes speed and nothing else. The same rows are written whatever it is set to, and a target that asks the import to slow down is waited out rather than counted against the rows.

It is also only a starting hint. If a batch turns out to be too large for one request, the import halves it and carries on — it tells you it has done so, and no rows are lost. You should not have to find the right number by trial and error.

#Relations between imported tables

#Importing several collections together

Map a collection as you normally would, then press Add to a multi-collection import instead of importing it. Do the same for the next one. A panel appears listing what is queued, and Import all N collections starts them.

They run one after another, in an order worked out from the Links to settings you have already made — what is pointed at is imported before what points at it. Press Show me the order to see it, with a line against each collection saying why it is where it is. That order is worked out on the server from the same mappings it is about to run, so what you are shown is what will happen.

Order does not change the outcome — Resolve links can fix any reference afterwards — but it means fewer links to fix, and it matters for a required reference, which fails its rows rather than waiting.

If one collection stops, the rest wait. Everything before it stays imported and can still be rolled back on its own; the ones after it never start. Fix the mapping and press Resume, and it picks up at the collection that stopped — continuing its existing run rather than starting a second one, so nothing is imported twice. Roll back all undoes the whole thing, newest collection first.

Two things it will not do. It refuses two collections importing into the same target, because then neither has a history you could roll back separately — import those one after another. And a dropped file is one collection by definition, so this only appears for a connected source.

#Who wrote these rows

Records usually carry an author — a user_created, an owner, a created_by pointing at somebody in the old system. That pointer means nothing here until something resolves it, because this platform owns its own sign-in.

Under Who wrote these rows on the mapping screen, pick the column that names the author and choose what to do with it:

Choice What happens
Match, and create the missing People with an account here are matched; accounts are created for everyone else
Match only People with an account here are matched; everyone else is left exactly as they are
Nothing Authorship is untouched — import the column as ordinary data if you want to keep it

Matching is on the email address and nothing else. A name that looks like a match is not one, and a row signed by the wrong person looks completely normal afterwards — you would never find it by reading the data. Anyone who cannot be matched keeps the authorship they would have had without any of this: nothing is guessed and nothing is blanked, so you can match them later and run the import again.

Before an import is allowed to create accounts, press Check who matches. It reads the source's people and tells you how many match, how many would be created and who they are — the count is worth seeing before it happens, which is why the import button waits for it.

This screen creates up to 500 accounts at a time. Past that, the command-line toolset is the path — it imports users as a phase with its own resume, which is what an estate that size needs.

If the source's people change between checking and starting, the import stops and asks you to check again rather than creating somebody who was never in the number you agreed to.

New accounts are created with no password and no email is sent. Each person signs in for the first time through a reset link, which you send when you are ready — from People, not as a side effect of an import.

A few things worth knowing:

  • Your token needs read access to the source's user list. Without it the option says so and you can import the author column as ordinary data instead.
  • Creating accounts needs permission to create users, not just to run imports. If you have one and not the other, Match only still works.
  • A rollback never deletes an account, whether it was matched or created — deleting someone destroys whatever they did afterwards. Remove them from People if you want them gone.
  • People matched on one import are remembered, so importing six collections does not create six accounts for the same person.

#If you leave the page

An import keeps going after you close the tab — it runs on the server, not in your browser. Come back to Data → Import and, if something is still running, it says so at the top with a Watch it button. It only offers; it never drops you into a progress screen you did not ask for.

That works from a different computer too, and after the tab that started it has been shut for a day: the wizard asks the server what is running rather than remembering anything itself. The history page lists everything, finished or not.

#Still to come

For a whole-estate migration — records and files, roles and permissions together — the command-line toolset is the supported path. See below.

When a column holds a key into another collection — region_id, author, parent — importing it as a plain value produces a reference that points at nothing. The target mints its own ids, so the source's 7 is not the row the target created for 7.

Set Links to on that column and the import resolves the key for you:

The situation What happens
The linked collection is already imported The key is swapped for the row it became, as the row is written
It is not imported yet The source key is kept, and the run reports pending links
You then import it and press Resolve links The kept keys are rewritten to the real rows
The linked row does not exist at all Reported as unresolved, and left alone

The order you import in therefore does not matter. Import children before parents, import halves of a circular reference, or come back to a migration you ran last month — Resolve links walks the rows that run imported and fixes whatever it can now resolve. It is safe to run repeatedly: it only rewrites keys it can resolve, and skips anything already linked.

Two things are worth knowing:

  • An unresolved key is never blanked. It stays in the column, because it is the only remaining record of what the row pointed at — blanking it would make the link unrepairable. A column that cannot be written empty is the exception: those rows fail at import, naming the collection to import first.
  • Rolling back a linked-to collection orphans the rows pointing at it. The references go back to holding source keys. Re-import that collection and run Resolve links again to repair them.

Only single-value references are handled here. Multi-value relations (many-to-many, translations, files) need the CLI — see below.

#Rollback and history

Data → Import → Import history lists every run: what it wrote, what failed and why, and what can still be undone. Failed rows are shown with their reasons, oldest first — the earliest failures usually explain the rest.

Rolling back permanently deletes the rows that run created — and only those. It is scoped to the run, so a later import into the same collection, and anything you created by hand, are untouched. A soft delete would leave unique values occupied and break the next attempt, so the delete is real; the ledger keeps the full record either way, because "imported, then removed" is history rather than something to erase.

One consequence worth knowing before you rely on it: a row this run imported is deleted even if someone has edited it since. The ledger tracks the row, not its contents, and there is no undo for the delete. If an import has been live long enough for people to work on, prefer fixing forward over rolling back.

#Finishing a test run

A test run stops at its cap with the rest of the rows still waiting. Import the remaining rows carries on the same run rather than starting a new one, so the rows already written are skipped instead of imported twice — and a rollback afterwards still undoes the whole import in one go.

Runs can also be paused mid-flight and resumed later; an interrupted run picks up where it left off without duplicating rows it already wrote.

#When something goes wrong

What you see What it means What to do
Directus did not recognise the token (401) The token does not exist on the instance — usually generated but never saved Re-open the user in Directus, generate the token, and press Save. Check the user's status is Active
Directus accepted the token but refused /collections (403) The role can read data but not the schema catalogue Grant read on directus_collections and directus_fields, or use the name the collections path instead
— in the Rows column of the report The row count failed — not "zero rows". A singleton, or a collection the token cannot read Grant read on that collection. It will import nothing until you do
transform failed on "price": … One row's value could not become what the field needs Open What failed, read the value, and fix the transform — or accept that those rows are bad and raise the error budget
"region" references regions "7", which has not been imported A required link whose target collection is not imported yet Import that collection first, then run this import again
Links reported as pending after an import The linked-to collection had not been imported when these rows were written Import it, then press Resolve links
A csv source cannot be pulled from; drop its file instead A run asked to pull from a definition that has no connection Use the file dropzone for this migration, or connect a source
The stored source credential could not be read The encryption key changed, or the stored credential is corrupt Reconnect the source — the token is never recoverable from storage by design
A collection shows not readable with 0 columns Directus filters /fields by what the token may read, so no columns means no Read permission on that collection — or a Read permission with an empty field list Open the row: it names the collection, the exact Directus screen, and links straight to it. If your admin app is served somewhere other than /admin, follow the written instruction instead
A collection shows a folder, not a table It is a Directus folder — a grouping entry with no table behind it Nothing to fix. Import the collections grouped under it instead
bulk write failed (413) A batch of rows was too large for one request Nothing to do — the import halves the batch, says so, and carries on. No rows are lost
bulk write failed (429) The target was rate limiting the import Nothing to do — the rows are put back and retried automatically, and none are lost. If it does not clear, the run stops and says so, and can be resumed
The source is on localhost or a private network Refused deliberately: the platform cannot reach it Use the CLI, which runs on a machine that can

#Leaving the page

The run does not belong to the page. It executes in a Durable Object on its own schedule, so closing the tab, navigating away or losing the connection changes nothing about it — rows keep landing. Data → Import → Import history is where you pick it back up, with pause, rollback and the failure list intact.

#The CLI power path

The wizard imports records into one collection at a time. Everything else about a migration is the CLI toolset — node scripts/migrate.mjs in the repository. It shares this design and the same ledger discipline; the wizard is one collection's worth of it with a friendlier front door.

Reach for the CLI when you need:

Why it is not in the wizard
Files A browser tab is the wrong place for gigabytes — copy them bucket-to-bucket instead (Importing files)
Users and passwords Password hashes can only be read from a database dump, which the platform cannot reach
Roles and permissions These decide who can see what, so they are imported deliberately and fail closed rather than being guessed
Flows and webhooks They are transpiled, and a transpiler needs review before it runs
A source on a private network The platform genuinely cannot reach localhost; your machine can
Many collections in dependency order The CLI plans a load order across the whole estate and holds back the links that order cannot solve

The two share the import ledger, so they can be mixed: pilot a collection in the wizard to check a transform, then run the estate from the CLI.