Data movement

Explore · Data movement

Copy data between engines, reviewed before it writes.

A migration job reads rows through the same read-only gate as every query, then writes them into one table of a database an Admin marked as a write target. It is the only feature that writes to your databases, so it is off until the host turns it on.

3write modes
100,000rows per run, default budget
Offuntil the host enables it

01Three gates before a write

Three decisions stand between a job and its first write.

Data migration is off by default. Before a job writes a single row, the host enables it, an Admin marks the destination, and, with the approval workflow enabled, a second person approves the job. The job's operator still needs project access to both ends.

1Host
Enable data migration
The host sets Beacon:Migrations:Enabled to true. It is a deployment lock: while it is false, nobody can create or run a job, an Admin included.
- Routes under /beacon/api/migrations are not mapped (404).
- The UI shows no Data Migration entry, page or Home card.
- No data source can be marked as a write target (409).
2Admin
Mark a write target
On the data source page, Migration writes → Mark as migration write target. Only a PostgreSQL, MySQL or SQL Server database; never a host-managed data source or one marked read-only.
- The Destination data source picker lists write targets only.
- Recorded in the settings history and as event 7115.
- A source that live queries read is marked only on a second confirmation.
3Approver
Approve the definition
When the approval workflow is enabled, a second person approves each new job on Approvals under Migration jobs: source, query, destination, table, mode and transformation script.
- Until then the job shows Awaiting approval and Run now is disabled.
- Any run is refused and recorded as failed.
- Nobody approves their own job.

{
  "Beacon": {
    "Migrations": {
      "Enabled": true,
      "MaxRows": 100000
    }
  }
}

Who runs a job

Data migration needs the MigrationOperator permission (Admins hold it too), and it grants no bypass of project access. Creating a job, every run of it and deleting it need Query on the source and on every step's data source, and Write on the destination.

The destination may be no more widely reachable than any source: every project that links it must link each source too. Copying data never makes it readable to more people.

02Modes

Three ways to write the same rows.

Each job writes in one mode. The same two source rows, one changed and one new, land differently in each. Columns map by name: the source's column names are the destination's.

Destination before the rundbo.customer_summary

customer_id	country	lifetime_value
1001	HR	1,240.00
1002	SI	380.50
1003	AT	2,915.10

Source rows this runchangednew

customer_id	country	lifetime_value
1002	SI	412.00
1004	HR	96.20

Insert only

Partial (MySQL)

Inserts every source row. A row that violates a key or constraint fails the write.

id	country	value
1001	HR	1,240.00
1002	SI	380.50
1003	AT	2,915.10
1004	HR	96.20

Row 1002 already exists. On PostgreSQL and SQL Server the write fails and the run is Failed. On MySQL Beacon falls back to inserting row by row: 1004 is written, 1002 is counted and listed in the run's error, and the run ends as Partial.

Needs INSERT on the destination table.

Insert or update

Completed

Stages the rows in a temporary table, then inserts new rows and updates existing ones by primary key.

id	country	value
1001	HR	1,240.00
1002	SI	412.00
1003	AT	2,915.10
1004	HR	96.20

Row 1002 is updated, 1004 is inserted, and rows the source did not return stay as they were.

A primary key on the destination, and every key column in each source row. Without a primary key the job inserts, as in Insert only.

Truncate & insert

Completed

Empties the destination table, then inserts every source row.

id	country	value
1002	SI	412.00
1004	HR	96.20

The table holds exactly what the source returned this run. A failed read never empties it: the destination is not opened.

Deletes every row in the destination on each run. Try it by hand against a test table first.

HOW EACH ENGINE WRITESdestination engine

Engine	Insert	Insert or update
PostgreSQL	`COPY … FROM STDIN (FORMAT BINARY)`	`COPY` into a temporary table, then `INSERT … ON CONFLICT`
SQL Server	bulk copy	bulk merge, staged in a session temporary table
MySQL	bulk insert; if it fails, row by row	bulk insert into a temporary table, then `INSERT … ON DUPLICATE KEY UPDATE`

03Worked example

A nightly customer summary for the reporting database.

Lifetime value per customer lives in the shop database (PostgreSQL). The reporting team reads it from SQL Server. Ana Horvat, a MigrationOperator, creates a job that copies it across; Marko Kovač reviews it.

DATA MIGRATION · NEW JOBform

Job name

Customer summary to reporting

Description

Lifetime value per customer for the reporting team

Source data source

Shop (PostgreSQL)

Source SQL

SELECT c.customer_id,
       c.country,
       SUM(o.total) AS lifetime_value
FROM shop.customers c
JOIN shop.orders o ON o.customer_id = c.customer_id
WHERE o.status = 'paid'
GROUP BY c.customer_id, c.country;

Destination data source

Reporting (SQL Server) Migration write target

Destination table

dbo.customer_summary

Migration mode

Insert or update

Cron schedule

0 2 * * *

Max retries

Timeout (minutes)

Enable migration job

Validate query before execution

Transformation script (optional)

empty

Create migration job

Oct 13 · 10:04Created, submitted for approvalSubmitted for approval: the job does not run, manually or on its schedule, until a second person approves it.
Oct 14 · 02:00The host's nightly job starts a run: refusedNot run: the job's definition waits for approval. A second person approves it on the approvals page; until then the job does not run.
Oct 14 · 09:12Marko Kovač approves the definitionSource, SQL, destination, table, mode and the empty transformation script. Name, schedule, retries and timeout are not part of it.
Oct 15 · 02:00Run completes48,210 rows read through the gate, within the 100,000-row budget, then merged into dbo.customer_summary.

MIGRATION HISTORY/beacon/migration-history

Started	Job	Status	Rows	Failed	Rows/s
2026-10-15 02:00	Customer summary to reporting	Completed	48,210	0	7,533
2026-10-14 02:00	Customer summary to reporting	Failed	0	0	0

-- Only the write target's login holds write grants.
GRANT INSERT, UPDATE ON dbo.customer_summary TO beacon_migrator;

RunningThe run has started.

CompletedEvery row was written.

PartialSome rows failed (MySQL row-by-row fallback).

FailedThe read or the write failed, or the run was refused. The error says why.

Every mode also reads information_schema to check that the table exists and, for Insert or update, its primary key. A job cannot be edited after it is created: to change it, create a new job and delete the old one. The old job's run history is kept.

04Guarantees

What every migration can rely on.

Beacon never reads a write target

Query steps, previews, subscriptions, data quality, MCP tools, AI grounding, documentation and metadata scans refuse it or leave it out. Only the migration writer connects with its credentials, so every data source Beacon reads keeps a read-only login.

A row budget, checked before writing

Beacon:Migrations:MaxRows (100,000 by default, 1 to 10,000,000) is written into the source SQL, and the read stops one row past it. A larger source fails the run before anything is written.

Failed reads write nothing

If any step or the final query fails or times out, the run is recorded as failed and the destination is not opened. A Truncate & insert job never empties its table on a failed read.

Every run recorded

Each run keeps rows read, written and failed, its duration and its error. A run refused because the job is not approved is recorded as Failed, with the reason.

05Good to know

How runs start and what a job keeps.

Scheduled runs start from the host

The cron schedule is stored with the job and shown on it. A run starts from Run now, from the API, or from the host's own background job, and the approval and write-target checks apply to every run.

A failed run waits for a person

Max retries is stored with the job and shown on each run as Retry. A failed run stays failed: read the error, fix the cause, then use Run now.

Transformation script left empty

Create jobs with Transformation script (optional) empty; a job with a script fails when it runs. To change one, create the job again with the field empty.

→Go deeper

Data migrationJobs, modes, write targets, run history and the REST API.Data sourcesRead-only logins and the migration write-target flag.ApprovalsFour-eyes review of job definitions before any run.

Data movement

Three decisions stand between a job and its first write.

Enable data migration

Mark a write target

Approve the definition

Who runs a job

Three ways to write the same rows.

Insert only

Insert or update

Truncate & insert

A nightly customer summary for the reporting database.

What every migration can rely on.

Beacon never reads a write target

A row budget, checked before writing

Failed reads write nothing

Every run recorded

How runs start and what a job keeps.

Scheduled runs start from the host

A failed run waits for a person

Transformation script left empty