MySQL

Overview

Write data to MySQL databases. Tables are created automatically from your source schema, and rows are upserted when a primary_key is provided.

Configuration

sinks:
  my_mysql_sink:
    type: mysql
    from: my_transform
    schema: my_database # MySQL treats `schema` and `database` as synonyms
    table: my_table
    secret_name: MY_MYSQL_SECRET
    primary_key: id # Optional - enables upsert (ON DUPLICATE KEY UPDATE)

Parameters

string

required

Must be mysql.

string

required

The transform or source to read data from.

string

required

Database name. MySQL treats schemas and databases as synonyms; this value is used as the database the sink writes to.

string

required

Table name to write to. Created automatically if it doesn’t exist.

string

required

Name of the Goldsky secret containing MySQL connection credentials.

string

Optional. Column (or comma-separated list of columns) to use as the table’s primary key. When set, inserts become upserts. Without it, writes are plain INSERTs and the auto-created table has no primary key.

string

default:"update"

Behavior when a row with the same primary key already exists. update uses ON DUPLICATE KEY UPDATE to overwrite non-key columns. nothing uses INSERT IGNORE to skip the duplicate. Only applies when primary_key is set.

integer

default:"1000"

Optional. Maximum rows per INSERT statement.

string

default:"1s"

Optional. Maximum time to wait before flushing a partial batch, parsed as a humantime duration (for example, "500ms", "1s", "2s").

Secret format

MySQL secrets are stored as a structured JSON object with the connection fields (host, port, user, password, databaseName). Create one with the CLI:

goldsky secret create MY_MYSQL_SECRET

The CLI accepts either a JDBC/MySQL URL (mysql://user:pass@host:3306/database) or individual fields at the prompt — both are parsed client-side into the structured shape before the secret is saved. The databaseName in the secret populates the database the plugin connects to; it does not need to match the schema field in your pipeline YAML, but typically does.

Behavior

Auto table creation: The sink issues CREATE TABLE IF NOT EXISTS on startup. If primary_key is set, the table is created with that PRIMARY KEY. Composite keys are supported (e.g., primary_key: id,version).
No schema migrations: If the table already exists, the sink does not ALTER TABLE. Adding a new upstream column against an existing table will fail the insert.
Upsert vs insert: With primary_key set, the sink uses INSERT ... ON DUPLICATE KEY UPDATE (or INSERT IGNORE when on_conflict: nothing). Without it, writes are plain inserts.
Deletes: Rows arriving with _gs_op = "d" are deleted by primary key. Deletes are a no-op when no primary_key is configured.
Type mapping: Arrow types map to MySQL as Int32 → INT, Int64 → BIGINT, UInt64 → BIGINT UNSIGNED, Float64 → DOUBLE, Utf8 → TEXT, Binary → LONGBLOB, Timestamp → DATETIME(6), Date → DATE, Decimal(p,s) → DECIMAL(p,s) (precision capped at 65), and nested types (struct/list/map) → JSON.

Example

sinks:
  mysql_transfers:
    type: mysql
    from: filtered_transfers
    schema: goldsky
    table: erc20_transfers
    secret_name: MY_MYSQL
    primary_key: id

Turbo

API

Reference

Overview

Configuration

Parameters

Secret format

Behavior

Example

​Overview

​Configuration

​Parameters

​Secret format

​Behavior

​Example

Overview

Configuration

Parameters

Secret format

Behavior

Example