Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.goldsky.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Write data to MySQL databases. Tables are created automatically from your source schema, and rows are upserted when a primary_key is provided.

​
Configuration

sinks:
  my_mysql_sink:
    type: mysql
    from: my_transform
    schema: my_database # MySQL treats `schema` and `database` as synonyms
    table: my_table
    secret_name: MY_MYSQL_SECRET
    primary_key: id # Optional - enables upsert (ON DUPLICATE KEY UPDATE)

​
Parameters

​
type
string
required
Must be mysql.
​
from
string
required
The transform or source to read data from.
​
schema
string
required
Database name. MySQL treats schemas and databases as synonyms; this value is used as the database the sink writes to.
​
table
string
required
Table name to write to. Created automatically if it doesn’t exist.
​
secret_name
string
required
Name of the Goldsky secret containing MySQL connection credentials.
​
primary_key
string
Optional. Column (or comma-separated list of columns) to use as the table’s primary key. When set, inserts become upserts. Without it, writes are plain INSERTs and the auto-created table has no primary key.
​
on_conflict
string
default:"update"
Behavior when a row with the same primary key already exists. update uses ON DUPLICATE KEY UPDATE to overwrite non-key columns. nothing uses INSERT IGNORE to skip the duplicate. Only applies when primary_key is set.
​
batch_size
integer
default:"1000"
Optional. Maximum rows per INSERT statement.

​
Secret format

MySQL secrets are stored as a structured JSON object with the connection fields (host, port, user, password, databaseName). Create one with the CLI: 
goldsky secret create MY_MYSQL_SECRET
The CLI accepts either a JDBC/MySQL URL (mysql://user:pass@host:3306/database) or individual fields at the prompt — both are parsed client-side into the structured shape before the secret is saved. The databaseName in the secret populates the database the plugin connects to; it does not need to match the schema field in your pipeline YAML, but typically does.

​
Behavior

  • Auto table creation: The sink issues CREATE TABLE IF NOT EXISTS on startup. If primary_key is set, the table is created with that PRIMARY KEY. Composite keys are supported (e.g., primary_key: id,version).
  • No schema migrations: If the table already exists, the sink does not ALTER TABLE. Adding a new upstream column against an existing table will fail the insert.
  • Upsert vs insert: With primary_key set, the sink uses INSERT ... ON DUPLICATE KEY UPDATE (or INSERT IGNORE when on_conflict: nothing). Without it, writes are plain inserts.
  • Deletes: Rows arriving with _gs_op = "d" are deleted by primary key. Deletes are a no-op when no primary_key is configured.
  • Type mapping: Arrow types map to MySQL as Int32 → INT, Int64 → BIGINT, UInt64 → BIGINT UNSIGNED, Float64 → DOUBLE, Utf8 → TEXT, Binary → LONGBLOB, Timestamp → DATETIME(6), Date → DATE, Decimal(p,s) → DECIMAL(p,s) (precision capped at 65), and nested types (struct/list/map) → JSON.

​
Example

sinks:
  mysql_transfers:
    type: mysql
    from: filtered_transfers
    schema: goldsky
    table: erc20_transfers
    secret_name: MY_MYSQL
    primary_key: id