ClickHouse - Goldsky Docs

Closed Beta

This feature is in closed beta and only available for our enterprise customers.

Please contact us at [email protected] to request access to this feature.

ReplacingMergeTree engine is used for all sink tables by default.

When appendOnlyMode is true (default), the sink behaves the following way:

All updates and deletes are converted to inserts.
is_deleted column is automatically added to a table. It contains 1 in case of deletes, 0 otherwise.
If versionColumnName is specified, it's used as a version number column for deduplication. If it's not specified, insert_time column is automatically added to a table. It contains insertion time and is used for deduplication.
Primary key is used in the ORDER BY clause.

When appendOnlyMode is false:

All updates and deletes are propagated as is.
No extra columns are added.
Primary key is used in the PRIMARY KEY clause.

Pipeline configuration

{
  "sources": [],
  "transforms": [],
  "sinks": [
    {
      "description": "Type.Optional(Type.String())",
      "type": "clickHouse",
      "sourceStreamName": "Type.String()",
      "secretName": "Type.String()",
      "table": "Type.String()",
      "batchSize": "Type.Optional(Type.Integer())",
      "flushInterval": "Type.Optional(Type.String())",
      "appendOnlyMode": "Type.Optional(Type.Boolean())",
      "versionColumnName": "Type.Optional(Type.String())"
    }
  ]
}

Secrets


goldsky secret create A_CLICKHOUSE_SECRET --type clickHouse --value '{
  "url": "Type.String()",
  "username": "Type.String()",
  "password": "Type.String()",
  "databaseName": "Type.String()"
}'