Skip to main content

​
Overview

Publish processed data back to Kafka topics with Avro or JSON serialization.

​
Configuration

sinks:
  my_kafka_sink:
    type: kafka
    from: my_transform
    topic: my_output_topic
    topic_partitions: 10
    data_format: avro

​
Parameters

​
type
string
required
Must be kafka
​
from
string
required
The transform or source to read data from
​
topic
string
required
Kafka topic name to publish to
​
topic_partitions
number
Number of partitions for the topic (created if doesn’t exist)
​
data_format
string
default:"avro"
Serialization format. Supported values: avro or json.
​
schema_registry_url
string
URL of the Schema Registry. Required when using avro format, optional for json format.

​
Features

  • Multiple Formats: Choose between Avro (binary) or JSON serialization
  • Auto Schema Registration: Schemas are automatically registered with Schema Registry (Avro only)
  • Avro Encoding: Efficient binary serialization with schema evolution support
  • JSON Encoding: Human-readable format without Schema Registry dependency
  • Operation Headers: _gs_op column is included as a message header (dbz.op)

​
Examples

​
Avro format (with Schema Registry)

sinks:
  kafka_output:
    type: kafka
    from: enriched_events
    topic: processed.events
    topic_partitions: 10
    data_format: avro
    schema_registry_url: http://schema-registry:8081

​
JSON format (no Schema Registry required)

sinks:
  kafka_output:
    type: kafka
    from: enriched_events
    topic: processed.events
    topic_partitions: 10
    data_format: json