Kafka Connect — System Integration Without Code¶

Kafka Connect is a framework for integrating Kafka with external systems without writing code. Source connectors read data into Kafka, sink connectors write from Kafka — hundreds of ready-made connectors.

Kafka Connect Architecture¶

Connect runs as a worker cluster managing connectors. Two types:

• Source connector — reads from an external system into a Kafka topic
• Sink connector — writes from a Kafka topic to an external system

Source Connector — PostgreSQL to Kafka¶

{
  "name": "postgres-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "app",
    "topic.prefix": "cdc",
    "table.include.list": "public.orders"
  }
}

Sink Connector — Kafka to S3¶

{
  "name": "s3-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "topics": "orders",
    "s3.bucket.name": "data-lake",
    "s3.region": "eu-central-1",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "flush.size": 10000,
    "rotate.interval.ms": 3600000
  }
}

Transforms (SMT)¶

• Single Message Transforms — simple transformations without code
• Field renaming, masking, filtering, routing

Summary¶

Kafka Connect is the easiest way to integrate Kafka with external systems. Hundreds of connectors without code.