1. Kubernetes Native Change Data Capture Conor Gallagher

2. What problem is being solved and why?

3. Problem Statement Teams across Zalando have adopted streaming architectures, where events are published to our event bus (Nakadi) for entities they manage in data stores (e.g. PostgreSQL or DynamoDB). Ensuring data store writes and event publishing is atomic is known as the Distributed Commit Problem. Until recently, it had no standard solution at Zalando. 3

4. Goals Standardise event generation across Zalando Solve the distributed commit problem centrally, so builders can focus on business logic in their applications Define a Kubernetes Custom Resource Definition to insulate builders from the the complexities of Change Data Capture 4

5. Change Data Capture (CDC)

6. Anatomy of a Postgres Change Record • before is an optional field that if present contains the state of the row before the event occurred. • after is an optional field that if present contains the state of the row after the event occurred. 6

7. Fabric Event Streams (FES)

8. FES - Overview A Kubernetes native Change Data Capture (CDC) eventing solution, consisting of two components: 1. FES Operator - The Kubernetes control-plane for FES, containing the FabricEventStream Custom Resource Definition (CRD) 2. FES Connector: ○ For Postgres, the connector uses Debezium to connect via Logical Replication to a Postgres databases for CDC ○ For DynamoDB, the connector uses the Kinesis Client Library (KCL) to connect to a DynamoDB Stream 8

9. FES - Architecture 9

10. FES - Custom Resource Definition Inspired by Akka Streams, the FabricEventStream CRD is modelled around Sources, Flows, Sinks, and Recovery: 10 ● Source - A stage with exactly one output ● Flow - An optional stage which has exactly one input and output. The Flow connects the Source and Sink by transforming the events flowing through it ● Sink - A stage with exactly one input. The sink defines the terminal point of the stream, a Nakadi event type or SQS queue to publish events to ● Recovery - Defines how the stream should handle bad events

11. FES - Resource Example 11 spec: applicationId: my-application eventStreams: - source: type: PostgresLogicalReplication jdbcConnection: slotName: fes jdbcUrl: "jdbc:postgresql://..." table: name: my_events_outbox flow: type: PostgresWalToGenericNakadiEvent payloadColumn: "my_event_payload" sink: type: Nakadi eventType: "my-important-business-events" recovery: type: DeadLetter sink: type: SqsStandard queueName: "my-dead-letter-queue"

12. SQS / Nakadi FES Postgres Connector

13. SQS / Nakadi DynamoDB Stream FES DynamoDB Connector

14. FES - Standardised Telemetry 14

15. Patterns

16. Transactional Outbox Pattern 16

17. Transactional Outbox Pattern spec: applicationId: my-application eventStreams: - source: type: PostgresLogicalReplication jdbcConnection: slotName: fes jdbcUrl: "jdbc:postgresql://..." table: name: my_events_outbox flow: type: PostgresWalToGenericNakadiEvent sink: type: Nakadi eventType: "my-important-business-events" 17

18. AWS Lambda Flow 18

19. AWS Lambda Flow spec: applicationId: my-application eventStreams: - source: type: DynamoDbStreamsSubscription subscription: leaseTableName: my_lease_table streamArn: "arn:aws:dydb:eu-1:343:my_table/stream/2022-05-" filter: "[?(@.OldImage.Status.Id != @.NewImage.Status.Id)]" flow: type: AwsLambdaConversion lambdaFunctionArn: "arn:aws:lambda:eu-1:343:func:cdc-converter" sink: type: SqsFifo queueName: "my-cdc-queue" recovery: 19 type: None

20. Questions?