{"id":15825,"date":"2019-01-25T18:04:00","date_gmt":"2019-01-26T02:04:00","guid":{"rendered":"https:\/\/devwww.3cloudsolutions.com\/post\/kappa-architecture-a-different-way-to-process-data-2\/"},"modified":"2024-01-08T14:36:38","modified_gmt":"2024-01-08T22:36:38","slug":"kappa-architecture-a-different-way-to-process-data","status":"publish","type":"post","link":"https:\/\/3cloudsolutions.com\/resources\/kappa-architecture-a-different-way-to-process-data\/","title":{"rendered":"Kappa Architecture: A Different Way to Process Data"},"content":{"rendered":"

Kappa architecture proposes an immutable data stream as the primary source of record. Unlike lambda, kappa mitigates the need to replicate code in multiple services. In my last post, I introduced the lambda architecture tooling options available in Microsoft Azure<\/a>, sample reference architectures, and some limitations. In this post, I\u2019ll discuss an alternative Big Data workload pattern: kappa architecture.<\/p>\n

\"Two<\/p>\n

<\/p>\n

Below, I\u2019ll give an overview of what kappa is, discuss some of the benefits and tradeoffs of implementing kappa versus lambda in Azure, and review a sample reference architecture. Finally, I\u2019ll offer some added considerations when implementing enterprise-scale Big Data architectures.<\/p>\n

Kappa Architecture: the Immutable, Persisted Log<\/h2>\n

Kappa architecture, attributed to Jay Kreps, CEO of Confluent, Inc.<\/a> and co-creator of Apache Kafka<\/a>, proposes an immutable data stream as the primary source of record, rather than point-in-time representations of databases or files. In other words, if a data stream containing all organizational data can be persisted indefinitely (or for as long as use cases might require), then changes to code can be replayed for past events as needed. This allows for unit testing and revisions of streaming calculations that lambda does not support. Kappa architecture also eliminates the need for a batch-based ingress process, as all data are written as events to the persisted stream. Kappa architecture is a novel approach to distributed-systems architecture, and I personally enjoy the design philosophy behind it.<\/span><\/p>\n

Apache Kafka<\/span><\/h2>\n

Kafka is a streaming platform purposefully designed for kappa, which supports time-to-live (TTL) of indefinite time periods. Utilizing log compaction on the cluster, the kafka event stream can grow as large as you can add storage. There are petabyte-sized (<\/span>imagine the U.S. Library of Congress<\/a>) kafka clusters in production today. This sets kafka uniquely apart from other streaming and messaging platforms because <\/span>it can replace databases as the system of record<\/strong>. Here are a few fascinating write-ups on kafka\u2019s capabilities:<\/span><\/p>\n