Event driven architecture with Kafka & Asp.NET

As our software evolves, it becomes more important to have an architecture capable of handling a growing customer base and increasing complexity ideally designed in a way that won’t repeat mistakes of the past such as tightly coupled codebases and a monolithic database. To that end, it matters how we connect our Apps and APIs and how we scale them for cost-effective future growth. In this post, I cover how Pellerex can leverage event-driven architecture and Kafka as tool to achieve those goals.

The Need

Looking at the way software assets are structured in the majority of organisations, you could find a lot of entry points like Apps and APIs which would ‘orchestrate’ how different services are tied together to generate a response to a request initiated by a ‘button click’. This orchestration however creates a lot of tight coupling between all those services in a way that adding or removing services to/from this chain as a result, will require (breaking?) code changes across one or many services.

On the other hand, there are many services which would abstract the data they operate on (in the form of a micro-db or a monolith), to produce the data for their outside world. This means services would need to depend on each other to fetch the data to produce the results in response to customers’ requests, and this will create even more coupling across the system.

We should also mention the giant monolithic database which has been created at the core of the organisation, which is hard (impossible?) to decouple, transform and modernise. On top of that the data we maintain there is mutable, meaning it is challenging to find out how we got to the data we have at hand today, as there is no versioning behind it.

And not to mention the evil batch jobs whenever there is a need for asynchrony which generally happen over working week nights and cause delays for operations that need to be completed fast but in an asynchronous way. Quite recently finance sector were forced to report trades within a 1-minute time span, and it only needs an over-night batch job to force them to use hacky solutions to achieve that.

Pellerex will use event-driven architecture with the aim to address some of these challenges (beside the existing request-response mode) by reversing how we look at data, service coupling, and versioning where necessary, and we believe Kafka is an effective vehicle to make this goal happen.

Opportunities

With event driven architecture here are some of the opportunities we have in our system design, which I will explain deeper a bit later:

1. Use Events as Notification to Collaborate: this will transform how our services collaborate to move away from central chain command to an stream of events where needed
2. Event carried state transfer: to carry state along the events and reduce remote service calls to fetch data necessary to complete the job intended to be done by event consumers
3. Event sourcing: rebuilding and replaying logs as an opportunity to fix side effects of past errors, create alternate states, or as a mean to recover from disasters or perform audits
4. CQRS: to segregate commands from queries leading to potential increased system efficiency

1. Use Events as Notification to Collaborate

Serial Service Execution using orchestration will become Serial Event Creation where one service responds to events created by another. Chain of command becomes stream of events which leads to more loose coupling, higher delivery velocity and less breaking changes across the system. While we follow Single Writer Principle to allocate the responsibility of certain events to a specific service, we will set the system up for future scalability and distribution. Our ecosystem becomes pluggable, and adding or removing functions in response to certain events won’t necessarily require change in existing services. Hence we will go from:

To a state of collaboration like:

Such powerful capability comes with a drawback that it will be harder to track the flow of information, and we need to trace events to understand how the system works in its entirety, which could be time consuming as the system grows.

2. Event carried state transfer

In the first model, there are scenarios that tha target service would need extra information to be able to perform its job, be it an email address to send a message to customer, or a unit price to calculate the total. If notifications just carry the facts regarding the event, the target service has to make extra calls to obtain that information and finish its job. The alternative is however, that the event producer augment the event and transfer the state with the all that information before passing the event to the consumer, and hence the target service would have all the information to run its job to completion.

The augmentation process could be done using the original service or a separate service to hold the state inside Kafka.

While this carries the necessary data to finish the job by the target service, it might create consistency issues at different points of processing the distributed data (Eventual Consistency).

3. Event Sourcing

The ability to replay and recreate the history through the steps recorded in Kafka, in some business cases is a powerful feature with notable examples in Audit, Versioning Control, and Change Management.

4. CQRS

Command and Query Responsibility Segregation can be achieved using Materialised/Computed Views & KSQL, to automatically update the view when there is a change in the underlying data behind the view. There won’t be a need to take an action and update the view data manually, as the view (or all the dependant views) are recalculated whenever their underlying data changes.

Business Scenarios to Handle with Kafka & Pellerex

There are various business scenarios that leveraging Kafka helps design and implement a more robust system:

• Need for Asynchrony: when systems in charge of processing the requests are not available at the time request is received, or are under such heavy load which would need them to process those requests at their own pace. In these situations those services will pull Kafka topics whenever they are ready to do so.
• Examples of such cases are mobile applications generating millions of events like app open, app close, app crash, etc.
• Other examples are cases where the end to end transaction can not be fulfilled immediately such as those containing audit or shipment steps outside digital channels.
• Event Storage: to ingest heavy event input (trillions) and store them as the “source of truth” permanently or for a short period, with the ability to update the views any time there is a change in the data behind those views.
• Messaging: to pass the events from the producers to consumers in a fault-tolerant and reliable way.
• Resource Intensive Processes: where generating a response might take enough time that delaying the process would lead to a poor customer experience.
• Plug and Play: complex business scenarios where you would need to add or remove services to the workflow without impacting existing services.
• CQRS: when there is a need to make certain views readily available to the rest of the application as soon as the underlying data changes. This will enable the data to move around the system instead of remaining at rest and hidden behind services.
• Message Order Guarantee: where there is a need for the messaging system to deliver messages in the order received.
• Connect to Legacy Systems: to make the data buried in legacy systems available to the rest of the modern stack through Kafka Connect

Implementation

Kafka implementation in the Cloud has a few aspects, which I will cover in depth below:

1. Standing up an instance of Kafka
2. Event Producers
3. Event Consumers

Kafka Instance

1. Local Machine

I can install Kafka’s components such as ZooKeeper, Broker, Schema Registry, etc individually, or I can just use the below docker-compose file and bring everything up using one command on my local machine:

docker-compose -f https://raw.githubusercontent.com/confluentinc/cp-all-in-one/6.2.1-post/cp-all-in-one/docker-compose.yml up

This will have an instance of Kafka up and running at localhost:9092 which I then can connect to using producers and consumers.

2. Cloud Managed Service

Confluent Cloud is the managed Kafka offering you could set up and use right out of the box in the Cloud. The registration process can be started from Azure Portal, by choosing Confluent Cloud from the market place.

By choosing Pay as you Go, your cost will be minimised.

The registration process is quite straightforward, and once you are done, it will give you a dashboard through which you can configure your Kafka cluster.

Event Producers

For the sake of this post, I am using .Net APIs as the event producers. I will create an injectable service, to producer the events and push them to Kafka. Here is the service:

Please consider the below points about this service:

• Message Format: I am using json as the message format when I push them to Kafka. I could even use binary, but I’d like the message to be readable in Kafka queues.
• SaslSsl: I have used SaslSsl to connect to Confluent Cloud, and as you can see all I need to pass is a key, password, and a server address to configure the connection.
• Injection: You could then inject this service whenever you would need to send messages to Kafka. You would however need to configure the dependency like this, in your startup file:

• Here are the links to learn more on how I manage application settings and application secrets using KeyVault. Also here are the links for standalone containerised API configuration and secret management and for when you want to run them in a Kubernetes Cluster.

Event Consumers

Once you were able to producer messages to certain topics, you would need your consumers to be able to read them. The reading process is a bit trickier than the producing, as you would need to poll Kafka topics for the messages you are interested in. To create that kind of process, I’ll need to have a long running service in my Asp.NET APIs as long as my API is running. A good and simple choice for that matter would be Asp.NET Hosted Services.

Here is the complete code for a Kafka topic consumer for Asp.NET APIs:

Please consider the below notes:

• Consumer Config: Instead of Producer Config, I am using ConsumerConfig session to provide the Kafka consumer with the authentication details to Kafka cluster.
• Subscribe: I will then need to subscribe to a certain topics to receive an update whenever an event comes through.
• Consume: Consume method is a blocking method, meaning it blocks the thread until a message is received. Once I got the message, I will then deserialise it, and perform the main action, which in my case is sending an email.
• While Loop: As I need to poll to receive new events, I have put the Consume in a while loop. Every time it receives a message, it will unblock the thread, do the work, and it goes back to the same Consume method, waiting for the next event to happen.
• Thread: As the while look blocks the main thread, I have run it in a separate thread using Task.Run, so the host of the application is not affected.

Also here is the code behind MyBackground service, which is just a HostedService:

And at the end, I’d need to inject this into my DI pool:

services.AddHostedService<KafkaHostedService>();

Pellerex: Foundation for Your Next Enterprise Software

How are you building your current software today? Build everything from scratch or use a foundation to save on development time, budget and resources? For an enterprise software MVP, which might take 8–12 months with a small team, you might indeed spend 6 months on your foundation. Things like Identity, Payment, Infrastructure, DevOps, etc. they all take time, while contributing not much to your actual product. These features are needed, but they are not your differentiators.

Pellerex does just that. It provides a foundation that save you a lot development time and effort at a fraction of the cost. It gives you source-included Identity, Payment, Infrastructure, and DevOps to build Web, Api and Mobile apps all-integrated and ready-to-go on day 1.

Check out Pellerex and talk to our team today to start building your next enterprise software fast.