Integrating .NET Core with Apache KAFKA

Integrating .NET Core with Apache Kafka is a powerful way to build real-time, scalable, and high-performance applications. Here are some key steps and considerations for getting started:


Setting Up Kafka

Install Kafka 

You can download Kafka from the official website and follow the installation instructions.

Run Kafka and Zookeeper

Use Docker or run them locally. 

For Docker, you can use a docker-compose.yml file to set up Kafka and Zookeeper.


Creating a .NET Core Project


Create a new .NET Core project

Use Visual Studio or the .NET CLI to create a new project.


dotnet new console -n KafkaExample
cd KafkaExample


Adding Kafka Dependencies


Add Confluent.Kafka

This is a popular .NET client for Kafka.


dotnet add package Confluent.Kafka


Producing Messages


Create a Producer

Write code to produce messages to a Kafka topic.


using Confluent.Kafka;

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
	​try
	​{
	​	​var dr = await producer.ProduceAsync("test-topic", new Message<Null, string> { Value = "Hello Kafka" });
	​	​Console.WriteLine($"Delivered '{dr.Value}' to '{dr.TopicPartitionOffset}'");
	​}
	​catch (ProduceException<Null, string> e)
	​{
		​​Console.WriteLine($"Delivery failed: {e.Error.Reason}");
	​}
}


Consuming Messages


Create a Consumer

Write code to consume messages from a Kafka topic.


using Confluent.Kafka;

var config = new ConsumerConfig
{
	​GroupId = "test-consumer-group",
	​BootstrapServers = "localhost:9092",
	​AutoOffsetReset = AutoOffsetReset.Earliest
};

using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
	​consumer.Subscribe("test-topic");
	​
	​try
	​{
	​	​while (true)
	​	​{
	​	​	​var cr = consumer.Consume();
	​	​	​Console.WriteLine($"Consumed message '{cr.Value}' at: '{cr.TopicPartitionOffset}'.");
	​	​}
	​​}
	​catch (ConsumeException e)
	​{
	​	​Console.WriteLine($"Error occurred: {e.Error.Reason}");
	​}
}


Running the Application


Start Kafka and Zookeeper

Ensure Kafka and Zookeeper are running.


Run your .NET Core application

Use the .NET CLI or Visual Studio to run your application.