Apache Kafka for Developers – Real-time Data Streaming, Messaging & Event-driven Architecture

In this course, you'll get a thorough and hands-on introduction to Apache Kafka — the leading platform for real-time data streaming, messaging, and event-driven architectures. You'll learn how Kafka works, how to build scalable streaming applications, and how to integrate Kafka into modern data platforms and microservices. Through hands-on labs, you'll work with producers, consumers, connectors, and stream processors — designing reliable, high-throughput data pipelines in Python or Java.

What you'll learn

• The core principles of Apache Kafka and why it's ideal for real-time data streaming.
• Key concepts: producers, consumers, topics, partitions, offsets, and consumer groups.
• Architectures for scalability, reliability, and fault tolerance.
• Working with Kafka Connect for data movement and database integrations.
• Real-time stream processing with the Kafka Streams API.
• Kafka within microservices and event-sourcing patterns.
• Advanced concepts such as exactly-once semantics, log compaction, and Kafka Raft.

After this course you'll be able to:

• Set up and manage Kafka clusters and clients independently.
• Build robust streaming applications with producers, consumers, and Kafka Streams.
• Integrate Kafka into existing systems using best-practice architectures.
• Move data efficiently with Kafka Connect and schema management.
• Apply event-driven design within microservice environments.
• Design scalable, production-ready real-time data pipelines.

Who is this for

• Data engineers and data architects.
• Software engineers working with microservices or event-driven architectures.
• Organisations looking to use Kafka for real-time analytics, data movement, or integrations.
• Anyone who wants to master real-time data processing with Apache Kafka.

Prerequisites

• Basic programming knowledge (e.g. Python or Java).
• Familiarity with distributed systems or data processing is helpful.
• No prior Kafka experience required.

Programme

Part 1 – Introduction to Kafka and Real-time Streaming

• Producers, consumers, topics, and partitions.

Part 2 – Architecture & Integration

• Scalability, reliability, replication, and fault tolerance.

Part 3 – Kafka Connect

• Source and sink connectors, database integration, schema evolution.

Part 4 – Kafka Streams

• Stateful processing, windowing, and real-time transformations.

Part 5 – Kafka in Microservices

• Event-sourcing, messaging patterns, and best practices.

Part 6 – Advanced Concepts

• Exactly-once semantics, log compaction, Kafka Raft.

Part 7 – Best Practices & Q&A