Apache Kafka Interview Questions and Answers 2026 (55 Questions)

Kafka interview questions in 2026 test whether you can run Apache Kafka as a reliable event backbone, not whether you can define a topic. Interviewers want to hear how you size partitions, choose acks and min.insync.replicas, explain KRaft now that ZooKeeper is gone, move consumer groups to the new rebalance protocol, use share groups for queue-style work, and debug lag, duplicates and hot partitions under pressure. This guide collects 55 high-value questions with model answers, from fundamentals to 11 production scenarios, including Kafka's growing role in real-time AI systems.

How to use this guide

The basics of topics, partitions, offsets, consumer groups, exactly-once and log-based CDC are covered in our data engineering interview questions. This page goes deeper into Kafka itself: the 4.x platform, producer and consumer internals, storage, operations, the ecosystem and scenarios. Answers reflect the Apache Kafka 4.x documentation (the 4.3 line at the time of writing); where a feature's status changed between releases, the answer says so.

• Freshers and early-career engineers: replication and ISR, acks, KRaft basics, retention versus compaction, partition count trade-offs and offset commit timing.
• Mid-level data and platform engineers: batching, idempotence, transactions, the classic and new consumer rebalance protocols, lag, dead-letter topics and Kafka Connect.
• Senior engineers and architects: leader election and Eligible Leader Replicas, tiered storage, ZooKeeper-to-KRaft migration, security, geo-replication, managed-service trade-offs, share groups and Kafka in AI architectures.

• Fundamentals and architecture (Q1–Q10)
• Producers: batching, idempotence and transactions (Q11–Q15)
• Consumers, groups and share groups (Q16–Q24)
• Storage, replication and operations (Q25–Q34)
• Connect, Streams, schemas and managed Kafka (Q35–Q40)
• Kafka for AI systems (Q41–Q44)
• Real-world scenario questions (Q45–Q55)
• Key takeaways
• Interview preparation checklist
• FAQ

Fundamentals and architecture

1. When is Kafka the right tool, and when is it the wrong one?

Answer: Kafka is the right tool when many independent consumers need the same ordered stream of events, when you need to replay history, and when throughput is high enough that a durable, partitioned log pays for its operational cost. Typical fits are event-driven microservices, CDC from operational databases, clickstream and IoT ingestion, log and metric pipelines, and feeding stream processors or a lakehouse.

It is the wrong tool for request-response calls that need an immediate answer, for small systems where a managed queue or a database table would do, for querying data by arbitrary fields (Kafka is not a database you search), and for very large binary payloads. Store a document in object storage and publish a reference to it instead.

Interview tip: Saying when you would not use Kafka shows judgement. Mention that Kafka 4.2 added production-ready share groups (Q23), which narrows the gap with traditional queues for work-distribution use cases.

2. How does a producer decide which partition a record goes to?

Answer: If the record names a partition, that partition is used. If it has a key, the default partitioner hashes the key, so the same key always lands on the same partition while the partition count stays the same. If there is no key, the default logic sticks to one partition until roughly batch.size bytes have been produced, then switches. This "sticky" behaviour builds fuller batches than the old per-record round-robin and cuts request overhead.

Two consequences matter in interviews. First, ordering holds only per partition, so keys are how you get per-entity ordering. Second, adding partitions changes the hash mapping, so new records for a key may go to a different partition than old ones.

3. Explain leaders, followers, the ISR and the high watermark.

Answer: Each partition has one leader replica that serves writes (and by default reads) and follower replicas that fetch from the leader. The in-sync replica set (ISR) is the leader plus the followers that are caught up within replica.lag.time.max.ms. The high watermark is the offset up to which every ISR member has the data. Consumers only see records below the high watermark, which is why a record acknowledged with acks=all survives the loss of the leader: any ISR member that becomes leader already has it.

If a follower falls behind, it is removed from the ISR (an "ISR shrink"); when it catches up it rejoins (an "ISR expand"). Frequent shrink and expand cycles are a classic sign of overloaded brokers, slow disks or network trouble.

4. How do acks and min.insync.replicas work together?

Answer: acks is a producer setting that says how many acknowledgements the producer waits for. min.insync.replicas is a topic or broker setting that says how many in-sync replicas must exist for an acks=all write to succeed.

Setting	Behaviour	Use
acks=0	Fire and forget; no acknowledgement, retries cannot help	Lossy metrics where speed matters more than completeness
acks=1	Leader writes locally and replies; data is lost if the leader dies before followers copy it	Rarely justified today
acks=all (default in 4.x)	Leader waits for all current ISR members	Default for business data
acks=all + RF 3 + min.insync.replicas=2	Writes continue with one replica down and fail fast with two down	The common durable baseline

The subtle point: acks=all alone is not enough. If the ISR shrinks to just the leader, "all" means one copy. min.insync.replicas=2 makes the producer receive NotEnoughReplicas errors instead of silently writing a single copy. Setting it equal to the replication factor is a mistake, because any single broker restart then blocks writes.

5. What is KRaft, and what changed in Kafka 4.0?

Answer: KRaft is Kafka's built-in Raft-based consensus mode. Cluster metadata (topics, partitions, configs, ACLs, leadership) is stored in an internal metadata log replicated across a quorum of controller nodes, instead of in Apache ZooKeeper. One controller is active; the others are hot standbys that already have the metadata, so failover is fast, and brokers receive metadata changes as a stream of events rather than through bulk RPCs.

Kafka 4.0 (March 2025) was the first major release to run entirely without ZooKeeper. ZooKeeper mode and the in-place ZooKeeper migration tooling were removed, so clusters still on ZooKeeper must first migrate on a 3.x bridge release (3.9 is the final 3.x line) and then upgrade to 4.x. Kafka 3.9 also added dynamic controller quorums (KIP-853), letting you add and remove controllers without downtime.

6. What are broker, controller and combined roles in KRaft, and how many controllers do you run?

Answer: The process.roles setting decides what a node does: broker (stores and serves data), controller (participates in the metadata quorum), or broker,controller (combined). Combined mode is simple and suits development or small clusters, but the docs note the controller is less isolated: you cannot roll or scale controllers separately from brokers, and a busy broker can starve the controller.

Production clusters usually run dedicated controllers. A majority must be alive, so three controllers tolerate one failure and five tolerate two. Running an even number buys nothing. Clients never talk to controllers; they connect to brokers via bootstrap.servers.

7. Explain at-most-once and at-least-once from the consumer's point of view.

Answer: It comes down to when you commit offsets relative to processing. Commit before processing and a crash loses the uncommitted work: at-most-once. Process, then commit, and a crash after processing but before the commit replays those records: at-least-once. With enable.auto.commit=true (the default), the consumer commits in the background on the next poll, so records you have polled but not finished can be committed, and a crash can lose them if you hand work to other threads.

Production pattern: turn off auto-commit for anything important, process a batch, write results idempotently, then commit synchronously (or asynchronously with a final synchronous commit on shutdown). For exactly-once inside Kafka, use transactions (Q13).

8. What is the difference between delete retention and log compaction?

Answer: cleanup.policy=delete (the default) removes whole log segments once they exceed retention.ms or retention.bytes. It suits event streams where old history expires. cleanup.policy=compact keeps at least the latest record for each key and discards older values, so the topic becomes a durable changelog of current state, which suits CDC tables, configuration, customer profiles and Kafka Streams changelogs. You can combine both (compact,delete) to keep the latest value per key but still drop everything older than the retention period.

A record with a key and a null value is a tombstone: it marks the key as deleted, and the tombstone itself is removed later, after delete.retention.ms.

Real-world example: Consider a retailer keeping a compacted product-price topic keyed by SKU. A new pricing service can rebuild its whole cache by reading the topic from the beginning, without querying the source database.

9. Why do teams use a schema registry with Kafka?

Answer: Kafka stores bytes and does not validate payloads. A schema registry (Confluent Schema Registry and several compatible alternatives exist) stores versioned Avro, Protobuf or JSON Schema definitions. Producers register or look up a schema and prepend a small schema ID to each message; consumers fetch the schema by ID to deserialise. The registry enforces compatibility rules when a new version is registered, so a producer cannot ship a change that breaks existing consumers.

The benefits are smaller messages than self-describing JSON, safe evolution, and a central catalogue of event contracts. Schema registries are separate components, not part of Apache Kafka itself.

10. How do you choose the number of partitions for a topic?

Answer: Partitions are the unit of parallelism. A consumer group cannot have more active consumers than partitions, so start from the throughput you need divided by what one consumer can sustain, then add headroom for growth. Because increasing partitions later remaps keys and breaks per-key ordering for in-flight data, it is better to start with some headroom on keyed topics.

More partitions are not free: each one adds open files, replication traffic, metadata and recovery work, and very wide topics make rebalances and leader elections heavier. Too few partitions cap consumer parallelism and concentrate load. A measured approach is to benchmark one partition's producer and consumer throughput with the tools that ship with Kafka (kafka-producer-perf-test.sh, kafka-consumer-perf-test.sh), then size from real numbers rather than rules of thumb.

Producers: batching, idempotence and transactions

11. How do batching and compression work in the producer?

Answer: The producer groups records per partition into batches in memory. A batch is sent when it reaches batch.size bytes or when linger.ms expires, whichever comes first. Since Kafka 4.0 the default linger.ms is 5 ms (previously 0), because the docs found the efficiency gain from larger batches usually gives similar or lower latency overall. Compression (compression.type: gzip, snappy, lz4, zstd) is applied per batch, so fuller batches compress better.

The trade-off is latency versus throughput. Raising linger.ms and batch.size increases throughput and lowers broker request load at the cost of a few milliseconds. buffer.memory bounds total buffered data; when it fills, send() blocks up to max.block.ms, which is often the first symptom of a slow or unreachable cluster.

12. How does the idempotent producer work?

Answer: With idempotence on, the broker assigns the producer a producer ID, and the producer numbers each batch per partition with a sequence number. If a retry resends a batch the broker already wrote, the broker recognises the sequence and discards the duplicate. That removes the classic "network timeout, retry, duplicate" problem and preserves ordering across retries.

In the 4.x docs, enable.idempotence defaults to true, provided no conflicting settings exist: it requires acks=all, retries above zero and max.in.flight.requests.per.connection of 5 or less (the broker tracks only the last five batches per producer). If you set a conflicting value without explicitly enabling idempotence, it is silently disabled, a common cause of unexpected duplicates. If you enable it explicitly with conflicting values, the client throws a ConfigException.

Idempotence protects one producer session against its own retries. It does not deduplicate two different producer instances sending the same business event, or an application that calls send() twice.

13. How do Kafka transactions work?

Answer: A transactional producer sets transactional.id, a stable name that survives restarts. On initTransactions(), the transaction coordinator bumps an epoch for that ID, which fences any older "zombie" instance still running. The producer then calls beginTransaction(), writes to any number of partitions, optionally adds consumer offsets with sendOffsetsToTransaction(), and calls commitTransaction() or abortTransaction(). The coordinator writes commit or abort markers into every partition involved.

Consumers with isolation.level=read_committed read only up to the last stable offset (LSO) and skip aborted records; the default is read_uncommitted. A long-open transaction therefore holds back read_committed consumers, which shows up as lag. Kafka 4.0 completed the second phase of KIP-890 (transactions server-side defence), which reduces zombie-transaction risks during producer failures.

poll(input) -> beginTransaction()
  -> send(output records)
  -> sendOffsetsToTransaction(input offsets)
  -> commitTransaction()
consumer (read_committed) sees all or nothing

14. Where does Kafka exactly-once stop, and how do you get end-to-end correctness?

Answer: Kafka's exactly-once covers the read-process-write loop when both input and output are Kafka topics: Kafka Streams with its exactly-once processing setting (exactly_once_v2) or a transactional producer that commits output and offsets atomically. Once a side effect leaves Kafka, such as a database write, an email, a payment API call or a vector-store upsert, Kafka cannot roll it back.

For external sinks you combine at-least-once delivery with one of:

• Idempotent writes keyed on a unique event ID (upsert or insert-if-absent).
• Storing the consumed offset in the same database transaction as the result, and seeking to it on restart.
• Connectors or engines with transactional sinks that commit output and progress together.

15. How do retries, delivery.timeout.ms and ordering interact?

Answer: Modern producers retry transient errors automatically; the docs recommend controlling retry behaviour with delivery.timeout.ms, the upper bound on the time between send() returning and the callback reporting success or failure, rather than tuning retries. When the timeout expires, the callback receives an exception, and your code must decide: log and drop, route to a fallback, or stop the application.

Ordering: with idempotence enabled, ordering is preserved even with up to five in-flight requests. With idempotence disabled and more than one in-flight request, a failed first batch that is retried after a successful second batch reorders records. The most common real bug is ignoring the send callback, so failed sends vanish without trace. Always handle the callback or check the returned future.

Consumers, groups and share groups

16. How do consumer groups store and manage offsets?

Answer: Each group has a group coordinator, a broker that leads the partition of the internal __consumer_offsets topic owning that group. Members commit offsets to the coordinator, which writes them to __consumer_offsets (a compacted topic). The committed offset is the position of the next record to read, not the last one processed, an off-by-one detail interviewers like to check.

When a consumer starts without a committed offset, or its committed offset has been deleted by retention, auto.offset.reset decides where to start: earliest, latest, none (throw an exception), or, since Kafka 4.0, by_duration:<ISO-8601 duration> to start a set time back from now (KIP-1106). You can inspect and reset group offsets with kafka-consumer-groups.sh while the group is inactive.

17. Explain the classic rebalance protocol, cooperative rebalancing and static membership.

Answer: In the classic protocol, a group leader (one of the consumers) computes assignments using a client-side assignor. Eager assignors revoke all partitions from every member and reassign, a "stop-the-world" pause. The cooperative sticky assignor rebalances incrementally: members keep partitions that do not move and give up only those being reassigned, at the cost of extra rebalance rounds.

Static membership (group.instance.id) gives each consumer a persistent identity. A restarted member with the same ID gets its partitions back without triggering a rebalance, as long as it returns before the session timeout. It is valuable for rolling deploys of stateful consumers on Kubernetes, where each pod can use its StatefulSet ordinal as the instance ID.

18. What is the new consumer rebalance protocol (KIP-848), and what is its status?

Answer: KIP-848 is the next generation of the consumer rebalance protocol. It became generally available (production-ready) in Kafka 4.0. Instead of clients synchronising through join and sync barriers, the broker-side group coordinator computes assignments and reconciles each member incrementally through regular heartbeats. One slow or restarting member no longer pauses the whole group, which shortens rebalances and improves stability in large groups.

• Server: enabled by default from 4.0, governed by the group.version feature. Assignors are server-side (uniform, the default, and range), configured via group.consumer.assignors; heartbeat and session timeouts are controlled by broker group configs.
• Client: the Java consumer must opt in with group.protocol=consumer; the default is still classic in 4.x. With the new protocol, partition.assignment.strategy, session.timeout.ms and heartbeat.interval.ms are no longer used on the client, and regex subscriptions are evaluated on the server (RE2J syntax).
• Roadmap (per the docs, KIP-1274): the consumer is expected to default to the new protocol in 5.0, and to support only it in 6.0. Kafka 4.3 already logs a recommendation when a consumer starts with the classic protocol.

Interview tip: Mention the limitation: custom client-side assignors are not supported under the new protocol. Teams with bespoke assignment logic must plan for that before migrating.

19. How would you migrate an existing consumer group to the new protocol?

Answer: The docs describe two paths. Offline: stop every consumer in the group; empty groups convert automatically, so restarting them with group.protocol=consumer switches the group. Online: roll consumers one by one with the new setting. When the first new-protocol member joins, the group converts and the coordinator interoperates with the remaining classic members. This works only if the classic group uses an assignor that does not embed custom metadata. Downgrade is the reverse: when the last new-protocol member leaves, the group converts back.

Checklist I would follow: confirm brokers are on 4.x with the feature enabled; map assignors (cooperative sticky and round robin map to uniform, range maps to range); remove client configs that no longer apply; check that non-Java clients your teams use support the protocol; roll one service in staging; and watch rebalance and lag metrics before moving critical groups.

20. What do max.poll.interval.ms and max.poll.records control?

Answer: max.poll.interval.ms is the maximum gap between calls to poll(). If processing a batch takes longer, the consumer is considered failed and its partitions are reassigned, which causes a rebalance and duplicate processing when another member re-reads uncommitted records. max.poll.records caps how many records one poll() returns, which bounds the work per loop.

When per-record work is slow (calling an LLM, an external API or a slow database), lower max.poll.records, process in parallel with care for per-partition ordering, or use pause() and resume() so the poll loop keeps running while workers finish. Raising max.poll.interval.ms hides the problem and slows failure detection. For one-at-a-time slow work, share groups (Q23) may be a better fit.

21. How do you measure and alert on consumer lag?

Answer: Lag per partition is the log-end offset minus the group's committed offset (or current position). Sources: kafka-consumer-groups.sh --describe, the consumer's own records-lag and records-lag-max metrics, the Admin API, or an external lag exporter for your monitoring stack.

Offset lag alone is misleading: 50,000 records is seconds on a busy topic and days on a quiet one. Good practice is to estimate time lag (age of the oldest unconsumed record), alert on sustained growth rather than spikes, and alert separately on groups that stop committing entirely. For read_committed consumers, remember lag is measured against the last stable offset, so open transactions affect it.

22. How do you handle poison messages and retries without blocking a partition?

Answer: A record that always fails (bad schema, unexpected nulls, a bug) will block its partition forever if you retry in place, because the consumer cannot commit past it. The usual pattern:

1. Retry transient errors a few times in memory with backoff.
2. For non-transient failures, publish the record, plus error details and the original topic, partition and offset in headers, to a dead-letter topic, then commit and move on.
3. For retriable-later errors, use one or more delayed retry topics consumed with a deliberate delay.
4. Monitor dead-letter volume and build a replay tool.

Ordering caveat: moving a record to a retry topic means later records for the same key can overtake it. For strictly ordered entities, pause that key's processing or stop the partition and alert instead. Kafka Connect has built-in dead-letter queue settings, and Kafka 4.2 added DLQ support to Kafka Streams exception handlers (KIP-1034).

23. What are share groups (Queues for Kafka, KIP-932), and what is their status?

Answer: A share group is a new kind of group, an alternative to consumer groups, in which multiple consumers cooperatively consume records from the same partitions without each partition being assigned to only one consumer. Records are acquired with a time-limited lock and acknowledged individually, and the broker counts delivery attempts. That gives queue-like semantics on ordinary Kafka topics: you can run more consumers than partitions, and a failed record is redelivered to another consumer without blocking the rest.

Status: early access in 4.0, preview in 4.1, production-ready in 4.2. The KafkaShareConsumer acknowledges with ACCEPT, RELEASE (make available again), REJECT (do not redeliver) and, from 4.2, RENEW to extend the lock for long processing. Kafka 4.3 added group-level configs such as share.delivery.count.limit. Share groups use an internal __share_group_state topic that by default needs three brokers.

Interview tip: The docs' own guidance is the key line: use share groups when records are processed one at a time, not as part of an ordered stream. They do not preserve per-key ordering.

24. Share group or consumer group: how do you choose?

Answer: Decide based on ordering and work shape.

Need	Consumer group	Share group
Per-key ordering	Yes, within a partition	No
Consumers beyond partition count	Extra members sit idle	Yes
Per-record acknowledgement and redelivery	No; offsets are a single position	Yes, with delivery counting
Stateful aggregation, CDC apply, joins	Yes	Poor fit
Independent slow jobs (OCR, LLM calls, notifications)	Head-of-line blocking risk	Good fit

Real-world example: Consider an insurer whose claim documents need OCR and an LLM summary, each taking several seconds. With a consumer group, a 12-partition topic caps it at 12 workers, and one stuck document holds back its partition. With a share group, it can scale workers on queue depth, and a failing document is retried and eventually rejected without holding up others.

Storage, replication and operations

25. How does log compaction actually run, and what are its gotchas?

Answer: Log cleaner threads rewrite older segments, keeping only the latest record per key. A partition becomes eligible when its "dirty" (uncompacted) share exceeds min.cleanable.dirty.ratio and records are older than min.compaction.lag.ms, or when dirty records exceed max.compaction.lag.ms. The active segment is never compacted, so recent duplicates per key are normal.

Gotchas interviewers probe:

• Compaction gives eventual "latest per key", not exactly one record per key at any moment. Consumers must tolerate duplicates.
• A consumer lagging more than delete.retention.ms (24 hours by default) can miss tombstones and keep deleted keys in its local state forever.
• Records without keys cannot be produced to a compacted topic.
• A stuck or failed cleaner thread lets compacted topics grow without bound; monitor cleaner metrics.

26. What is tiered storage, and what is its status?

Answer: Tiered storage (KIP-405) gives a cluster two tiers: local broker disks for recent segments and a remote store, such as S3-compatible object storage or HDFS, for completed older segments. Consumers reading recent data still hit local disks and the page cache; reads of old offsets are served from the remote tier transparently. It was announced as production-ready in Kafka 3.9.

Configuration: enable remote.log.storage.system.enable on brokers, plug in a RemoteStorageManager implementation (Apache Kafka does not ship one for a specific cloud store, so you use a plugin), then set remote.storage.enable=true per topic with local.retention.ms or local.retention.bytes for the local tier and retention.ms for the total. Documented limitations include no support for compacted topics.

Benefits: longer retention without buying broker disks, faster broker replacement and rebalancing because less data lives locally, and easier replay for backfills and AI reprocessing.

27. How does leader election work, and what are unclean election and Eligible Leader Replicas?

Answer: When a leader fails, the KRaft controller picks a new leader from the ISR. If no ISR member is alive, Kafka by default waits rather than electing an out-of-sync replica, choosing consistency over availability. Setting unclean.leader.election.enable=true lets an out-of-sync replica become leader, which restores availability but can lose acknowledged records. It is only acceptable for data where uptime matters more than completeness.

Eligible Leader Replicas (KIP-966) refine this. Because the high watermark cannot advance when the ISR is smaller than min.insync.replicas, some replicas that drop out of the ISR are still known to have all committed data. The controller tracks them as ELR and can safely elect one when the ISR is empty. ELR arrived in preview in 4.0 and, per the docs, is enabled by default on new clusters from 4.1 (controlled by the eligible.leader.replicas.version feature).

28. What is rack awareness, and why does it matter in the cloud?

Answer: Setting broker.rack (for example to the availability zone) makes Kafka spread a partition's replicas across racks, so losing one zone does not lose every copy. Consumers can also set client.rack and, with a rack-aware replica selector on the brokers, fetch from a follower in their own zone instead of the leader in another zone.

In AWS, Azure and Google Cloud, cross-zone data transfer is often a noticeable part of the Kafka bill, so follower fetching can reduce cost as well as latency. The trade-off is slightly higher end-to-end latency, because a follower may lag the leader briefly.

29. How do you add partitions, move data and scale a cluster safely?

Answer: Adding partitions is quick (kafka-topics.sh --alter --partitions) but cannot be undone, and it remaps keys. For keyed, ordered topics, either accept a coordinated cut-over or create a new topic with more partitions and migrate consumers.

New brokers receive no existing partitions automatically. You move replicas with kafka-reassign-partitions.sh (or a tool such as Cruise Control), throttling replication so the copy does not starve production traffic, and watch under-replicated partitions until it completes. Kafka 4.3 added cordoning (KIP-1066) via cordoned.log.dirs, so new partitions are not placed on log directories you are decommissioning. Tiered storage reduces how much data a reassignment must copy.

30. How do you upgrade a ZooKeeper-based cluster to Kafka 4.x?

Answer: You cannot jump straight from a ZooKeeper-mode cluster to 4.x, because 4.x has neither ZooKeeper mode nor migration tooling. The path is: upgrade to a bridge release (3.9 is the recommended final 3.x line), provision a KRaft controller quorum, run the ZooKeeper-to-KRaft migration (the KRaft controllers copy metadata from ZooKeeper and keep both in sync during migration), move brokers to KRaft mode, finalise, decommission ZooKeeper, and then upgrade to 4.x. Kafka 4.0 also raised Java requirements: Java 17 for brokers, Connect and tools, and Java 11 for clients and Kafka Streams.

Production consideration: Check client compatibility first. Kafka 4.0 raised the minimum supported client and broker protocol versions (KIP-896), so very old clients may need upgrading before the brokers move to 4.x. Inventory clients through broker request metrics or client-telemetry before the change window.

31. Which metrics do you watch on a production Kafka cluster?

Answer: Group them by question:

• Is data safe? UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflinePartitionsCount, ISR shrink and expand rates. Any non-zero offline partition count is an incident.
• Is the control plane healthy? Exactly one active controller across the cluster (ActiveControllerCount summed), plus metadata quorum lag.
• Are brokers saturated? RequestHandlerAvgIdlePercent and network processor idle percentage, request queue time, produce and fetch latency percentiles, disk usage and I/O wait.
• Are clients OK? Consumer lag in time, rebalance rate, producer error and retry rates, record send rate.

Kafka exposes metrics over JMX, and KIP-714 lets brokers collect client metrics through a plugin.

32. How do you secure a Kafka cluster?

Answer: In three layers, configured per listener:

• Encryption: TLS on client and inter-broker listeners. Plaintext listeners should not exist outside a local lab.
• Authentication: mutual TLS certificates or SASL. Kafka supports SASL/GSSAPI (Kerberos), PLAIN (only over TLS, ideally backed by a proper credential store), SCRAM-SHA-256/512 and OAUTHBEARER for OAuth 2.0 identity providers; 4.1 added the jwt-bearer grant type and 4.3 added client assertion for client credentials.
• Authorisation: ACLs on resources (topics, groups, cluster, transactional IDs) per principal, with operations such as Read, Write, Describe and Create. Use prefixed ACLs per team namespace and deny-by-default.

Also: separate principals per application (never one shared "kafka-user"), quotas to stop a noisy client hurting others, encryption at rest on the disks or remote store, and audit of ACL changes. If events contain personal data, field-level encryption or tokenisation before produce keeps sensitive values out of every downstream copy.

33. How do you run Kafka as a shared multi-tenant platform?

Answer: A central platform team in a GCC often hosts topics for dozens of product teams. Practices that work:

• Naming conventions (domain.team.entity.version) with prefixed ACLs per team.
• Self-service topic requests through code (Terraform providers or GitOps) with guardrails on partitions, retention and replication factor.
• Produce, fetch and request quotas per principal, so one runaway batch job cannot saturate brokers.
• Chargeback or showback on storage and throughput per team.
• Schema registry compatibility rules per subject and clear ownership metadata on every topic.

34. How does geo-replication work with MirrorMaker 2?

Answer: MirrorMaker 2 is built on Kafka Connect. Source connectors replicate topics from one cluster to another, by default prefixing remote topic names with the source cluster alias, while checkpoint and heartbeat connectors replicate consumer group offsets and monitor connectivity. Because offsets differ between clusters, MirrorMaker translates committed offsets so a consumer can fail over and resume close to where it stopped. RemoteClusterUtils.translateOffsets() gained a batch variant in 4.3.

Common patterns are active-passive for disaster recovery and active-active with prefixed topics. Replication is asynchronous, so failover can lose the last seconds of data and replay some records; consumers must be idempotent.

Connect, Streams, schemas and managed Kafka

35. Explain Kafka Connect's architecture.

Answer: Connect is a framework for moving data between Kafka and external systems without writing producer and consumer code. Workers (ideally in distributed mode, as a cluster) run connectors. A connector splits work into tasks, which are the unit of parallelism. Source connectors read from systems such as databases and SaaS APIs into Kafka; sink connectors write from Kafka into warehouses, search indexes or object storage. Converters serialise records (JSON, Avro, Protobuf), and single message transforms (SMTs) make lightweight per-record changes such as renaming fields or routing.

In distributed mode, connector configs, offsets and status live in three internal Kafka topics, so workers are stateless and tasks rebalance when workers join or fail. For error handling, errors.tolerance=all with a dead-letter topic keeps a sink running past bad records. Since 4.1, Connect can run multiple versions of the same plugin (KIP-891), which makes connector upgrades and rollbacks easier.

36. What do you watch for when running CDC with Debezium into Kafka?

Answer: The data engineering guide covers why log-based CDC beats polling. Operationally, Debezium runs as Kafka Connect source connectors that read a database's change log and emit one event per row change, usually with before and after images, keyed by primary key so changes to a row stay ordered in one partition. Points I would raise:

• Snapshots: the initial snapshot of large tables can take hours and load the source; plan it, and know how incremental snapshots work in your version.
• Source log retention: if the connector stops, a PostgreSQL replication slot keeps WAL on the database disk; alert on slot lag so the primary does not fill up.
• Deletes: a delete emits a delete event and typically a tombstone, which compacted topics need to drop the key.
• Schema changes: DDL changes flow into the events; pair CDC with a schema registry and compatibility rules.
• Exposure: raw CDC exposes internal table structure. For cross-team events, prefer the outbox pattern, where the application writes curated domain events to an outbox table that Debezium publishes.

37. What is Kafka Streams, and how does it handle state and scaling?

Answer: Kafka Streams is a Java library (not a separate cluster) for building stream processing applications that read from and write to Kafka. Its DSL offers KStream (an event stream), KTable (a changelog view of latest value per key) and GlobalKTable (fully replicated to every instance), with joins, windowed aggregations and the lower-level Processor API.

State lives in local state stores (RocksDB by default) backed by compacted changelog topics, so a failed instance's state is restored on another instance; standby replicas cut restore time. The application scales by running more instances; work is split into tasks by input partition. Kafka 4.2 made the new broker-driven Streams rebalance protocol (KIP-1071, built on KIP-848) production-ready for its core feature set, aiming for faster and more stable rebalances.

38. Explain schema compatibility modes and how you evolve an event safely.

Answer: Schema registries commonly offer these modes per subject:

• Backward: consumers on the new schema can read data written with the previous one. You may delete fields or add fields with defaults. Upgrade consumers first.
• Forward: consumers on the old schema can read data written with the new one. You may add fields, or delete fields that have defaults. Upgrade producers first.
• Full: both directions; the safest for shared topics.
• Transitive variants check against all previous versions, not just the last one, which matters when consumers replay old data.

Safe evolution habits: add optional fields with defaults; never change a field's type or meaning in place; for breaking changes, publish a new versioned topic and run both during migration; and enforce compatibility checks in CI so a breaking schema fails the pull request, not production.

39. Kafka Streams, Apache Flink or Spark Structured Streaming: how do you choose?

Answer: It depends on the deployment model and the processing needs.

Aspect	Kafka Streams	Apache Flink	Spark Structured Streaming
Deployment	Library inside your app	Separate cluster or managed service	Spark cluster or lakehouse platform
Sources and sinks	Kafka in and out	Many connectors	Many, strong lakehouse sinks
Latency	Per-record, low	Per-record, low	Micro-batch by default
Language	Java and other JVM languages	Java, SQL, Python	Python, SQL, Scala, Java
Sweet spot	Microservices enriching and aggregating events	Large stateful pipelines, complex event time	Teams already on Spark, streaming into Delta or Iceberg

Interview tip: Tie the choice to the team. A Java microservices team gets Kafka Streams running fastest; a data platform team on a lakehouse usually picks Spark; heavy stateful CEP or a SQL-first streaming platform points to Flink. For the lakehouse side, see our Databricks interview questions.

40. How do you compare managed Kafka options such as Confluent Cloud, Amazon MSK and Azure Event Hubs?

Answer: Described generally, because tiers and features change often:

• Confluent Cloud: a fully managed Kafka service from the company founded by Kafka's original creators, with managed connectors, schema registry, stream processing and governance features in the same platform. Multi-cloud.
• Amazon MSK: AWS-managed Apache Kafka, with provisioned clusters where you choose broker types and a serverless option, plus MSK Connect for Connect workloads. It integrates with IAM, VPC networking and other AWS services.
• Azure Event Hubs: a native Azure event streaming service that exposes a Kafka-protocol-compatible endpoint, so many Kafka clients work by changing configuration. It is not Apache Kafka underneath, so verify feature support (transactions, compaction, specific admin APIs) for your workload.

Questions I ask before choosing: which Kafka APIs and features does the workload actually use; where are the producers and consumers (cross-cloud egress cost); networking and private connectivity; identity integration; the upgrade policy and how quickly new Kafka versions arrive; and whether the team wants to run Connect and a schema registry itself. Check current documentation for supported versions and limits.

Kafka for AI systems

41. How does Kafka support real-time features for ML models?

Answer: Many models need features that change by the second: transactions in the last ten minutes, items in a cart, failed logins in the last hour. A stream processor (Kafka Streams, Flink or Spark) consumes raw events from Kafka, computes windowed aggregates per entity, and writes them to an online feature store or low-latency key-value store for inference, while the same events land in the lakehouse for training.

The hard part is training-serving consistency: offline features must be computed with the same logic and with point-in-time correctness, so the training set only uses values that would have been known at prediction time. Using event time (not processing time) and keeping the raw events replayable in Kafka or the lakehouse lets you backfill features when logic changes. Fraud detection is the textbook case; our fraud and anomaly detection interview questions go deeper on the modelling side.

42. How would you keep a RAG vector index fresh using Kafka?

Answer: Treat the index as a materialised view of source content, updated by events rather than nightly rebuilds.

source DB / docs -> CDC or change events -> Kafka
  -> parse + chunk (stable chunk IDs)
  -> embed (rate-limited workers)
  -> upsert / delete in vector store
  -> ACL metadata synced with each chunk

Design points:

• Key events by document ID so all versions of a document are processed in order.
• Derive deterministic chunk IDs (document ID plus chunk position or content hash) so reprocessing upserts rather than duplicates.
• Handle deletes and permission changes as first-class events; a removed document must be deleted from the index, not just stop being updated.
• Version by embedding model: a model change means a re-embedding backfill into a new index, then an alias switch.
• Use a dead-letter topic for documents that fail parsing, and expose a "last indexed" timestamp per source so stale answers can be traced.

Our guide to data pipelines for RAG covers parsing, chunking and freshness in depth, and the vector database interview questions cover the index side.

43. How do event-driven AI agents use Kafka?

Answer: Instead of waiting for a user to type a prompt, an agent subscribes to business events, such as a new support ticket, a failed payment or an inventory threshold breach, decides what to do, calls tools, and publishes its decisions and actions as new events. Kafka provides decoupling (many agents and services react to the same event), replay (re-run an agent on past events after a prompt change to compare behaviour), and an audit trail of what the agent saw and did.

Engineering rules: make every agent action idempotent and keyed by the triggering event ID, because at-least-once delivery means the agent will sometimes see an event twice; publish proposed high-risk actions to an approval topic for a human-in-the-loop step rather than executing them directly; keep long-running agent state in a durable workflow engine rather than in a consumer's memory (see durable, long-running AI agents); and use share groups or rate-limited consumers so a spike of events cannot exceed LLM rate limits.

44. What changes when an LLM call sits inside a Kafka consumer?

Answer: LLM calls are slow, variable, rate-limited, costly and occasionally fail or return malformed output. Inside a consumer, that affects several things at once:

• Poll timeouts: long calls can exceed max.poll.interval.ms and cause rebalances (Q20). Keep batches small or decouple work with a share group.
• Back-pressure: use pause() when provider rate limits are hit instead of retrying hot.
• Cost control: a replay or backfill can trigger thousands of paid calls; add a budget guard and cache by input hash.
• Validation: validate structured output against a schema before producing it downstream, and route failures to a dead-letter topic.
• Privacy: events sent to an external model may contain personal data; mask before the call, and log prompts and responses to a restricted topic with short retention. Our DPDP Act guide covers the Indian obligations.

If you want to practise streaming pipelines, CDC and the data foundations behind real-time AI hands-on, Cloudsoft's HORIZON Data Engineering & AI program combines modern data engineering with the data work that AI systems depend on.

Real-world scenario questions

45. Lag on one consumer group started growing right after a deployment, while other groups on the same topic are fine. What do you do?

Answer: Other groups being healthy rules out broker capacity and producer spikes, so the cause is almost certainly in the new consumer release or its configuration. The data engineering guide covers general lag triage; this is the "it was a deploy" variant.

What I would check:

1. Lag per partition: uniform growth points to throughput; growth on a few partitions points to stuck members or a hot key.
2. Rebalance rate since the deploy: a new code path that exceeds max.poll.interval.ms causes a rebalance loop where the group makes almost no progress.
3. Error logs for a new exception thrown on a common record type, causing retries in place.
4. Config diffs: lower max.poll.records, a changed fetch.min.bytes, auto-commit turned off without manual commits (lag "grows" because nothing is committed), or a changed group.protocol.
5. Downstream call latency added by the release: a new per-record API call or database lookup.

Production consideration: Roll back first if lag threatens an SLA, then debug in staging with a replay of production traffic. Lag alerts should be tied to deploy events on the same dashboard.

46. A downstream team reports duplicate events in their database. How do you find the source?

Answer: Duplicates come from three places: the producer (retries without idempotence, or the application sending twice), the topic itself (two publishers, or MirrorMaker replay after failover), or the consumer (reprocessing after a rebalance or crash before commit). Find where they first appear.

What I would check:

1. Are the duplicates at different offsets in Kafka? Read the topic around one duplicated business ID. Same payload at two offsets means producer-side; one offset written twice to the database means consumer-side.
2. Producer config: is idempotence actually on, or disabled by a conflicting setting such as acks=1? Do upstream services retry at the application level after a timeout?
3. Consumer behaviour: correlate duplicate timestamps with rebalances, restarts or deploys. Check whether commits happen before or after the database write.
4. Sink design: is the write an insert or an upsert on a unique event ID?

Production consideration: The durable fix is end-to-end: give every event a unique ID at creation, keep idempotent producers, and make sinks upsert or deduplicate on that ID.

47. A consumer group is stuck in a rebalance storm: members keep joining and leaving, and throughput collapses. How do you stabilise it?

Answer: A rebalance storm is usually a feedback loop: a rebalance pauses processing, backlog grows, the next poll returns a huge batch, processing exceeds the poll interval, the member is kicked out, and another rebalance starts.

What I would check:

1. Coordinator logs for why members leave: poll interval exceeded, session timeout (missed heartbeats from GC pauses or CPU throttling), or explicit leave on shutdown.
2. Kubernetes events: liveness probes killing pods during slow processing, autoscalers adding and removing pods on CPU, or OOM kills.
3. Batch size versus processing time per record.
4. The protocol and assignor in use: eager assignment makes every rebalance stop-the-world.

Stabilise with smaller max.poll.records, static membership for rolling deploys, autoscaling on lag rather than CPU with cooldowns, and generous resource limits. Longer term, moving the group to the KIP-848 protocol (group.protocol=consumer) makes rebalances incremental and server-driven, so one slow member no longer stalls everyone.

Production consideration: Alert on rebalance rate per group. A storm is often visible in metrics long before it shows as lag.

48. A bank needs every event for an account processed in order, but also needs high throughput. How do you design it?

Answer: Kafka orders records only within a partition, so key by account ID: all events for one account go to one partition, while different accounts spread across many partitions for throughput.

What I would check:

1. Producer side: idempotence on (preserves order across retries); one logical producer per account stream or a sequence number in the payload so consumers can detect gaps.
2. Partition count sized for growth up front, because adding partitions later remaps accounts mid-stream.
3. Consumer side: process each partition sequentially, or parallelise inside a partition by account key with a keyed executor so per-account order holds.
4. Failure handling: a poison event for one account should park that account (with an alert), not jump to a retry topic that breaks order.
5. Cross-topic ordering: if debits and credits arrive on different topics, order is not preserved across them; use one topic per aggregate or reconcile with sequence numbers.

Production consideration: Ask how strict "in order" really is. Often only some event types per account need ordering, and relaxing the rest lets you use share groups or wider parallelism for them.

49. One partition receives far more traffic than the others and its consumer is always behind. How do you fix a hot partition?

Answer: A hot partition is almost always a skewed key: a large merchant, a default value such as "unknown" or an empty string, or a test account flooding events.

What I would check:

1. Per-partition bytes-in and message rates to confirm the skew, then sample the hot partition's keys to find the heavy key.
2. Whether the hot key is a bug (null or default keys) or real traffic.
3. Whether that key truly needs ordering.

Fixes, from least to most invasive: fix the bug producing default keys; if ordering is not needed for that entity, salt the key (merchant-123#0..7) to spread it, and aggregate downstream; route very large tenants to a dedicated topic with its own consumers; or, if ordering is needed only per sub-entity (per terminal rather than per merchant), use the finer-grained key. Adding partitions does not help: the hot key still hashes to one partition.

50. During broker maintenance, producers start failing with NotEnoughReplicas errors. What is happening?

Answer: With acks=all, a write fails when the ISR is smaller than min.insync.replicas. Taking one broker down for patching should leave two of three replicas in sync, so errors mean either more replicas were already out of sync, or the configuration leaves no headroom.

What I would check:

1. Under-replicated partitions before maintenance started: if a follower was already lagging, removing one more broker drops the ISR below the minimum.
2. Topics with replication factor 2 and min.insync.replicas=2, or RF 3 with min ISR 3: any single outage blocks writes.
3. Rack placement: if two replicas of a partition sit on brokers being patched together.
4. Whether the maintenance runbook waited for under-replicated partitions to reach zero before moving to the next broker.

Production consideration: Make "zero under-replicated partitions" a gate between broker restarts, enforce RF 3 with min ISR 2 for business topics through topic-creation policy, and do not lower min.insync.replicas during an incident without recording the durability risk you are accepting.

51. Broker disks are filling up faster than expected. How do you investigate without losing data?

Answer: Find which topics are growing and why their retention is not removing data.

What I would check:

1. Per-topic and per-partition size (kafka-log-dirs.sh, or the partition size metrics added in 4.3) to find the growing topics.
2. Retention overrides: a topic created with very long or infinite retention, or retention.bytes unset.
3. Compacted topics where the cleaner has stopped, or with high-cardinality keys that never compact.
4. Producer timestamps far in the future, which delay time-based segment deletion; Kafka 4.0 tightened the default for future timestamps.
5. Uneven distribution after adding brokers without reassigning partitions.

Short-term relief: lower retention on non-critical topics after confirming consumers do not need the data, and expand storage. Longer term: tiered storage for long-retention topics, quotas and retention policy at topic creation, and disk-usage alerts with enough lead time to act calmly.

Production consideration: Never delete segment files by hand from the broker filesystem. Change retention through configs and let Kafka remove segments.

52. An upstream team deployed a schema change and several consumers started crashing. How do you respond and prevent it?

Answer: Restore service first, then close the gap that let the change through.

What I would check:

1. Whether the topic's subject has compatibility checking enabled, or was set to none.
2. Whether the producer bypassed the registry (plain JSON) or changed meaning without changing structure (a renamed unit, a status code reused).
3. The first offset with the new format, so consumers can be fixed and replayed from a known point.

Immediate options: roll back the producer; deploy consumers that tolerate both versions; or temporarily route bad records to a dead-letter topic so healthy records keep flowing. Prevention: compatibility mode set per subject (full or backward transitive for shared topics), schema checks in the producer's CI, a data contract with an owner and consumers listed, and versioned topics for genuinely breaking changes.

53. Users say an AI assistant still answers from a policy document that was deleted last week. The index is updated through Kafka. Where do you look?

Answer: Additions usually work and deletions are where event-driven indexing breaks, so trace the delete event end to end.

What I would check:

1. Did a delete event or tombstone reach Kafka at all? Some connectors or source APIs do not emit deletes by default.
2. Did the indexing consumer handle it? A consumer that ignores null-value records drops tombstones silently.
3. Did the delete target the right IDs? If chunk IDs are random rather than deterministic, the consumer cannot find the old chunks to remove.
4. Is the indexing consumer lagging, or did it miss tombstones on a compacted topic because it lagged beyond delete.retention.ms?
5. Is there a cache in front of retrieval, or a second index that is not wired to the same events?

Production consideration: Add a periodic reconciliation job that compares source document IDs with indexed IDs, deletes orphans, and alerts on drift. Event-driven updates plus scheduled reconciliation is far more reliable than either alone.

54. A slow-processing consumer group for document enrichment cannot keep up, and adding consumers does not help. What do you change?

Answer: The group has reached its partition count; extra members are idle. Each record takes seconds (OCR, an LLM call), and one slow record blocks its partition.

What I would check:

1. Whether the work needs ordering per key. Enrichment of independent documents usually does not.
2. Where the time goes: external call latency, rate limits, or retries.
3. Whether the external service can handle more concurrency at all.

Options: move the workload to a share group on a 4.2 or later cluster, so consumers can exceed the partition count, acknowledge per record and use RENEW for long jobs; or keep the consumer group and process records concurrently within each consumer with bounded parallelism, committing only up to the lowest completed offset per partition; or increase partitions if ordering is not a concern. Check managed-service support before choosing share groups, since availability varies by provider and version.

Production consideration: Whatever the model, cap concurrency to the downstream rate limit; scaling consumers past what the LLM provider allows only creates throttling errors.

55. Design a Kafka platform for a retailer that needs real-time inventory, fraud signals and an AI shopping assistant.

Answer: Separate concerns by topic domain, and give each consumer the delivery model it needs.

POS / e-com / WMS --CDC + events--> Kafka (KRaft, RF 3)
  inventory.* (keyed by SKU+store, compacted view)
     -> Streams app -> stock cache -> storefront
  payments.* (keyed by account)
     -> feature job -> online store -> fraud model
  catalog.* (keyed by product ID)
     -> embed workers -> vector index -> assistant
  all topics -> sink connector -> lakehouse

What I would check:

1. Ordering needs per domain: per SKU-store for stock, per account for payments, none for enrichment.
2. Durability: acks=all, min ISR 2 and idempotent producers for stock and payments; looser settings acceptable only for clickstream.
3. Retention: compacted current-state topics plus tiered storage for long event history and replay.
4. Governance: schema registry with compatibility per subject, prefixed ACLs per team, personal data masked before it reaches AI topics.
5. Operations: lag and under-replication alerts, rebalance metrics, DR via MirrorMaker 2 or the managed service's equivalent, and a managed versus self-run decision based on team skills.

Production consideration: Start with fewer, well-owned topics and clear contracts. Most Kafka platforms that fail do so through ungoverned topic sprawl, not broker limits. For the system design framing of answers like this, see our AI system design interview questions.

Key takeaways

• Kafka 4.x runs only in KRaft mode; know the bridge-release path for ZooKeeper clusters and how controller quorums are sized.
• Durability comes from acks=all, replication factor 3 and min.insync.replicas=2 together, not from any one setting.
• Idempotent producers and transactions give exactly-once inside Kafka; outside it, design idempotent sinks keyed on event IDs.
• The KIP-848 consumer protocol is GA since 4.0 but opt-in on the client; share groups are production-ready since 4.2 for unordered, per-record work.
• Most production incidents are lag, duplicates, rebalance storms, hot keys and schema breaks; practise diagnosing each from metrics.
• For AI, Kafka keeps features, vector indexes and agent triggers fresh, but deletes, idempotency and rate limits need deliberate design.

Interview preparation checklist

• Run a local three-broker KRaft cluster in Docker and kill a broker while producing with acks=all and min ISR 2.
• Write a producer with idempotence and a transactional consume-transform-produce loop, and read it with both isolation levels.
• Run one consumer group with group.protocol=classic and one with consumer, and compare rebalance behaviour during a rolling restart.
• Try a share group on a 4.2 or later cluster with deliberate failures, and watch redelivery and rejection.
• Create a compacted topic, produce tombstones and observe what a new consumer sees.
• Set up Debezium against PostgreSQL through Kafka Connect, then break a schema and handle it.
• Build a small Kafka-to-vector-store indexer that handles updates and deletes with deterministic chunk IDs.
• Prepare one story each about lag, duplicates and a hot partition, with what you checked and what you changed.
• Review orchestration and warehouse topics through the Airflow interview questions and Snowflake interview questions, since Kafka roles often include both.

FAQ

What skills are required for a Kafka interview?

Solid understanding of partitions, replication, producer and consumer configuration, consumer groups and offsets, plus Java or Python client code. Mid-level and senior roles add Kafka Connect, Kafka Streams or Flink, schema management, security, monitoring and incident diagnosis.

How should I prepare for a Kafka interview in 2026?

Run a local KRaft cluster, write producers and consumers, and deliberately break things: kill brokers, cause lag, trigger rebalances and send bad records. Then practise explaining what happened and how you fixed it, because scenario questions decide most interviews.

Is ZooKeeper still asked about in Kafka interviews?

Sometimes, mainly as history or in migration questions. Kafka 4.0 removed ZooKeeper entirely, so explain KRaft first and mention that ZooKeeper-based clusters must migrate on a 3.x bridge release before upgrading to 4.x.

Do I need Java to work with Kafka?

It helps, because Kafka, Kafka Streams and Connect are JVM-based and the Java client is the reference implementation. Many data engineers use Python or other language clients successfully, but reading Java examples and stack traces is a useful skill.

Should I learn Kafka Streams or Apache Flink?

Learn the shared concepts first: event time, windows, state and checkpointing. Then pick by role. Kafka Streams suits Java microservice teams, while Flink and Spark Structured Streaming are more common on data platform teams.

Is Kafka a good skill for data engineers and AI engineers?

Yes. Real-time data is central to fraud detection, personalisation, operational dashboards and keeping AI systems current, and Kafka is widely used for it. Combining Kafka with SQL, a lakehouse platform and cloud skills makes a strong data engineering profile.

Do I need Kafka certification to get a job?

No. Vendor certifications can help a resume pass screening, but interviewers judge whether you can reason about durability, ordering and failure. A documented hands-on project usually carries more weight.

Can a fresher get a Kafka or streaming data engineering role?

Freshers usually join as data or backend engineers and pick up Kafka on the job. Strong SQL, Python or Java, and one end-to-end streaming project that you can explain clearly make a fresher application much stronger.

Which managed Kafka service should I learn?

Learn Apache Kafka concepts first, because they transfer everywhere. Then get familiar with the managed service used by employers you target, such as Amazon MSK in AWS-heavy teams, Azure Event Hubs in Azure shops, or Confluent Cloud.

Ready to build streaming pipelines, CDC and the governed data foundations behind enterprise AI with guidance? The HORIZON data engineering and AI course combines hands-on pipeline engineering with AI data work, in classroom sessions in Ameerpet or live online. If you want a broader path across AI, ML, cloud and security, look at the APEX AI, ML, Cloud and Cyber Security program. Call +91 96660 19191 to book a free demo.

Related Cloudsoft resources

• Data engineering interview questions
• Data pipelines for RAG
• Airflow interview questions
• Snowflake interview questions
• Fraud and anomaly detection interview questions
• AI for data engineers
• Databricks interview questions