admin 管理员组

文章数量: 1086019

I’m designing a multi-datacenter architecture using Apache Pulsar with geo-replication enabled.

Architecture Overview:

  • Apache Pulsar version: 4.0.2
  • Helm Chart version: pulsar-3.9.0
  • BookKeeper: 5 replicas
  • Broker: 5 replicas
  • Proxy: 3 replicas
  • Zookeeper: 3 replicas
  • Recovery: 1 replica
  • Deployed on Kubernetes (Rancher)
  • Bookie storage: vSphere CSI via Persistent Volume Claims (PVC)

The architecture includes 2 different datacenters, and each of them hosts local consumers (around 200 per DC). My goal is to enable geo-replication between these DCs, but at the same time I must strictly prevent message duplication during consumption — messages should be processed exactly once, even in failover scenarios.

enter image description here

Requirements:

  • Messages must be durable (no data loss allowed)
  • Active-active or active-passive setup is acceptable
  • Each datacenter has its own Pulsar cluster and
  • Consumer duplication must not happen, even during failover or replay
  • Pulsar Deduplication and Failover Subscriptions are enabled

Questions:

  1. What is the best practice to ensure geo-replication between clusters without consumers processing the same message multiple times?
  2. Is it possible to achieve synchronous geo-replication via BookKeeper, writing each message to multiple DCs before ack?
  3. Would a combination of deduplication + idempotent consumer logic + failover subscription be enough?
  4. Any gotchas or caveats you’ve experienced with Pulsar multi-cluster deployments in this scenario?

Thanks in advance!

I deployed Apache Pulsar 4.0.2 using Helm chart pulsar-3.9.0 on a Kubernetes (Rancher) cluster across 2 datacenters. I enabled geo-replication, created separate clusters for each DC, and connected them via pulsar-admin CLI using the set-replication-clusters and set-clusters commands.

I also configured:

  • deduplication on the topics
  • failover subscription on the consumers
  • idempotent logic in the consumer code using message_id checking with Redis

What I expected:

  • Each message would be written once and consumed only once per global system
  • No duplicates during failover or network interruptions

What actually happened:

  • During failover testing, I noticed duplicate consumption in some edge cases (likely due to replay after disconnect)
  • I'm looking for a more reliable strategy to ensure exactly-once processing across DCs

本文标签： queueHow to perform GeoReplication in Apache Pulsar without creating duplicate messagesStack Overflow

版权声明：本文标题：queue - How to perform Geo-Replication in Apache Pulsar without creating duplicate messages? - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://roclinux.cn/p/1744092971a2532379.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。