前端之家

  1. 首页
  2. 问答

librdkafka批处理使生产者暂停产生消息

• 问答

我们有一个工具，每秒可产生数千个事件。但是，它实际上并不会每毫秒创建一批消息以达到目标TPS。取而代之的是，它在循环中生成消息，并且在循环中，每个消息都由goroutine提供服务。现在，它可以很好地运行到某个TPS，但是当我们尝试进入一个分区的数千个TPS时，即使我的linger.ms是2 [也是batch.num.messages 40]，我们也会看到更大的延迟。关于设置调试：我们在生产者中看到的所有内容如下。我看到一口气将2393条消息发送到xmit队列，此后它花费了数毫秒的时间来生成批处理。我的邮件大小约为1.6KB。

所以我的问题是-

a）是什么触发了从分区队列到xmit队列的传输？
b）每个分区是否有一个xmit队列和一个分区队列？
c）记录/消息什么时候从分区队列移动到xmit队列？
d）当kafka生产者为分区创建消息集时-它会阻止该分区的所有操作吗？
e）谁能一口气负责这2393条消息，是我的负载生成器工具还是librdkafka？

请注意-我有一个预感，问题在于将一个分区标记为一个xmit和一个分区队列，因此我们在具有24个分区的主题上尝试了相同的TPS，延迟立即降至2-3 ms从15到20毫秒。

日志-

%7|1573686057.912|TOPPAR|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0] 2393 message(s) in xmit queue (2393 added from partition queue)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65167 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65101 bytes,ApiVersion 3,MsgVersion 2)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65165 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65099 bytes,65161 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65095 bytes,65162 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65096 bytes,65160 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65094 bytes,65163 bytes)
%7|1573686057.913|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65097 bytes,MsgVersion 2)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65163 bytes)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65097 bytes,65164 bytes)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65098 bytes,65158 bytes)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65092 bytes,65161 bytes)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65095 bytes,65166 bytes)
%7|1573686057.914|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65100 bytes,65165 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65099 bytes,MsgVersion 2)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65166 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65100 bytes,65162 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65096 bytes,65161 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65095 bytes,65167 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65101 bytes,65163 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65097 bytes,65164 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65098 bytes,65158 bytes)
%7|1573686057.915|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65092 bytes,MsgVersion 2)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65162 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65096 bytes,65167 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65101 bytes,65166 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65100 bytes,65163 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65097 bytes,65164 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65098 bytes,65161 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65095 bytes,65160 bytes)
%7|1573686057.916|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65094 bytes,MsgVersion 2)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65163 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65097 bytes,65165 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65099 bytes,65160 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65094 bytes,65161 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65095 bytes,65162 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65096 bytes,65167 bytes)
%7|1573686057.917|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65101 bytes,MsgVersion 2)
%7|1573686057.918|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: No more space in current MessageSet (40 message(s),65160 bytes)
%7|1573686057.918|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65094 bytes,65166 bytes)
%7|1573686057.918|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 40 message(s) (65100 bytes,MsgVersion 2)
%7|1573686057.918|PRODUCE|rdkafka#producer-1| [thrd:testserver22.visa.com:6667/1004]: testserver22.visa.com:6667/1004: perf.ovn.perf.engg3 [0]: Produce MessageSet with 33 message(s) (53716 bytes,MsgVersion 2)
lucifer0777777777777 回答：librdkafka批处理使生产者暂停产生消息
暂时没有好的解决方案，如果你有好的解决方案，请发邮件至：iooj@foxmail.com
confluent-kafkagolibrdkafka
本文链接：https://www.f2er.com/3098716.html

大家都在问