Producing messages from inside a Sidekiq Job

[alan-sapaad]

Hello,

We are using Sidekiq jobs to generate the payloads and broadcast Kafka messages in our Rails application. We need to do this as a background job because there are multiple DB queries involved in generating the Kafka payload. We are using ruby-kafka gem to produce messages and we are using the async-producer with a delivery_interval of 120ms. We are using Heroku Kafka and sometimes when the periodic Kafka broker maintenance happens, we get the error BufferOverflow and most of the time we will have to restart the specific worker that is throwing the error.

1. Would using the async producer add any value to our setup when we are anyway using a background job to producer messages?
2. Would using the sync producer make the setup a little more resilient - We can have the job to retry when a production fails?
3. Would there be a performance penalty in doing this?
4. Is this a better approach when we are trying to tune the system for better resilience and lower latency. Our throughput is not super high - 15 to 20 messages per second now.

Thank you.

[mensfeld]

quote from wiki:

This library is no longer actively developed and has been superseded by librdkafka via rdkafka-ruby bindings. While this library may still receive security patches and bug fixes, it is no longer recommended for production usage.