Logging, MDC, and Interceptors

We were trying to use SLF4J and MDC to log add some helpful context to our log messages (producer, consumer, and application logic messages) to our Kafka Streams app.

When the app is deployed Kafka tries to be helpful and consumes a batch of messages initially which flows through the Consumer Interceptor.

The problem is MDC is static per thread. So once the batch of messages is passed to be processed by the stream we can’t get a useful context info about the record cleanly.

The MDC gets updated per record in the batch consumed prior to executing the stream logic per record.

If we do something like: log.debug(“Interesting log message”) we can’t use anything useful stored in MDC. It would be whatever record was last processed in the batch ingested.

When we log we want things like the incoming topic, the message offset, a uuid, etc.

We have a lot of different code bases so setting the MDC per streams application (in fact per task in the topology) means touching a lot of different repos.

Is there a hook (besides producer and consumer) that we can use to set the MDC per execution of the topology per record?