-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Closed
Description
Testing with code based on the example to stream data from rsyslog to kafka. I'm hit by the poor performance - when I would expect 5000 messages / s, I'm only getting ~300 messages / s.
Reading the code, I'm concerned that there's at least two mallocs in the producer thread:
First, in the client:
char *opbuf = malloc(len + 1);
strncpy(opbuf, buf, len + 1);
...then the actual producer:
void rd_kafka_produce (rd_kafka_t *rk, char *topic, uint32_t partition,
int msgflags,
char *payload, size_t len) {
rd_kafka_op_t *rko;
rko = calloc(1, sizeof(*rko));
It seems that it would be best for librdkafka to allocate, track usage and reuse of rd_kafka_op_t and rd_kafka_op_t->rko_payload for optimal performance.
Metadata
Metadata
Assignees
Labels
No labels