Add a kafka_consumer_lag input to collect consumer group lag natively via the Kafka Admin API

[SungJin1212]

Use Case

Telegraf can produce to and consume from Kafka, but the single most common Kafka health metric — consumer group lag — currently requires deploying Burrow or kafka_exporter. This plugin computes lag directly using the existing common/kafka client configuration, with no external components.

Expected behavior

1. Connects to the cluster
2. Lists consumer groups, fetches committed offsets per group
3. Emit one metric per (group, topic, partition)

A new input plugin, inputs.kafka_consumer_lag:

[[inputs.kafka_consumer_lag]]
  ## Kafka brokers
  brokers = ["localhost:9092"]

  ## Consumer groups to monitor, glob patterns. Default: all groups.
  # groups_include = ["*"]
  # groups_exclude = []

  ## Topics to monitor, glob patterns. Default: all topics except internal ones.
  # topics_include = ["*"]
  # topics_exclude = []

  ## Emit per-partition metrics (kafka_consumer_lag).
  # partition_metrics = true
  ## Emit per-topic aggregated metrics (kafka_consumer_lag_topic).
  # topic_metrics = true

  ## Only collect groups whose coordinator is this broker ID. Intended for
  ## agent-per-broker deployments so each group is collected exactly once
  ## and collection fails over automatically with coordinator moves.
  # coordinator_broker_id = -1

  ## Optional SASL / TLS settings, same as inputs.kafka_consumer
  # sasl_username = ""
  # sasl_password = ""
  # sasl_mechanism = ""
  # tls_ca = ""
  # insecure_skip_verify = false

Actual behavior

There is no Telegraf-only way to collect consumer group lag today. Users must deploy Burrow or a Prometheus exporter, or attach Jolokia to every consumer application. On a broker-side Jolokia setup the lag simply cannot be derived because the broker never exposes committed offsets over JMX.

Additional info

I am willing to implement this and submit a PR.

[SungJin1212]

@skartikey @srebhan
What do you guys think about this issue? Can I implement it?