This article is contributed. See the original author and article here.

Azure Data Explorer has a mature, open source Kafka Connect sink connector that is in use in production by multiple customers. The connector was *gold* certified in September 2020 by Confluent and is now listed in Confluent Connector Hub.  The certification involved stringent testing by Confluent for quality, standards, best practices, feature completeness, delivery guarantees and performance.  To meet the certification requirements, we added a number of new features to our connector, renamed some, and removed support for some with version 1.x.  This article details the same. 

For comprehensive information about our connector, visit our Git repo.

Improved delivery guarantees

Our connector now offers “At least once” delivery guarantees (no data loss).

What’s new

 

1. Comprehensive upfront validation

The connector validates for all required and dependent sink properties, access and grants and shuts down upfront with appropriate error messages.

 

2. Configurable behavior on error

A new sink property with possible values of fail (shut down connector task), ignore (ignore and continue processing), and log (log errors and continue processing).  

 

behavior.on.error

 

 

3. *New* configurable retries

A set of new sink properties-

 

errors.retry.max.time.ms
errors.retry.backoff.time.ms

 

errors.retry.max.time.ms: Period of time in milliseconds to retry for transient errors

errors.retry.backoff.time.ms: Period of time in milliseconds to backoff before retry for transient errors

 

4. Additional converters supported

We have added support for the following converters-

 

org.apache.kafka.connect.json.JsonConverter
io.confluent.connect.avro.AvroConverter
io.confluent.connect.json.JsonSchemaConverter

 

JsonSchemaConverter is for Json with schema.  


4. Schema registry support

The connector now supports schema registry with the property-

 

value.converter.schema.registry.url

 

We support schema registry for avro and json, protobuf is on the roadmap.

 

5. Schema with message

The connector now supports messages with schema with the following sink property that accepts a boolean-

 

value.converter.schemas.enable

 


6. Kafka Connect deadletterqueue

The optional sink properties below can be provided for Kafka Connect to redirect messages that failed deserialization-

 

errors.deadletterqueue.bootstrap.servers
errors.deadletterqueue.topic.name
errors.deadletterqueue.security.protocol
errors.deadletterqueue.sasl.mechanism
errors.deadletterqueue.sasl.jaas.config
errors.deadletterqueue.sasl.kerberos.service.name

 

The last 4 sink properties are required for secure clusters and*kerberos* property is specific to kerberized Kafka clusters.

 

7. Miscellaneous deadletterqueue

The optional sink properties below can be provided for the connector to redirect messages that failed besides deserialization, e.g. transient failures beyond retry period –

 

misc.deadletterqueue.bootstrap.servers
misc.deadletterqueue.topic.name
misc.deadletterqueue.security.protocol
misc.deadletterqueue.sasl.mechanism
misc.deadletterqueue.sasl.jaas.config
misc.deadletterqueue.sasl.kerberos.service.name

 

The last 4 sink properties are required for secure clusters and*kerberos* property is specific to kerberized Kafka clusters.

 

What’s renamed

For the purpose of standardization and consistency, we have renamed some properties-

Previous name New name
kusto.auth.authority aad.auth.authority
kusto.auth.appid aad.auth.appid
kusto.auth.appkey aad.auth.appkey
kusto.tables.topics_mapping kusto.tables.topics.mapping
kusto.sink.tempdir tempdir.path
kusto.sink.flush_interval_ms flush.interval.ms
kusto.sink.flush_size
flush.size.bytes

 

What’s removed

The following properties are no longer supported with version 1.x-

Property name Rationale
kusto.auth.username Customers typically only use application IDs/service principals; We therefore removed support for user principals
kusto.auth.password Customers typically only use application IDs/service principals; We therefore removed support for user principals
eventDataCompression We now compress all data shipped to Azure Data Explorer, to improve performance

Connector documentation

Visit out Git repo

Connector download 

Visit out Git repo release page

Confluent Connector Hub

https://www.confluent.io/hub/microsoftcorporation/kafka-sink-azure-kusto

Need support?

Ran into a bug? Need hep getting started?  Have a feature request?  Open an issue.

Want to contribute?

Read our guidelines and get involved.  We welcome community contribution.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.