AWS Multi-shards and Multi-streams

This section describes about enabling the processing of AWS multi-shards and multi-streams.

The EKS Ingester consumer feature supports multi-shards and multi-streams with the help of the latest Kinesis Client Library (KCL) version 2.3.4 with Ingester, which allows to consume records in the Kinesis Data Stream. The KCL is a library that simplifies the records consumption and serves as a layer of abstraction over the AWS SDK Java APIs for Kinesis Data Streams.

Configuring AWS Multi-shards, Multi-part and Multi-streams

The following property keys should be set to configure the consumer properties and enable the Avro Ingester.

Enabling AWS Multi-shards, Multi-part and Multi-streams using KCL

The following section describes about enabling the processing of AWS multi-shards and multi-streams using KCL.

Confluent Schema Registry for AVRO Ingester

As Kinesis represents data as bytes, it is common to use an external schema to serialise and de-serialise data into bytes according to that schema. Rather than sending a copy of that schema with each message, which is an expensive overhead, it is common to keep the schema in a registry and send just an ID with each message.

The Confluent Schema Registry provides an easy way to store, retrieve and manage schemas. It exposes several useful RESTFul APIs. The schemata are stored by subject, and by default, the registry does a compatibility check before allowing a new schema to be uploaded against a subject.

Each producer is aware of the schema it is working with, and each consumer must either consume data in any format or have a specific schema it prefers to read. The producer consults the registry to establish the correct ID for usage when sending a message. The consumer uses the registry to fetch the sender's schema. When the consumer is aware of both its preferred message format and the sender's schema, the Avro library can convert the data into the consumer's preferred format.

Dependency of KCL Library

The following dependency of the KCL library is available in the framework for processing multi-shards.

<dependency>
    <groupId>software.amazon.kinesis</groupId>
    <artifactId>amazon-kinesis-client</artifactId>
    <version>${aws.kcl.kinesis.client.verison}</version>
</dependency>

Deployment of Schema Registry

The following schema registry must only be deployed while enabling the EKS Avro Ingester. To deploy the schema registry,

1. Create a repository in AWS, as mentioned below.

   aws ecr create-repository --repository-name <repositoryname> --region <ecrregion>

   aws ecr create-repository --repository-name cp-schema-registry --region us-east-2

2. Create a repository for Confluentinc images using the below commands.

   Pull:
   docker pull confluentinc/cp-schema-registry:5.2.2
   docker pull confluentinc/cp-kafka:5.2.2
   docker pull confluentinc/cp-zookeeper:5.2.2
     
   Tag:
   docker tag confluentinc/cp-schema-registry:5.2.2 <<ECRNAME>>/cp-schema-registry:5.2.2
   docker tag confluentinc/cp-kafka:5.2.2 <<ECRNAME>>/cp-kafka:5.2.2
   docker tag confluentinc/cp-zookeeper:5.2.2 <<ECRNAME>>/cp-zookeeper:5.2.2
     
   Push:
   docker push <<ECRNAME>>/cp-schema-registry:5.2.2
   docker push <<ECRNAME>>/cp-kafka:5.2.2
   docker push <<ECRNAME>>/cp-zookeeper:5.2.2
   

3. Start the schema registry in EKS using the below command.

   helm install kafka-oss cp-helm-charts --version 0.5.0 -n kafka --set cp-zookeeper.image=<<ECRNAME>>/cp-zookeeper --set cp-zookeeper.imageTag=5.2.2 --set cp-kafka.image=<<ECRNAME>>/cp-kafka --set cp-kafka.imageTag=5.2.2 --set cp-schema-registry.image=<<ECRNAME>>/cp-schema-registry --set cp-schema-registry.imageTag=5.2.2 --create-namespace