Self-managed Kafka cluster

Using LocalStack lambda with self-managed Kafka cluster

LocalStack does not currently support AWS MSK out of the box, but you can run your own self-managed Kafka cluster and integrate it with your own applications.

Running self-managed Kafka

You can find the example Docker Compose file which contains a single-noded ZooKeeper and a Kafka cluster and a simple LocalStack setup as well as Kowl, an Apache Kafka Web UI.

  1. Run Docker Compose:
  1. $ docker-compose up -d
  1. Create the Lambda function:
  1. $ awslocal lambda create-function \
  2.     --function-name fun1 \
  3.     --handler lambda.handler \
  4.     --runtime python3.8 \
  5.     --role r1 \
  6.     --zip-file fileb://lambda.zip
  7. {
  8.     "FunctionName": "fun1",
  9.     "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1",
  10.     "Runtime": "python3.8",
  11.     "Role": "r1",
  12.     "Handler": "lambda.handler",
  13.     "CodeSize": 294,
  14.     "Description": "",
  15.     "Timeout": 3,
  16.     "LastModified": "2021-05-19T02:01:06.617+0000",
  17.     "CodeSha256": "/GPsiNXaq4tBA4QpxPCwgpeVfP7j+1tTH6zdkJ3jiU4=",
  18.     "Version": "$LATEST",
  19.     "VpcConfig": {},
  20.     "TracingConfig": {
  21.         "Mode": "PassThrough"
  22.     },
  23.     "RevisionId": "d85469d2-8558-4d75-bc0e-5926f373e12c",
  24.     "State": "Active",
  25.     "LastUpdateStatus": "Successful",
  26.     "PackageType": "Zip"
  27. }
  1. Create an example secret:
  1. $ awslocal secretsmanager create-secret --name localstack
  2. {
  3.     "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI",
  4.     "Name": "localstack",
  5.     "VersionId": "32bbb8e2-46ee-4322-b3d5-b6459d54513b"
  6. }
  1. Create an example Kafka topic:
  1. $ docker exec -ti kafka kafka-topics --zookeeper zookeeper:2181 --create --replication-factor 1 --partitions 1 --topic t1
  2. Created topic t1.
  1. Create the event source mapping to your local kafka cluster:
  1. $ awslocal lambda create-event-source-mapping \
  2.     --topics t1 \
  3.     --source-access-configuration Type=SASL_SCRAM_512_AUTH,URI=arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI \
  4.     --function-name arn:aws:lambda:us-east-1:000000000000:function:fun1 \
  5.     --self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:9092"]}}'
  6. {
  7.     "UUID": "4a2b0ea6-960c-4847-8684-465876dd6dbd",
  8.     "BatchSize": 100,
  9.     "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1",
  10.     "LastModified": "2021-05-19T04:02:49+02:00",
  11.     "LastProcessingResult": "OK",
  12.     "State": "Enabled",
  13.     "StateTransitionReason": "User action",
  14.     "Topics": [
  15.         "t1"
  16.     ],
  17.     "SourceAccessConfigurations": [
  18.         {
  19.             "Type": "SASL_SCRAM_512_AUTH",
  20.             "URI": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI"
  21.         }
  22.     ],
  23.     "SelfManagedEventSource": {
  24.         "Endpoints": {
  25.             "KAFKA_BOOTSTRAP_SERVERS": [
  26.                 "localhost:9092"
  27.             ]
  28.         }
  29.     }
  30. }
  1. Additionally visit http://localhost:8080 for Kowl’s UI.

Last modified July 26, 2022: fix some typos (#214) (6ab8502d)