AWS DynamoDB with Clojure

Running Clojure unit tests in IntelliJ IDEA / Cursive.

Introduction

After my Five Languages project I was searching something new to learn. I decided to refresh my AWS skills a bit and and compare AWS container services EKS and Fargate and compare their deployment models for Docker containers. I decided to use my Clojure Simple Server exercise as a demo application. There was one problem, though. The Simple Server was just a demo application and it simulated database by reading csv files to internal data structures — therefore making the server stateful. First I needed to change the implementation to make the server fully stateless — storing all state outside the server in a real external database. I decided to use AWS DynamoDB nosql database. In this blog post I write about my experiences to manipulate DynamoDB using Clojure and how to use local DynamoDB Docker container instance as a development database. Next I’ll be using this DynamoDB version of Simple Server to deploy the server to EKS and Fargate and write new blog posts regarding those experiences, so stay tuned.

Local DynamoDB Docker Container Based Development Database

For a small exercise with minimal demo data set the DynamoDB price is almost neglectable, but there there is always the round-trip latency to AWS data center. Therefore I decided to use a local DynamoDB Docker container based development database — and have a nice excuse to learn to use it. The local DynamoDB was a positive surprise — all DynamoDB APIs I used (cli, Python/boto3, Clojure/amazonica) worked nicely out of the box. I found only one item that worked differently in local DynamoDB compared to real AWS DynamoDB service (more about that later).

docker run -p 8000:8000 amazon/dynamodb-local:latest

Let’s Create Tables

I created a sub-directory [dynamodb] for DynamoDB related utilities I used in this project. You can find in that directory create-tables.sh script which creates the four tables needed in the project: session table, users table, product groups table and products table. It is pretty straightforward to use aws cli to create the tables, and the script works the same way with local DynamoDB Docker container instance and with real AWS DynamoDB service. Example of one aws cli call to create the product table:

AWS_PROFILE=$MY_AWS_PROFILE aws dynamodb create-table $MY_ENDPOINT — table-name $MY_PRODUCT_TABLE — attribute-definitions AttributeName=pgid,AttributeType=S AttributeName=pid,AttributeType=S — key-schema AttributeName=pid,KeyType=HASH AttributeName=pgid,KeyType=RANGE — provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 — global-secondary-indexes IndexName=PGIndex,KeySchema=[“{AttributeName=pgid,KeyType=HASH}”,”{AttributeName=pid,KeyType=RANGE}”],Projection=”{ProjectionType=INCLUDE ,NonKeyAttributes=[“title”,”price”]}”,ProvisionedThroughput=”{ReadCapacityUnits=5,WriteCapacityUnits=5}”
MY_ENDPOINT=” — endpoint-url http://localhost:8000"

Let’s Import Data

The next task is to import the test data. I used this as an excuse to refresh my Python/boto3 skills. I could have used Clojure for this as well but since this is my personal project I can do it any way I like. I thought that it might be interesting to do the data import using Python.

# Create the virtual environment for Python.
./create-virtual-env.sh
# Activate the virtual env.
source venv3/bin/activate
# Install aws library boto3.
pip install boto3
[local-dynamodb]
aws_access_key_id = XXXXXXXXXXXXXX___NOT
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX___NOT
./run-local-dynamodb.sh # Start the DynamoDB Docker container in another terminal.
./import-all-tables.sh local-dynamodb dev # Import all tables to that instance. Calls table_importer.py for each 4 tables.
if my_aws_profile == ‘local-dynamodb’: 
dynamodb = session.resource(service_name=’dynamodb’, endpoint_url=’http://localhost:8000')
else:
dynamodb = session.resource(service_name=’dynamodb’)
with open(my_csv_file, ‘r’) as csvfile: 
reader = csv.reader(csvfile,delimiter=’\t’)
with table.batch_writer() as batch:
for user_id, email, first_name, last_name, hashed_password in reader:
batch.put_item(Item={“userid”: user_id, “email”: email, “firstname”: first_name, “lastname”: last_name, “hpwd”: hashed_password})

Clojure Implementation

Programming Clojure was once again a real joy. Clojure REPL and especially the Cursive REPL is the best productivity tool I have ever used in any programming language. Once you have configured all your favorite hot keys e.g. for jumping between code editor and REPL editor using REPL to explore new APIs, try various ways to create functions etc makes programming an interesting interaction between the programmer and the code — and code evolves in an organic manner during this interaction.

(dynamodb/query (ss-aws-utils/get-dynamodb-config)
:table-name my-table
:select “ALL_ATTRIBUTES”
:key-conditions {:pgid {:attribute-value-list [(str pg-id)]
:comparison-operator “EQ”}
:pid {:attribute-value-list [(str p-id)]
:comparison-operator “EQ”}})

Conclusions

Using local DynamoDB Docker container instance makes DynamoDB development really fast. Clojure is really superb for data manipulation, and using Clojure with amazonica library it is pretty effortless to work with DynamoDB.

I’m a Software architect and developer. Currently implementing systems on AWS / GCP / Azure / Docker / Kubernetes using Java, Python, Go and Clojure.