Creating a Choropleth Map with Clojurescript

World Statistics App.

Introduction

After implementing my web-store exercise using six languages (Clojure, Python, Javascript, Java, Go, and Kotlin) and with 5 data stores (CSV, DynamoDB, Azure Tables, PostgreSQL, and Datomic, read more about in my blog posts, e.g. Five Languages — Five Stories ) I was pretty much fed up with that exercise. I wanted to do something totally different. But what? I spent quite a lot of time figuring out what to do. I was dreaming to do something with Finnish Open data that could benefit the general public. I was browsing the Finnish Open data but couldn’t figure out anything useful that would contain a lot of data points. Then I decided to take a look at the World Bank Open data. Just out of curiosity I downloaded various metrics between the years 2002–2017, some 150.000 data points altogether. Then I just started to experiment with the data using the Clojure REPL — a great tool for studying and experimenting with data. Little by little a new exercise was formed in my head: Implement a full-stack app using Clojure and ClojureScript to filter and enrich the data and visualize the data sets using a frontend. This blog post depicts the story of this exercise.

Technology Overview

I used Clojure, and some Clojure libraries (like reitit) to implement the backend. The backend reads the data from CSV files that I downloaded from the Worldbank site, and manipulates the data so that we have the data and meta-data nicely as Clojure data structures stored as part of the Integrant state. I used kixi as the statistics library.

Backend — Creating the Data

In main.clj I read the Integrant configuration: the data files (the CSV files that I downloaded from the Worldbank, and the toposon file) and inject them to init.clj:

(defmethod ig/init-key :backend/data [_ {:keys [data-files topojson-file]}]
(init/get-data data-files topojson-file))
(defn get-data [data-files topojson-file]
(log/debug "ENTER get-data")
(let [topojson-country-codes (read-topojson-country-codes topojson-file)
ids (into {} (map (juxt :acr :id) topojson-country-codes))
data-points (get-data-points data-files ids)
{:keys [countries series years]} (get-metadata data-points)]
{:raw-points data-points
:points (remove-non-country-values data-points non-country-codes)
:countries countries
:series series
:years years
:country-codes (country-codes-to-names countries)
:series-codes (series-codes-to-names series)
:non-country-codes non-country-codes
:country-ids ids
:topojson-country-codes topojson-country-codes
}))
(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824
(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
:country_code :FIN,
:series-name "Hospital beds (per 1,000 people)",
:series-code :SH.MED.BEDS.ZS,
:year 2002,
:value 7.4,
:country-id 246}

Backend — Using Clojure REPL to Experiment with the Data

Using a real REPL is such a joy with Clojure. Let’s use it to experiment with the data:

(set! *print-length* 100)
=> 100 ; So that we don't accidentally list 150.000 data points in repl...
;; Require some namespaces.
(require '[worldstat.backend.data.world :as ws-world])
=> nil
(require '[worldstat.common.data.filter :as ws-filter])
=> nil
(require '[kixi.stats.core :as kixi])
=> nil
;; Check how many points there are...
(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824
;; What keys do we have in our data?
(keys (user/data))
=>
(:country-codes
:raw-points
:series
:non-country-codes
:country-ids
:points
:series-codes
:countries
:topojson-country-codes
:years)
;; Let's sort countries...
(sort-by :country_name (:countries (user/data)))
=>
({:country_name "Afghanistan", :country_code :AFG, :country-id 4}
{:country_name "Albania", :country_code :ALB, :country-id 8}
{:country_name "Algeria", :country_code :DZA, :country-id 12}
{:country_name "American Samoa", :country_code :ASM, :country-id 16}
...)
;; Get hospital beds, filter data points belonging to Finland, and get first data point.
(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
:country_code :FIN,
:series-name "Hospital beds (per 1,000 people)",
:series-code :SH.MED.BEDS.ZS,
:year 2002,
:value 7.4,
:country-id 246}
;; Let's check some basic statistics, mean and standard deviation for certain data.
(transduce (map :value) kixi/mean (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 3.668379451410944
(transduce (map :value) kixi/standard-deviation (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 2.601440577329428

Backend — API

I used the excellent Metosin reitit library for routing: routes.clj:

["/data" {:swagger {:tags ["data"]}}
["/metric/:metric" {:get {:summary "metric"
:parameters {:path [:map [:metric string?]]}
:responses {200 {:description "data"}}
:handler (fn [req]
(let [metric (get-in req [:parameters :path :metric])]
(-> (world/get-world-data env (keyword metric))
r/ok)))}}]
["/countries" {:get {:summary "countries"
:responses {200 {:description "data"}}
:handler (fn [req]
(-> (world/get-countries env)
r/ok))}}]
["/years" {:get {:summary "years"
:responses {200 {:description "data"}}
:handler (fn [req]
(-> (world/get-years env)
r/ok))}}]
["/metric-names" {:get {:summary "metric-names"
:responses {200 {:description "data"}}
:handler (fn [req]
(-> (world/get-metric-names env)
r/ok))}}]]]])
API Swagger documentation.

Frontend — Reagent and Re-frame

I used ClojureScript, Reagent, and Re-Frame to implement the frontend. For frontend development I used shadow-cljs. Reagent provides a nice interface to React, and Re-frame is a ClojureScript framework, that uses Reagent and provides simple state management to build single-page applications using Clojurescript. All this tooling works pretty seamlessly with the Clojure backend repl so that when I refresh the Integrant state (with one hotkey, of course) the frontend also refreshes. One can get a Clojurescript repl and connect to the frontend application running inside the browser if you need it. Usually, you don’t, since there are nice tools and tricks to examine the frontend state.

Clojurescript frontend tools.

CSS — Bulma

I used Bulma as a CSS framework. Bulma is really great — I barely added any custom css in this application. Let’s see part of the frontend using the hiccup notation:

[:div.row
[:div.columns
[:div.column.is-9
[oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]
[:div.column.auto
(if country-stats
[:div.rows
[:div.row
[:p.is-size-3 selected-country-name]]
[:div.row
[:p.is-size-5 "Min: " (ws-util/one-decimal (:min country-stats))]
[:p.is-size-5 "Max: " (ws-util/one-decimal (:max country-stats))]
[:p.is-size-5 "Mean: " (ws-util/one-decimal (:mean country-stats))]
[:p.is-size-5 "Std.dev: " (ws-util/one-decimal (:standard-deviation country-stats))]]
[:div.row
[oz/vega-lite (ws-data/country-year-diagram points selected-metric-name selected-country-code min max) (ws-util/vega-debug)]]]
[:p.is-size-4 "Click a country!"])]]]
[:div.row
[:p "(Missing data for country is shown with color gray)"]]
[:div.row
[:p "Read more about this app in my blog "
[:a {:href "https://www.karimarttila.fi/clojure/2021/01/19/world-statistics-exercise.html"} "www.karimarttila.fi"]]]
(defn slider [id initial step min max type]
(let [slider-value (r/atom initial)]
[:div.slider-content
[:input.slider.is-fullwidth {:id id :step step :min min :max max :value @slider-value :type type
:on-change (fn [event]
(.preventDefault event)
(let [new-value (.-value (.getElementById js/document id))]
(reset! slider-value new-value)
(re-frame/dispatch [::ws-state/select-year (js/parseInt new-value)])))}]]))

Frontend — Creating the Map

For data visualization, I used Vega Lite, and as a Clojure wrapper for Vega I used Metasoarous’ Oz library. The world-110m.json world map is from Vega.

(defn world-schema [points year metric-name min max]
{:schema "https://vega.github.io/schema/vega/v5.json"
:autosize "none"
:width 900
:height 500
:scales [{:name "color",
:type "quantize",
...
:marks [{:type "shape"
:from {:data "graticule"}
:encode {:update {:strokeWidth {:value 1}
...
{:type "shape"
:from {:data "world"}
:encode {:enter {:tooltip {:signal "datum.value === null ? {'country': datum.country_name, 'value': 'Missing data'} : {'country': datum.country_name, 'value': format(datum.value, '.2f')}"}
:href {:field "url" :type "nominal"}}
;; If missing data show as grey.
:update {:fill [{:test "datum.value === null" :value "gray"}
{:scale "color" "field" "value"}]}
:hover {:fill {:value "red"}}}
:transform [{:type "geoshape"
...
:data [{:name "countries"
:values (transduce (ws-filter/filter-by-year year) conj points)}
{:name "world"
:url "data/world-110m.json"
:format {:type "topojson" :feature "countries"}
:transform [{:type "lookup"
:from "countries"
:key "country-id"
:fields ["id"]
:values ["value" "country_name" "country_code"]}
{:type "formula"
:from "countries"
:expr "'#/country/' + datum.country_code" :as "url"}]}
{:name "graticule"
:transform [{:type "graticule"}]}]})
{selected-metric-code :code selected-metric-name :name} @(re-frame/subscribe [::ws-state/current-metric])
selected-year @(re-frame/subscribe [::ws-state/current-year])
{selected-country-code :code selected-country-name :name} @(re-frame/subscribe [::ws-state/current-country])
{:keys [points min max mean standard-deviation]} @(re-frame/subscribe [::ws-state/world-data selected-metric-code])
metric-names @(re-frame/subscribe [::ws-state/metric-names])
_ (if-not points (re-frame/dispatch [::ws-state/get-world-data selected-metric-code]))
country-stats (if selected-country-code (ws-data/get-stats points selected-country-code))]
...
[:div.column.is-9
[oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]

Using Vega Editor

Since the map component is a bit complicated and for a Vega newcomer the transform operations are a bit hairy Vega provides excellent development tools. See the three dots icon here:

Going to Vega Editor.
Vega Editor.

Frontend — Navigation etc.

There is very little navigation in the application. I used Reitit also to do the frontend navigation:

(def routes-dev
["/"
[""
{:name ::ws-state/home
:view home-page
:link-text "Home"
:controllers
[{:start (fn [& params] (ws-log/clog (str "Entering home page, params: " params)))
:stop (fn [& params] (ws-log/clog (str "Leaving home page, params: " params)))}]}]
["country/:country-code"
{:name ::ws-state/country
:parameters {:path {:country-code string?}}
:view home-page
:link-text "country"
:controllers
[{:start (fn [& params] (ws-log/clog (str "Entering country, params: " params)))
:stop (fn [& params] (ws-log/clog (str "Leaving country, params: " params)))}]}]])
(re-frame/reg-event-db
::navigated
(fn [db [_ new-match]]
(let [old-match (:current-route db)
new-path (:path new-match)
country-code (get-in new-match [:parameters :path :country-code])
country-name (get-country-name country-code)
controllers (rfc/apply-controllers (:controllers old-match) new-match)]
(ws-log/clog (str "::state/navigated, new-path: " new-path))
(ws-log/clog (str "::state/navigated, country-code: " country-code))
(cond-> (assoc db :current-route (assoc new-match :controllers controllers))
country-code (assoc :current-country {:code country-code :name country-name})))))

Docker Container and Heroku Deployment

I considered deploying the app to AWS but since I’m pretty stingy I didn’t want to spend my personal money on that. Then I realized that I can use Heroku Free Tier to deploy the app to the cloud (at least for the time the Heroku free dyno hours run out, or the World Bank wants to start sponsoring the app :-) ).

  • docker build -f Dockerfile . -t worldstat:latest
  • docker run -e PORT=9999 -it -p9999:9999 worldstat:latest
  • Then basically just follow the instructions in the Heroku documentation.
  • Create a Heroku account
  • heroku create
  • heroku container:login
  • heroku container:push web -a <YOUR-HEROKU-APP-NAME>
  • heroku container:release web -a <YOUR-HEROKU-APP-NAME>

Conclusions

This exercise was really interesting. I learned quite a lot how to experiment data using REPL in the backend. I also learned quite a lot how to create nice visualizations using the Vega library. I also learned that using Clojure and Clojurescript you can create quite powerful fullstack applications without the need being a Javascript / React guru.

I’m a Software architect and developer. Currently implementing systems on AWS / GCP / Azure / Docker / Kubernetes using Java, Python, Go and Clojure.