Configuring Database Connection Pooling, Migration and Reloaded Workflow
In the last blog post, we bootstrapped the Clojure project using Mount and Aero. We are going to continue from we left off and configure database connection pooling, migration & Reloaded Workflow in this blog post.
This blog post is a part 2 of the blog series Building an E-Commerce Marketplace Middleware in Clojure.
Configuring Hikari-CP
Let's get started by adding the hikari-cp, a Clojure wrapper to HikariCP, and the Postgres driver dependencies in the project.clj
.
(defproject wheel "0.1.0-SNAPSHOT"
; ...
:dependencies [; ...
[org.postgresql/postgresql "42.2.6"]
[hikari-cp "2.8.0"]]
; ...
)
NOTE: If you already have a running (and jacked in) REPL, you need to stop and start it again after adding any dependencies in the
project.clj
file.
To configure the Hikari connection pool, Let's create a new file database.clj
in the infra
directory.
> touch src/wheel/infra/database.clj
Then define a mount state datasource
to manage the life-cycle of the connection pool.
(ns wheel.infra.database
(:require [wheel.infra.config :as config]
[mount.core :as mount]
[hikari-cp.core :as hikari]))
(defn- make-datasource []
(hikari/make-datasource (config/database))) ;<1>
(mount/defstate datasource
:start (make-datasource)
:stop (hikari/close-datasource datasource))
<1> Retrieves the database configuration and creates the datasource object.
Now if we start the application through Mount, we will get the following output.
wheel.infra.database=> (mount/start)
{:started ["#'wheel.infra.config/root"
"#'wheel.infra.database/datasource"]}
As the state datasource
depends on the config's root
state, Mount starts it first and then it starts the datasource
.
Database Migration Using Flyway
There are multiple libraries in Clojure (and Java) to perform database migration. Our preference is Flyway, based on our success in other Java projects.
Let's get started by adding the dependency in the project.clj
.
(defproject wheel "0.1.0-SNAPSHOT"
; ...
:dependencies [; ...
[org.flywaydb/flyway-core "5.2.4"]]
; ...
)
Then in the database.clj
file, import
the Flyway
namespace and add a new function migrate
(ns wheel.infra.database
; ...
(:import [org.flywaydb.core Flyway]))
; ...
(defn migrate []
(.. (Flyway/configure) ; <1>
(dataSource datasource)
(locations (into-array String ["classpath:db/migration"])) ; <2>
load
migrate))
<1> Creates an instance of Flyway and setup the configuration using the dot special form
<2> Setting the migration files path to db/migration
.
This path doesn't exist yet. So, let's create it.
> mkdir -p resources/db/migration
To verify the database migration, let's load the updated database.clj
in the REPL and invoke the migrate
function.
wheel.infra.database=> (mount/start)
{:started ["#'wheel.infra.config/root"
"#'wheel.infra.database/datasource"]}
wheel.infra.database=> (migrate)
0
wheel.infra.database=> (mount/stop)
{:stopped ["#'wheel.infra.database/datasource"]}
If we check the database now, it will have the flyway_schema_history
table.
> psql -d wheel
wheel=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------+-------+----------
public | flyway_schema_history | table | postgres
(1 row)
Separating Database Migration From Application Bootstrap
Performing database migration during application bootstrap has certain limitations, and it is a best practice to decouple it. In our application, we did it by having separate entry-points, one to start the application and another to migrate the database.
Let's create a new file core.clj
in the infra
directory.
> touch src/wheel/infra/core.clj
Then define two functions, start-app
and migrate-database
, to start the application and perform database migration, respectively.
(ns wheel.infra.core
(:require [wheel.infra.config :as config]
[wheel.infra.database :as db]
[mount.core :as mount]))
(defn start-app []
(mount/start))
(defn migrate-database []
(mount/start #'config/root #'db/datasource) ; <1>
(db/migrate) ; <2>
(mount/stop #'db/datasource)) ; <3>
<1> Invokes Mount's start
function with that states that we wanted to start. Note that Mount doesn't start the transitive dependent states in this function overload.
<2> Performs the database migration
<3> Stops the datasource after completing the migration.
Let's add the stop-app
function as well to stop the application.
(ns wheel.infra.core
;...
)
; ...
(defn stop-app []
(mount/stop))
Reloaded Workflow
Reloaded Workflow is one of the common practice in Clojure development to do interactive REPL driven development.
Adding this to our current codebase is straight-forward.
As a first step, add a Leiningen user profile called dev
in the 's project.clj
file.
(defproject wheel "0.1.0-SNAPSHOT"
; ...
:profiles { ;...
:dev {:source-paths ["dev"] ; <1>
:dependencies [[org.clojure/tools.namespace "0.3.1"]]}}) ; <2>
<1> Adds a new source-path to load the Clojure files from the dev
directory.
<2> Adds a dev dependency tools.namespace to reload the modified source files interactively.
Then, create a new file user.clj
in the dev
directory.
; dev/user.clj
(ns user
(:require [wheel.infra.core :refer [start-app ; <1>
stop-app
migrate-database]
:as infra]
[clojure.tools.namespace.repl :as repl]))
(defn reset [] ; <2>
(stop-app)
(repl/refresh :after 'infra/start-app))
<1> Makes start-app
, stop-app
and migrate-database
function to be available in the user
namespace.
<2> Adds a reset
function, which stops the application and reloads all the modified codes. Using the :after
parameter, we are informing the refresh
function to start the app after the reload.
To see it in action, stop the current REPL session and start it again. This time profile selection Calva prompt will include the dev
profile in the list of options.
When we select the dev
profile, it will start the Leiningen REPL along with the configuration specified the dev
profile.
Then from the REPL, we can include the user
profile and manage the application's life cycle.
wheel.core=> (in-ns 'user)
#namespace[user]
wheel.core=> (start-app)
{:started ["#'wheel.infra.config/root"
"#'wheel.infra.database/datasource"]}
wheel.core=> (stop-app)
{:stopped ["#'wheel.infra.database/datasource"]}
wheel.core=> (migrate-database)
{:stopped ["#'wheel.infra.database/datasource"]}
wheel.core=> (reset)
:reloading (wheel.infra.config wheel.infra.database
wheel.infra.core wheel.core wheel.core-test user)
{:started ["#'wheel.infra.config/root"
"#'wheel.infra.database/datasource"]}
Summary
In this blog post, we learnt how to configure the database connection pooling using Hikari, database migration using Flyway and Reloaded Workflow using the Mount and Leiningen profile. There are other libraries and ways to do this as well, but the approach that I described here is the one that worked for us.
I would love to learn your preferences and approaches.
The source code associated with this part is available on this GitHub repository.