Before getting started, I am going to assume you already have node and npm installed and are somewhat familiar with the solidity programming language.

What will you learn from this article?

• Setting up a hardhat project from scratch

• Using Alchemy and Hardhat to fork mainnet

• How to leverage Uniswap v3 Flash Swaps to create a flash loan.

Let's get started! Fire up a terminal 🖥️

Setting up a hardhat project

First, create a directory for your project cd into it and use npm init -y to generate a package.json file.

> mkdir uniswapv3-flashloan
> cd uniswapv3-flashloan
> npm init -y

Next, go ahead and install Hardhat.

> npm install --save-dev hardhat

You can now use npx hardhat to generate a Hardhat project. Following the instructions create a TypeScript project and answer yes to the rest of the questions. This will generate the necessary files and install the required dependencies.

> npx hardhat      
888    888                      888 888               888
888    888                      888 888               888
888    888                      888 888               888
8888888888  8888b.  888d888 .d88888 88888b.   8888b.  888888
888    888     "88b 888P"  d88" 888 888 "88b     "88b 888
888    888 .d888888 888    888  888 888  888 .d888888 888
888    888 888  888 888    Y88b 888 888  888 888  888 Y88b.
888    888 "Y888888 888     "Y88888 888  888 "Y888888  "Y888

Welcome to Hardhat v2.12.3

✔ What do you want to do? · Create a TypeScript project
✔ Hardhat project root: · /home/alex/Code/Sandbox/uniswapv3-flashloan
✔ Do you want to add a .gitignore? (Y/n) · y
✔ Do you want to install this sample project's dependencies with npm (@nomicfoundation/hardhat-toolbox)? (Y/n) · y

At this point, you should have a directory structure that looks like this.

.
├── contracts
├── .gitignore
├── hardhat.config.ts
├── node_modules
├── package.json
├── package-lock.json
├── README.md
├── scripts
├── test
└── tsconfig.json

If you'd like you can delete the sample code that was generated in the contracts , scripts and test directories.

Great, you now have a TypeScript Hardhat project, ready to go!

Forking Minnet

Hardhat allows you to create a fork of mainnet locally. Essentially what this means is you can interact with contracts that are on the mainnet using your local set-up.

In this section, you'll create an application in Alchemy, generate an API key and make some modifications to the hardhat.config.ts file.

1. Create an Alchemy Application

If you haven't already created an account on Alchemy, go ahead and sign up. Once you have logged in, click the create app button.

Make sure you selected Ethereum for the chain and Mainnet for the network.

Once you have created the application click on View Details and then View Key.

Copy the HTTPS URL. Keep this handy you will use it in the section that follows.

2. Create a .env file

You wouldn't want to share your API key with the whole world. To avoid this you will create a .env file, this SHOULD NOT be committed to version control. Double-check that you have .gitignore a file at the root of your project with the following content.

node_modules
.env
coverage
coverage.json
typechain
typechain-types

# Hardhat files
cache
artifacts

Create a .env file with only one environment variable MAINNET_URL and paste the URL that you copied from Alchemy in the previous section. You should end up with something like this.

MAINNET_URL=https://eth-mainnet.g.alchemy.com/v2/<api_key>

Remember to replace <api_key> with the actual API key, you copied before.

3. Modify the hardhat.config.ts file

To read environment variables from a .env file you can make use of a package called dotenv. Go ahead and install that package now.

> npm install --save-dev dotenv

Finally, modify the hardhat.config.ts file to look like this.

import * as dotenv from "dotenv";

import { HardhatUserConfig } from "hardhat/config";
import "@nomicfoundation/hardhat-toolbox";

dotenv.config();

const config: HardhatUserConfig = {
  solidity: "0.8.17",
  networks: {
    hardhat: {
      forking: {
        url: process.env.MAINNET_URL || "",
      },
    },
  },
};

export default config;

And voila you now have a Hardhat project that forks mainnet!

Creating a Flash Loan Contract

Finally, it's time for the juicy parts! In this section, you will build the contract that will perform the flash loan on Uniswap V3.

Before diving into creating the contract a little bit of theory might help you better understand what we doing here. For each token pair, Uniswap has what is called a liquidity pool. For example, let's say you have ETH/USDC token pair, you could think of the liquidity pool as one big vault that is split into two sections. One section would hold X amount of ETH while the other section would hold the equivalent amount in USDC.

When you create a flash loan on Uniswap essentially what you doing is taking liquidity out of the pool and putting it back in a single transaction. Using the vault example above, you borrow some ETH or USDC from the vault and immediately put it back before anyone notices.

You could also borrow both ETH and USDC in the same transaction, as long as you have enough of the same token to pay the cost of the fees.

So as promised now for the juice!

The complete contract including the supporting interface and libraries will look like the below.

pragma solidity 0.8.17;

import "hardhat/console.sol";

import "./interfaces/IERC20.sol";
import "./interfaces/IUniswapV3Pool.sol";
import "./libraries/PoolAddress.sol";

contract Flashloan {
    address private constant FACTORY =
        0x1F98431c8aD98523631AE4a59f267346ea31F984;

    struct FlashCallbackData {
        uint256 amount0;
        uint256 amount1;
        address caller;
    }

    IERC20 private immutable token0;
    IERC20 private immutable token1;

    IUniswapV3Pool private immutable pool;

    constructor(
        address _token0,
        address _token1,
        uint24 _fee
    ) {
        token0 = IERC20(_token0);
        token1 = IERC20(_token1);
        pool = IUniswapV3Pool(getPool(_token0, _token1, _fee));
    }

    function getPool(
        address _token0,
        address _token1,
        uint24 _fee
    ) public pure returns (address) {
        PoolAddress.PoolKey memory poolKey = PoolAddress.getPoolKey(
            _token0,
            _token1,
            _fee
        );
        return PoolAddress.computeAddress(FACTORY, poolKey);
    }

    function flash(uint256 amount0, uint256 amount1) external {
        bytes memory data = abi.encode(
            FlashCallbackData({
                amount0: amount0,
                amount1: amount1,
                caller: msg.sender
            })
        );
        IUniswapV3Pool(pool).flash(address(this), amount0, amount1, data);
    }

    function uniswapV3FlashCallback(
        uint256 fee0,
        uint256 fee1,
        bytes calldata data
    ) external {
        require(msg.sender == address(pool), "not authorized");

        FlashCallbackData memory decoded = abi.decode(
            data,
            (FlashCallbackData)
        );

        // Do your abitrage below...

        console.log(token0.balanceOf(address(this)));
        console.log(token1.balanceOf(address(this)));

        // Repay borrow
        if (fee0 > 0) {
            token0.transferFrom(decoded.caller, address(this), fee0);
            token0.transfer(address(pool), decoded.amount0 + fee0);
        }
        if (fee1 > 0) {
            token1.transferFrom(decoded.caller, address(this), fee1);
            token1.transfer(address(pool), decoded.amount1 + fee1);
        }
    }
}

Inside the contracts directory create a new file called Flashloan.sol and copy the above contents into it. While you are at it go ahead and create two new directories called interfaces and libraries you will make use of this shortly.

Let's walk through the important parts of the contract.

address private constant FACTORY = 0x1F98431c8aD98523631AE4a59f267346ea31F984;

This is the address of the UniswapV3Factory contract. You can find a complete list of deployed contracts and their address in the Uniswap documentation here.

Moving on to the constructor.

    ...
    constructor(
        address _token0,
        address _token1,
        uint24 _fee
    ) {
        token0 = IERC20(_token0);
        token1 = IERC20(_token1);
        pool = IUniswapV3Pool(getPool(_token0, _token1, _fee));
    }
    ...

When deploying the contract the constructor takes three parameters. _token0 and _token1 are the addresses of the tokens that will form a pair, as mentioned at the beginning of this section. The _fee refers to the swapping fee of the token pair pool. As of Uniswap v3, there are four fee levels 0.01%, 0.05%, 0.30%, and 1%.

You will also notice the use of two interfaces in the constructor IERC20 and IUniswapV3Pool.

Inside of the interfaces directory you created previously two new files, IERC20.sol and IUniswapV3Pool.sol.

The IERC20 interface will allow you to interact with all tokens that have implemented the ERC-20 standard. Copy the below content into your IER20.sol file.

// SPDX-License-Identifier: MIT
// OpenZeppelin Contracts (last updated v4.6.0) (token/ERC20/IERC20.sol)

pragma solidity ^0.8.0;

/**
 * @dev Interface of the ERC20 standard as defined in the EIP.
 */
interface IERC20 {
    /**
     * @dev Emitted when `value` tokens are moved from one account (`from`) to
     * another (`to`).
     *
     * Note that `value` may be zero.
     */
    event Transfer(address indexed from, address indexed to, uint256 value);

    /**
     * @dev Emitted when the allowance of a `spender` for an `owner` is set by
     * a call to {approve}. `value` is the new allowance.
     */
    event Approval(
        address indexed owner,
        address indexed spender,
        uint256 value
    );

    /**
     * @dev Returns the amount of tokens in existence.
     */
    function totalSupply() external view returns (uint256);

    /**
     * @dev Returns the amount of tokens owned by `account`.
     */
    function balanceOf(address account) external view returns (uint256);

    /**
     * @dev Moves `amount` tokens from the caller's account to `to`.
     *
     * Returns a boolean value indicating whether the operation succeeded.
     *
     * Emits a {Transfer} event.
     */
    function transfer(address to, uint256 amount) external returns (bool);

    /**
     * @dev Returns the remaining number of tokens that `spender` will be
     * allowed to spend on behalf of `owner` through {transferFrom}. This is
     * zero by default.
     *
     * This value changes when {approve} or {transferFrom} are called.
     */
    function allowance(address owner, address spender)
        external
        view
        returns (uint256);

    /**
     * @dev Sets `amount` as the allowance of `spender` over the caller's tokens.
     *
     * Returns a boolean value indicating whether the operation succeeded.
     *
     * IMPORTANT: Beware that changing an allowance with this method brings the risk
     * that someone may use both the old and the new allowance by unfortunate
     * transaction ordering. One possible solution to mitigate this race
     * condition is to first reduce the spender's allowance to 0 and set the
     * desired value afterwards:
     * https://github.com/ethereum/EIPs/issues/20#issuecomment-263524729
     *
     * Emits an {Approval} event.
     */
    function approve(address spender, uint256 amount) external returns (bool);

    /**
     * @dev Moves `amount` tokens from `from` to `to` using the
     * allowance mechanism. `amount` is then deducted from the caller's
     * allowance.
     *
     * Returns a boolean value indicating whether the operation succeeded.
     *
     * Emits a {Transfer} event.
     */
    function transferFrom(
        address from,
        address to,
        uint256 amount
    ) external returns (bool);
}

source

Similar to the IERC20 interface, the IUniswapV3Pool interface is used to interact with theUniswapV3Pool contract. You can find a reference to this on Uniswaps' GitHub repository here. However, the only action you should be interested in is the flash function, which is what you will use to perform the flash loan. A more simplified version of this interface would look like this.

// SPDX-License-Identifier: GPL-2.0-or-later
pragma solidity >=0.5.0;

/**
 * @dev Simplified IUniswapV3Pool interface with only the flash action.
 */
interface IUniswapV3Pool {
    /// @notice Receive token0 and/or token1 and pay it back, plus a fee, in the callback
    /// @dev The caller of this method receives a callback in the form of IUniswapV3FlashCallback#uniswapV3FlashCallback
    /// @dev Can be used to donate underlying tokens pro-rata to currently in-range liquidity providers by calling
    /// with 0 amount{0,1} and sending the donation amount(s) from the callback
    /// @param recipient The address which will receive the token0 and token1 amounts
    /// @param amount0 The amount of token0 to send
    /// @param amount1 The amount of token1 to send
    /// @param data Any data to be passed through to the callback
    function flash(
        address recipient,
        uint256 amount0,
        uint256 amount1,
        bytes calldata data
    ) external;
}

Go ahead and copy the content above into your IUniswapV3Pool.sol file.

The getPool function is responsible for calculating the token pair pool address.

    ...
    function getPool(
        address _token0,
        address _token1,
        uint24 _fee
    ) public pure returns (address) {
        PoolAddress.PoolKey memory poolKey = PoolAddress.getPoolKey(
            _token0,
            _token1,
            _fee
        );
        return PoolAddress.computeAddress(FACTORY, poolKey);
    }
    ...

To do this you need to make use of the PoolAddress library provided by Uniswap. Inside the libraries directory, create a new file called PoolAddress.sol. Copy the content below into the file you just creates.

// SPDX-License-Identifier: GPL-2.0-or-later
pragma solidity >=0.5.0;

/// @title Provides functions for deriving a pool address from the factory, tokens, and the fee
library PoolAddress {
    bytes32 internal constant POOL_INIT_CODE_HASH = 0xe34f199b19b2b4f47f68442619d555527d244f78a3297ea89325f843f87b8b54;

    /// @notice The identifying key of the pool
    struct PoolKey {
        address token0;
        address token1;
        uint24 fee;
    }

    /// @notice Returns PoolKey: the ordered tokens with the matched fee levels
    /// @param tokenA The first token of a pool, unsorted
    /// @param tokenB The second token of a pool, unsorted
    /// @param fee The fee level of the pool
    /// @return Poolkey The pool details with ordered token0 and token1 assignments
    function getPoolKey(
        address tokenA,
        address tokenB,
        uint24 fee
    ) internal pure returns (PoolKey memory) {
        if (tokenA > tokenB) (tokenA, tokenB) = (tokenB, tokenA);
        return PoolKey({token0: tokenA, token1: tokenB, fee: fee});
    }

    /// @notice Deterministically computes the pool address given the factory and PoolKey
    /// @param factory The Uniswap V3 factory contract address
    /// @param key The PoolKey
    /// @return pool The contract address of the V3 pool
    function computeAddress(address factory, PoolKey memory key) internal pure returns (address pool) {
        require(key.token0 < key.token1);
        pool = address(
            uint160(
                uint256(
                    keccak256(
                        abi.encodePacked(
                            hex'ff',
                            factory,
                            keccak256(abi.encode(key.token0, key.token1, key.fee)),
                            POOL_INIT_CODE_HASH
                        )
                    )
                )
            )
        );
    }
}

source.

⚠️The computeAddress function in the above code is different than that of the source. I had to further wrap the hash with uint160 to get the contract to correctly compile. Further info can be found here.

Next is the flash function, this is the actual function that creates the flash loan.

    ...
    function flash(uint256 amount0, uint256 amount1) external {
        bytes memory data = abi.encode(
            FlashCallbackData({
                amount0: amount0,
                amount1: amount1,
                caller: msg.sender
            })
        );
        IUniswapV3Pool(pool).flash(address(this), amount0, amount1, data);
    }
    ...

The amount0 and amount1 correspond directly to the amount of token0 and/or token1 you wish to borrow. Essentially all this function does is call flash on the token pair pool.

To successfully get the amount you requested in your flash loan your contract must include a callback function name uniswapV3FlashCallback . This leads us to the final part of the contract.

    ...
    function uniswapV3FlashCallback(
        uint256 fee0,
        uint256 fee1,
        bytes calldata data
    ) external {
        require(msg.sender == address(pool), "not authorized");

        FlashCallbackData memory decoded = abi.decode(
            data,
            (FlashCallbackData)
        );

        // Do your abitrage below...

        console.log(token0.balanceOf(address(this)));
        console.log(token1.balanceOf(address(this)));

        // Repay borrow
        if (fee0 > 0) {
            token0.transferFrom(decoded.caller, address(this), fee0);
            token0.transfer(address(pool), decoded.amount0 + fee0);
        }
        if (fee1 > 0) {
            token1.transferFrom(decoded.caller, address(this), fee1);
            token1.transfer(address(pool), decoded.amount1 + fee1);
        }
    }
    ...

At this point, you are free to do what you want with the borrowed amount. Maybe leverage your position while yield farming or taking advantage of an arbitrage opportunity. As long as you cover the gas fees and pay back the amount borrowed in a single transaction.

As a final step to see that everything works you going to create a script that deploys the contract and executes the flash function locally.

Inside the scripts directory, create a file and call it flashloan.ts with the below contents.

import { ethers } from "hardhat";

async function main() {
  const WETH_ADDRESS = "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2";
  const USDC_ADDRESS = "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48";
  const POOL_FEE = 3000; // 0.30% tier

  // Need this to convert ETH into WETH (Wrapped Ether) to cover the fees
  const weth = await ethers.getContractAt("IWETH", WETH_ADDRESS);

  // Deploy Flashloan contract
  const Flashloan = await ethers.getContractFactory("Flashloan");
  const flashloan = await Flashloan.deploy(
    USDC_ADDRESS,
    WETH_ADDRESS,
    POOL_FEE
  );

  // Get some WETH to cover fee and approve Flashloan contract to use it.
  // Fee: 1 ETH * 0.3% = 0.003 ETH
  await weth.deposit({ value: ethers.utils.parseEther("0.003") });
  await weth.approve(flashloan.address, ethers.utils.parseEther("0.003"));

  // Execute flashloan to borrow 1 ETH.
  await flashloan.flash(0, ethers.utils.parseEther("1"));
}

// We recommend this pattern to be able to use async/await everywhere
// and properly handle errors.
main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

Open up your terminal cd to the project root and run npx hardhat run scripts/flashloan.ts. With any luck, you should see the console logs from within the contract.

❯ npx hardhat run scripts/flashloan.ts          
0
1000000000000000000

Conclusion

Give yourself a hand 👏 for making it this far, you just created a flash loan contract using Uniswap v3!

As a quick recap, you created a Hardhat project, set things up to fork mainnet and finally deployed and executed your contract locally.

Happy hacking!

If you enjoyed reading this article and would like to stay tuned for more, or just want to connect, follow me on Twitter @alexvanzyl

Series Content :book:

• Part 1: Laying the foundation
• Part 2: Migrations
• Part 3: Dockerize (this post)

Prerequisites

You will need to install:

• Docker; and
• Docker Compose

What we will cover in this post? :memo:

• Create a Dockerfile for our application
• Create Docker Compose files
• Update .env and config.py
• Create a prestart.sh file to run migrations

Create a Dockerfile for our application

Our Dockerfile will contain the details of how docker should build our server image. This is where our FastAPI application will run.

Dockerfile

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8

WORKDIR /app/

# Install Poetry
RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | POETRY_HOME=/opt/poetry python && \
    cd /usr/local/bin && \
    ln -s /opt/poetry/bin/poetry && \
    poetry config virtualenvs.create false

# Copy poetry.lock* in case it doesn't exist in the repo
COPY ./pyproject.toml ./poetry.lock* /app/

# Allow installing dev dependencies to run tests
ARG INSTALL_DEV=false
RUN bash -c "if [ $INSTALL_DEV == 'true' ] ; then poetry install --no-root ; else poetry install --no-root --no-dev ; fi"

# For development, Jupyter remote kernel, Hydrogen
# Using inside the container:
# jupyter lab --ip=0.0.0.0 --allow-root --NotebookApp.custom_display_url=http://127.0.0.1:8888
ARG INSTALL_JUPYTER=false
RUN bash -c "if [ $INSTALL_JUPYTER == 'true' ] ; then pip install jupyterlab ; fi"

COPY . /app
ENV PYTHONPATH=/app

Info: The base image we using is one created by the author of FastAPI, Sebastián Ramírez. See the repository on GitHub here.

The comments in the Dockerfile provide an explanation into what happens when the image gets built. By including JupyterLab, this gives us the ability to spin up a notebook and experiment with ideas while having full access to our FastAPI application.

Since we copy the whole context of our project into the image, it's usually a good idea to create a .dockerignore file. This will prevent adding any unnecessary files to our image as well as sensitive information. This helps to reduce the size of the image and prevent potential security risks.

.dockerignore

.env
.env.*
.vscode
.idea
*.egg-info
.mypy-cache
.cache
.git
.gitignore
docker-compose.*
Dockerfile
*.md

Create Docker Compose files

We will use both a docker-compose.yml and docker-compose.override.yml file to orchestrate our services.

The services we will be running are PostgreSQL, pgAdmin (a graphical interface to administer PostgreSQL), and our FastAPI application server.

docker-compose.yml

version: "3.3"

services: 
  db:
    image: postgres:12
    volumes:
      - db-data:/var/lib/postgresql/data/pgdata
    env_file:
      - .env
    environment:
      - PGDATA=/var/lib/postgresql/data/pgdata

  pgadmin:
    image: dpage/pgadmin4
    depends_on:
      - db
    env_file:
      - .env

  server:
    build:
      context: ./
      dockerfile: Dockerfile
      args:
        INSTALL_DEV: ${INSTALL_DEV-false}
    depends_on:
      - db
    env_file:
      - .env

volumes:
  db-data:

All our services declare the environment variables from reading the .env file. We need to make adjustments to this file, which we will do shortly. For PostgreSQL, we use the official image and for pgAdmin, we use the image created by dpage, listed on Docker Hub.

Info: You can find the details of the PostgreSQL image here and pgAdmin here

You'll notice we haven't exposed any ports for any of our services. Usually, in production our application would be served behind some sort of proxy, for example, nginx over https protocol. Yet while developing it would be nice to access our services locally. This is where we can make use of the docker-compose.override.yml file. Let's do so now.

docker-compose.override.yml

version: "3.3"

services: 
  pgadmin:
    ports:
      - "8080:8080"

  server:
    ports:
      - "8888:8888"
      - "80:80"
    volumes:
      - ./:/app
    environment:
      - JUPYTER=jupyter lab --ip=0.0.0.0 --allow-root --NotebookApp.custom_display_url=http://127.0.0.1:8888
    build:
      context: ./
      dockerfile: Dockerfile
      args:
        INSTALL_DEV: ${INSTALL_DEV-true}
        INSTALL_JUPYTER: ${INSTALL_JUPYTER-true}
    command: /start-reload.sh

For pgAdmin it is pretty straightforward we exposing port 8080.

The server has a bit more going on though. First, we expose the ports 80 used to serve our API and 8888 used to access JupyterLab. Also by default, we install our development dependencies and JupyterLab. For our command, we tell our server that we want to reload after each file change. The changes in files are detected because we create a volume binding between our host machine and the container.

Info: The start-reload.sh file is part of the base image. You can see the content of this file here.

Update .env and config.py

We will update our .env file to include more environment variables. It should look like this now.

.env

# PostgreSQL
POSTGRES_SERVER=db
POSTGRES_USER=postgres
POSTGRES_PASSWORD=password
POSTGRES_DB=app


# PgAdmin
PGADMIN_DEFAULT_EMAIL=admin@local.host
PGADMIN_DEFAULT_PASSWORD=password
PGADMIN_LISTEN_PORT=8080

We no longer need to read the .env file from our config.py file. Since the environment variables, are declared inside the running container. We will comment out line 33 in the config.py file.

app/config.py

...
    class Config:
        case_sensitive = True

        # If you want to read environment variables from a .env
        # file instead un-comment the below line and create the
        # .env file at the root of the project.

        # env_file = ".env"


settings = Settings()

Create prestart.sh file

By default, the base image we use for our server looks for a prestart.sh file. This is handy to run scripts before our application starts. We will use it to run our migrations on pre-start.

prestart.sh

#! /usr/bin/env bash

# Let the DB start
sleep 10;
# Run migrations
alembic upgrade head

Let's also give this file execution permissions.

chmod +x prestart.sh

Conclusion :bulb:

Finally, we can now spin up our service using docker-compose up -d. We now can access all our services locally like before.

docker-compose up -d

FastAPI Swagger UI

http://localhost/docs

pgAdmin

http://localhost:8080

• email: admin@local.host
• password: password

JupyterLab

Running JypyterLab requires spinning it up inside our server container. Like so.

docker-compose exec server bash

# Inside the container
root@a09f3c12bf5e:/app# $JUPYTER

This will print a link that should look something like this http://127.0.0.1:8888/?token=<token>

If you have been following since part one, congratulations on making this far! :trophy:

We now have an application that is fully containerized. :whale:

The final code for this post can be found on GitHub.

If you enjoyed reading this article and would like to stay tuned for more, or just want to connect, follow me on twitter @alexvanzyl.

Series Content :book:

• Part 1: Laying the foundation
• Part 2: Migrations (this post)
• Part 3: Dockerize

Warning: To avoid any issues, make sure to drop the posts table from your database and run poetry install again.

What we will cover in this post? :memo:

• What is Alembic?
• Install Alembic
• Restructure project to support auto migrations
• Create a migrations directory
• Configure Alembic
• Generate migration file

What is Alembic? :thinking:

From Alembic's GitHub repository.

Alembic is a database migrations tool written by the author of SQLAlchemy.

Install Alembic

Since we will be running many commands in our virtual environment, let's switch that shell now.

poetry shell

The first thing we need to do is add Alembic as a dependency.

poetry add alembic

Create a migrations directory :file_folder:

Now that we have Alembic installed let's go ahead and generate the migrations directory.

alembic init migrations

After executing the above command a migrations directory and alembic.ini file will be generated. Our new project structure will look like this now.

.
├── alembic.ini
├── app
├── migrations
├── mypy.ini
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests

The alembic.ini file holds the configurations parameter for Alembic, such as the path to our migration scripts. Let's also go over the content of the migrations directroy.

migrations
├── env.py
├── README
├── script.py.mako
└── versions

The env.py file reads the configurations from the alembic.ini and handles running the migration scripts. We can also edit this file and tweak to our needs, something we will do later.

The README file can be used to add informative details about our migrations. We will leave this file as-is is for now.

The script.py.mako file is a Mako template file. Alembic uses this template when generating revision files (migration scripts).

Finally, we have the versions directory this is where all our revision files will live.

Restructure project to support auto migrations

In order for us to take advantage of Alembics auto generated migrations we need to restructure our project and make some minor changes to existing files.

Let's create a new db directory now.

mkdir app/db

We going to add three new files to this directroy, __init__.py, base.py and session.py

touch app/db/__init__.py app/db/base.py app/db/session.py

What we now have is an empty db package. This will replace the db.py module and we will delete this file later.

Edit __init__.py file.

app/db/__init__.py

from .base import Base  # noqa

# Import all the models, so that the Base class 
# has them before being imported by Alembic.
from .. import models # noqa

The schema for each table is derived from the attributes specified on each model. We import all our models afterwards to make sure the Base.metadata attribute gets populates with all the table schemas. Alembic makes use of this when using autogenerate.

Info: Base.metadata.tables contains a collection of SQLAlchemy Table objects.

Edit base.py file.

We going to move the original Base class from the db.py module into this file, but we will make a few changes to it.

First, we going to update our Base class to automatically generate the __tablename__ attribute for or models. The name will be derived from the model class name. For example, in our case, the Post class would generate a post table.

I prefer my table names to be in plural form, so instead of post it would posts. For this, we can make use of the inflect.py module. If you share the same preference, then let's install that now quick.

poetry add inflect

app/db/base.py

from typing import Any    

import inflect     
from sqlalchemy.ext.declarative import as_declarative, declared_attr    

p = inflect.engine()    


@as_declarative()    
class Base:    
    id: Any    
    __name__: str    

    # Generate __tablename__ automatically in plural form.   
    # i.e 'Post' model will generate table name 'posts'   
    @declared_attr    
    def __tablename__(cls) -> str:    
        return p.plural(cls.__name__.lower())

The main changes to the Base class are we added a new __name__ and declarative __tablename__ attribute.

This means we no longer need to declare the __tablename__ attribute on our models. Let’s edit the Post model in our models.py file to reflect this change.

Update models.py file

app/models.py

from uuid import uuid4

from sqlalchemy import Column, String, Text
from sqlalchemy.dialects.postgresql import UUID

from .db.base import Base


class Post(Base):
    id = Column(UUID(as_uuid=True), primary_key=True, index=True, default=uuid4)
    title = Column(String)
    body = Column(Text)

We updated our import statement for the Base class and dropped the __tablename__ attribute on the Post model.

Edit session.py file

Again we can copy the session details from db.py file into this one.

app/db/session.py

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

from ..config import settings

engine =  create_engine(settings.SQLALCHEMY_DATABASE_URI, pool_pre_ping=True)
SessionLocal =  sessionmaker(autocommit=False, autoflush=False, bind=engine)

Update main.py file

We need to update our import statements on line 8 and 9. Also, let's comment out line 13 since we will be using migrations.

app/main.py

...
from . import actions, schemas
from .db.session import SessionLocal
...

# Create all tables in database.
# Comment this out if you using migrations.
# models.Base.metadata.create_all(bind=engine)
...

We can now delete the db.py file.

rm app/db.py

Configure Alembic :gear:

Update alembic.ini file

We going to configure our connection details in the env.py file. So let's open up the alembic.ini and comment out the line sqlalchemy.url = driver://user:pass@localhost/dbname, like below.

alembic.ini

...
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8

# sqlalchemy.url  = driver://user:pass@localhost/dbname
...

Update env.py file

Let's configure the env.py file now.

migrations/env.py line 21:22

...
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata

from app.db import Base # noqa
target_metadata = Base.metadata
...

Note: We import the Base class from app.db and not app.db.base for reason explained before.

Next we will create our own function the will get the URL to our database connection.

migrations/env.py line 31:38

...
def get_url():
    from app.config import settings

    user = settings.POSTGRES_USER
    password = settings.POSTGRES_PASSWORD
    server = settings.POSTGRES_SERVER
    db = settings.POSTGRES_DB
    return f"postgresql://{user}:{password}@{server}/{db}"
...

All we are doing here is importing and using our database configuration settings to generate the connection URL.

To make use of this new function we need to update the run_migrations_offline and run_migrations_online functions. Let's do that now.

migrations/env.py line 53 and 72:78

...
def run_migrations_offline():
    ...
    url = get_url()
    ...

def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    configuration = config.get_section(config.config_ini_section)
    configuration["sqlalchemy.url"] = get_url()
    connectable = engine_from_config(
        configuration,
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )
    ...
...

The final env.py file should like this now.

migrations/env.py

from logging.config import fileConfig

from sqlalchemy import engine_from_config
from sqlalchemy import pool

from alembic import context

# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config

# Interpret the config file for Python logging.
# This line sets up loggers basically.
fileConfig(config.config_file_name)

# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata

from app.db import Base # noqa
from app import models # noqa
target_metadata = Base.metadata

# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.


def get_url():
    from app.config import settings

    user = settings.POSTGRES_USER
    password = settings.POSTGRES_PASSWORD
    server = settings.POSTGRES_SERVER
    db = settings.POSTGRES_DB
    return f"postgresql://{user}:{password}@{server}/{db}"


def run_migrations_offline():
    """Run migrations in 'offline' mode.

    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.

    Calls to context.execute() here emit the given string to the
    script output.

    """
    url = get_url()
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )

    with context.begin_transaction():
        context.run_migrations()


def run_migrations_online():
    """Run migrations in 'online' mode.

    In this scenario we need to create an Engine
    and associate a connection with the context.

    """
    configuration = config.get_section(config.config_ini_section)
    configuration["sqlalchemy.url"] = get_url()
    connectable = engine_from_config(
        configuration,
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    connectable = engine_from_config(
        config.get_section(config.config_ini_section),
        prefix="sqlalchemy.",
        poolclass=pool.NullPool,
    )

    with connectable.connect() as connection:
        context.configure(
            connection=connection, target_metadata=target_metadata
        )

        with context.begin_transaction():
            context.run_migrations()


if context.is_offline_mode():
    run_migrations_offline()
else:
    run_migrations_online()

Generate migration file :sparkles:

Finally, at this point we can now generate our first migraion script.

alembic revision --autogenerate -m "Create posts table"

This will generate a new migration file in the migrations/versions/ directory. In my case it created a file named ee48ba03fe9f_create_posts_table.py with the below content.

migrations/versions/ee48ba03fe9f_create_posts_table.py

"""Create posts table

Revision ID: ee48ba03fe9f
Revises: 
Create Date: 2020-05-24 15:39:50.164129

"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql

# revision identifiers, used by Alembic.
revision = "ee48ba03fe9f"
down_revision = None
branch_labels = None
depends_on = None


def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table(
        "posts",
        sa.Column("id", postgresql.UUID(as_uuid=True), nullable=False),
        sa.Column("title", sa.String(), nullable=True),
        sa.Column("body", sa.Text(), nullable=True),
        sa.PrimaryKeyConstraint("id"),
    )
    op.create_index(op.f("ix_posts_id"), "posts", ["id"], unique=False)
    # ### end Alembic commands ###


def downgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_index(op.f("ix_posts_id"), table_name="posts")
    op.drop_table("posts")
    # ### end Alembic commands ###

To run this migration now all we need to do is run the following command.

alembic upgrade head

We can also roll back changes by running the below.

alembic downgrade head

Conclusion :bulb:

Alembic is now in place to manage all our migration scripts. We had to make some minor changes to achieve this and refactored db.py file into multiple files that make up a package.

The final code for this post can be found on GitHub.

If you enjoyed reading this article and would like to stay tuned for more, or just want to connect, follow me on twitter @alexvanzyl.

Series Content :book:

• Part 1: Laying the foundation (this post)
• Part 2: Migrations
• Part 3: Dockerize

What will we cover in this post? :memo:

• Generate a base project with Poetry.
• Install FastAPI, SQLAlchemy and other dependencies.
• Create the necessary files that will serve as the base of the application

Before getting started... :warning:

I am going to make the following assumptions:

• that you have a basic understanding of Python and Python types;
• that you already have the below installed:
  • Python 3.6+ (I am using Python 3.8)
  • PostgreSQL; and
  • Poetry

Without further ado, let's get started :slightly_smiling_face:

Create a base project with Poetry :running:

Open up a terminal and enter the below command.

poetry new app

After running the above command a new directory called app has been created. Let's go ahead and cd into that directory now.

cd app

The directory structure should look like the below.

.
├── app
│   └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
    ├── __init__.py
    └── test_app.py

Let's quickly go over what we have here.

The app directory is our main python package.

A directory with a __init__.py file in it is considered a package in Python. Any .py files we add to this directory will be considered modules of this package. Usually, this file is empty but in this case, Poetry has gone ahead and added __version__ = '0.1.0'.

The pyproject.toml file is where all our dependencies will be added to. Later on, when we start installing our dependencies you will notice a poetry.lock file will be created, more on that later.

The README file can be used to add details about the project or any useful instructions that will help other developers working on the project. Personally, I prefer writing documentation in Markdown over reStructuredText so I will go ahead and rename README.rst to README.md. Feel free to do the same or leave it as is.

mv README.rst README.md

Finally, we have our tests directory that contains all the unit tests.

Install FastAPI and other dependencies :package:

In this section, we will install only the required dependencies to get a basic CRUD ( Create, Read, Update, Delete) application going.

What we will be installing?

• FastAPI - this goes without saying :slightly_smiling_face:
• SQLAlchemy Object Relational Mapper (ORM)
• psycopg2-binary PostgreSQL database adapter
• Uvicorn a lightning-fast ASGI server

poetry add fastapi sqlalchemy psycopg2-binary uvicorn

We will also install the following development dependencies, mainly to maintain code quality and for testing.

• pytest testing framework
• Mypy static type checker for Python
• sqlalchemy-stubs Mypy plug-in and type stubs for SQLAlchemy
• Flake8 for code linting
• autoflake removes unused imports and unused variables
• isort sort import statements
• black Python code formatter

poetry add -D mypy sqlalchemy-stubs flake8 autoflake isort black

Note: We have not included pytest in the above command as it is usually installed by default when using Poetry to create a new project.

At this point, nothing has really changed in our directory structure but you will notice that the pyproject.toml file has been updated and a new poetry.lock file has been created. The poetry.lock file locks the installed dependencies to a specific version. This is in particular helpful when multiple developers are working on the same project, to ensure everyone is using the same versions of each package.

To recap our directory structure should look something like this now.

.
├── app
│   └── __init__.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
    ├── __init__.py
    └── test_app.py

Add project files :page_facing_up:

In this section, we will start adding the files that will make up the base of our application.

The first file we will create is the main.py file, it will serve as the entry point to our application and house all our routes. Let's create this file now under the app package directory.

main.py

app/main.py

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def index():
    return {"message": "Hello world!"}

At this point, we actually have a basic application that we can run. If we switch back to our terminal and run the following commands.

poetry shell
uvicorn app.main:app

Tip: If you want the server to reload on file changes you can use the --reload flag, like so uvicorn app.main:app --reload

Now if we head over to a browser and hit http://127.0.0.1:8000 we will be greeted with {"message":"Hello world!"}. FastAPI also gives us API documentation out of the box so if you now navigate to http://127.0.0.1:8000/docs you will now see the Swagger UI. Pretty awesome, right! :metal:

We will come back later and update the main.py file but for now, let's hit Ctrl+C in the terminal to stop Uvicorn and continue adding the rest of our files.

Next, let's create the db.py under the same directory. This file will contain our database session and a base class that all models will extend from.

db.py

app/db.py

from typing import Any

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import as_declarative
from sqlalchemy.orm import sessionmaker

from .config import settings

engine = create_engine(settings.SQLALCHEMY_DATABASE_URI, pool_pre_ping=True)
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)


@as_declarative()
class Base:
    id: Any

Info: You can read more about the sessionmaker function here and as_declarative decorator here.

You may have noticed we import settings from config but we haven't actually created that file yet, so let's do so now.

config.py

app/config.py

from typing import Any, Dict, Optional

from pydantic import BaseSettings, PostgresDsn, validator


class Settings(BaseSettings):
    POSTGRES_SERVER: str
    POSTGRES_USER: str
    POSTGRES_PASSWORD: str
    POSTGRES_DB: str

    SQLALCHEMY_DATABASE_URI: Optional[PostgresDsn] = None

    @validator("SQLALCHEMY_DATABASE_URI", pre=True)
    def assemble_db_connection(cls, v: Optional[str], values: Dict[str, Any]) -> Any:
        if isinstance(v, str):
            return v
        return PostgresDsn.build(
            scheme="postgresql",
            user=values.get("POSTGRES_USER"),
            password=values.get("POSTGRES_PASSWORD"),
            host=values.get("POSTGRES_SERVER"),
            path=f"/{values.get('POSTGRES_DB') or  ''}",
        )

    class Config:
        case_sensitive = True
        env_file = ".env"


settings = Settings()

Info: When loading configurations from a .env file the python-dotenv package is required.

Here we are making use of Pydantic's settings management. By default, the BaseSettings class will try to read the environment variables set at system level using os.environ. However in our case instead we are specifying that we would like our environment variables to be read from a .env file. Pydantic relies on the python-dotenv package to achieve this, let's add it as a dependency now.

poetry add python-dotenv

And now we will create the .env file at the root of the project directory.

.env

.env

# PostgreSQL
POSTGRES_SERVER=localhost
POSTGRES_USER=postgres
POSTGRES_PASSWORD=password
POSTGRES_DB=app

Make sure to edit this file to reflect your set up.

.gitignore

Since the .env file can contain sensitive information we wouldn't want to commit this to version control. So now would probably be a good time to add a .gitignore file to our project. We will copy the Python .gitignore template provided by GitHub here. .gitignore

wget https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore
mv Python.gitignore .gitignore

Next, we will create a models.py and schemas.py file.

models.py

The models.py file will contain all our models that extend from the SQLAlchemy Base class we defined in db.py We will create that file now with an example User model.

app/models.py

from uuid import uuid4
from sqlalchemy import Column, String, Text
from sqlalchemy.dialects.postgresql import UUID

from .db import Base


class Post(Base):
    __tablename__ = "posts"

    id = Column(UUID(as_uuid=True), primary_key=True, index=True, default=uuid4)
    title = Column(String)
    body = Column(Text)

schemas.py

Let's create the schemas.py file now. This file will contain all our Pydantic models. Under the hood, FastAPI makes use of these models to validate the incoming request body, parse the response body and generate automatic docs for our API. Really cool, at least I think so! :ok_hand:

app/schemas.py

from typing import Optional

from pydantic import BaseModel, UUID4


# Shared properties
class PostBase(BaseModel):
    title: Optional[str] = None
    body: Optional[str] = None


# Properties to receive via API on creation
class PostCreate(PostBase):
    title: str
    body: str


# Properties to receive via API on update
class PostUpdate(PostBase):
    pass


class PostInDBBase(PostBase):
    id: Optional[UUID4] = None

    class Config:
        orm_mode = True


# Additional properties to return via API
class Post(PostInDBBase):
    pass


# Additional properties stored in DB
class PostInDB(PostInDBBase):
    pass

The final file we will create for now is the actions.py file. This file will contain all our use cases or actions that will be performed, such as CRUD operations.

actions.py

app/actions.py

from typing import Any, Dict, Generic, List, Optional, Type, TypeVar, Union

from fastapi.encoders import jsonable_encoder
from pydantic import UUID4, BaseModel
from sqlalchemy.orm import Session

from . import schemas
from .db import Base
from .models import Post

# Define custom types for SQLAlchemy model, and Pydantic schemas
ModelType = TypeVar("ModelType", bound=Base)
CreateSchemaType = TypeVar("CreateSchemaType", bound=BaseModel)
UpdateSchemaType = TypeVar("UpdateSchemaType", bound=BaseModel)


class BaseActions(Generic[ModelType, CreateSchemaType, UpdateSchemaType]):
    def __init__(self, model: Type[ModelType]):
        """Base class that can be extend by other action classes.
           Provides basic CRUD and listing operations.

        :param model: The SQLAlchemy model
        :type model: Type[ModelType]
        """
        self.model = model

    def get_all(
        self, db: Session, *, skip: int = 0, limit: int = 100
    ) -> List[ModelType]:
        return db.query(self.model).offset(skip).limit(limit).all()

    def get(self, db: Session, id: UUID4) -> Optional[ModelType]:
        return db.query(self.model).filter(self.model.id == id).first()

    def create(self, db: Session, *, obj_in: CreateSchemaType) -> ModelType:
        obj_in_data = jsonable_encoder(obj_in)
        db_obj = self.model(**obj_in_data)  # type: ignore
        db.add(db_obj)
        db.commit()
        db.refresh(db_obj)
        return db_obj

    def update(
        self,
        db: Session,
        *,
        db_obj: ModelType,
        obj_in: Union[UpdateSchemaType, Dict[str, Any]]
    ) -> ModelType:
        obj_data = jsonable_encoder(db_obj)
        if isinstance(obj_in, dict):
            update_data = obj_in
        else:
            update_data = obj_in.dict(exclude_unset=True)
        for field in obj_data:
            if field in update_data:
                setattr(db_obj, field, update_data[field])
        db.add(db_obj)
        db.commit()
        db.refresh(db_obj)
        return db_obj

    def remove(self, db: Session, *, id: UUID4) -> ModelType:
        obj = db.query(self.model).get(id)
        db.delete(obj)
        db.commit()
        return obj


class PostActions(BaseActions[Post, schemas.PostCreate, schemas.PostUpdate]):
    """Post actions with basic CRUD operations"""

    pass


post = PostActions(Post)

Before going back and updating our main.py file, let's review our final directory structure.

.
├── app
│   ├── actions.py
│   ├── config.py
│   ├── __init__.py
│   ├── main.py
│   ├── models.py
│   └── schemas.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
    ├── __init__.py
    └── test_app.py

Let's update our main.py file now and connect all the dots.

Update main.py

app/main.py

from typing import Any, List

from fastapi import Depends, FastAPI, HTTPException
from pydantic import UUID4
from sqlalchemy.orm import Session
from starlette.status import HTTP_201_CREATED, HTTP_404_NOT_FOUND

from . import actions, models, schemas
from .db import SessionLocal, engine

# Create all tables in the database.
# Comment this out if you using migrations.
models.Base.metadata.create_all(bind=engine)

app = FastAPI()


# Dependency to get DB session.
def get_db():
    try:
        db = SessionLocal()
        yield db
    finally:
        db.close()


@app.get("/")
def index():
    return {"message": "Hello world!"}


@app.get("/posts", response_model=List[schemas.Post], tags=["posts"])
def list_posts(db: Session = Depends(get_db), skip: int = 0, limit: int = 100) -> Any:
    posts = actions.post.get_all(db=db, skip=skip, limit=limit)
    return posts


@app.post(
    "/posts", response_model=schemas.Post, status_code=HTTP_201_CREATED, tags=["posts"]
)
def create_post(*, db: Session = Depends(get_db), post_in: schemas.PostCreate) -> Any:
    post = actions.post.create(db=db, obj_in=post_in)
    return post


@app.put(
    "/posts/{id}",
    response_model=schemas.Post,
    responses={HTTP_404_NOT_FOUND: {"model": schemas.HTTPError}},
    tags=["posts"],
)
def update_post(
    *, db: Session = Depends(get_db), id: UUID4, post_in: schemas.PostUpdate,
) -> Any:
    post = actions.post.get(db=db, id=id)
    if not post:
        raise HTTPException(status_code=HTTP_404_NOT_FOUND, detail="Post not found")
    post = actions.post.update(db=db, db_obj=post, obj_in=post_in)
    return post


@app.get(
    "/posts/{id}",
    response_model=schemas.Post,
    responses={HTTP_404_NOT_FOUND: {"model": schemas.HTTPError}},
    tags=["posts"],
)
def get_post(*, db: Session = Depends(get_db), id: UUID4) -> Any:
    post = actions.post.get(db=db, id=id)
    if not post:
        raise HTTPException(status_code=HTTP_404_NOT_FOUND, detail="Post not found")
    return post


@app.delete(
    "/posts/{id}",
    response_model=schemas.Post,
    responses={HTTP_404_NOT_FOUND: {"model": schemas.HTTPError}},
    tags=["posts"],
)
def delete_post(*, db: Session = Depends(get_db), id: UUID4) -> Any:
    post = actions.post.get(db=db, id=id)
    if not post:
        raise HTTPException(status_code=HTTP_404_NOT_FOUND, detail="Post not found")
    post = actions.post.remove(db=db, id=id)
    return post

Finally, if we run the server again and hit http://127.0.0.1:8000/docs we now have a basic API that can perform CRUD operations on our Post entity. :rocket:

uvicorn app.main:app

Conclusion :bulb:

If you have made it this far, well done! :+1:

We created a simple application that can serve as a good starting point for small to medium projects. There are still a number of things we can include in this base project such as migrations or adding Docker to our stack. (Hint: we will cover this in future posts, stay tuned :wink:)

The final code for this post can be found on GitHub.

If you enjoyed reading this article and would like to stay tuned for more, or just want to connect, follow me on twitter @alexvanzyl.