Testing and Local Development with MSSQL (1 of 2)

The most challenging part of designing an effective automated testing strategy is accurately simulating databases. Microsoft’s SQL Server (MSSQL) is particularly irksome because of its large footprint, lengthy startup time, plethora of configuration options, ability to store custom logic (stored procedures, user-defined functions, …), and exorbitant licensing fees. These characteristics make it onerous to materialize production equivalent instances within automated pipelines efficiently. This post is the first in a two-part series demonstrating a technique to assuage said difficulties.

The time-constrained can jump directly to the TL;DR.

MSSQL Container Image

Microsoft’s relatively recently¹ released MSSQL container images - available for both Windows and Linux - are a giant leap forward for local development and automated testing (see the Docker hub repo for full details). While it is possible to use local installations of MSSQL, the most notable advantage of containers is configuration management. Installing, patching, and maintaining MSSQL installations across an array of build servers and development machines is expensive. Thanks to the magic of containers, it’s easy to construct fully-featured instances with full fidelity across many environments. Try it yourself using the shell command below²:

docker run \
    -e 'ACCEPT_EULA=Y' \
    -e 'SA_PASSWORD=<YOUR_STRONG_PASSWORD>' \
    -p 1433:1433 \
    --rm \
    -d \
    mcr.microsoft.com/mssql/server:2019-latest

Take special note of the two environment variables specified in the command above. ACCEPT_EULA confirms your acceptance of the End User Licensing Agreement. Omission of this variable results in failure to initialize. SA_PASSWORD is also required - it, as the name suggests, sets the System Administrator (sa) password³. The MSSQL Password Policy specifies the minimum complexity requirements. A non-compliant password results in a silent failure.

Next, connect to the containerized instance using your favorite SQL management tool (SQL Server Management Studio (SSMS) shown below).

Finally, destroy the container:

docker stop $(docker ps --filter ancestor=mcr.microsoft.com/mssql/server:2019-latest -q)

MSSQL containers are ideal for local development and automated testing. There are, however, a few obstacles that are worthy of mention. The licensing restrictions differ based on the supplied MSQSL_PID environment variable. The default value is Developer, which is free for non-production use-specifying other values such as Standard or Enterprise subject the container to the same licensing fees as its non-containerized counterpart. Some advanced features such as replication, SQL agent, and full-text indexing aren’t supported by default⁴. Additionally, the ephemeral nature and performance impediments posed by MSSQL containers may deem them unsuitable for production workloads. There are options to overcome these limitations, but they are outside of the scope of this post.

Given the ability to easily create and destroy instances, the next challenge to overcome is initializing a database and associated schema⁵.

Generating DDL Scripts

To obtain the Data Definition Language (DDL) scripts from the database under test, one could use the SSMS GUI to generate scripts; however, that’s not conducive to automated testing. A better solution is to use mssql-scripter, a command-line utility that duplicates Management Studio’s scripting capabilities.

It’s a great tool; however, it has a notable flaw: dependency on an older version of libssl⁶. Running it on an active machine isn’t advisable. A dedicated mssql-scripter container is an excellent option because it sequesters the outdated library from the rest of the system. To create the container image, start with the docker file below:

FROM python:3.9-slim

WORKDIR /usr/src/app

RUN echo "deb http://security.debian.org/debian-security jessie/updates main" >> /etc/apt/sources.list
RUN apt-get update && \
    apt-get install -y libicu67 libssl1.0.0 libffi-dev libunwind8 python3-dev

RUN pip install --upgrade pip && \
    pip install mssql-scripter

ENTRYPOINT ["mssql-scripter"]

Build the container image:

docker build -t mssql-scripter .

Next, generate the DDL script (See the complete list of configuration options here: Usage Guide).

docker run --rm mssql-scripter --connection-string="<YOUR_CONNECTION_STRING>" > schema.sql

The shell command above generates a schema.sql file containing all the DDL scripts required to create an identical database complete with all its components (tables, functions, stored procedures, …). The final puzzle piece is to execute the SQL script against an MSSQL instance.

Executing DDL Scripts

Once again, it’s entirely possible to invoke a DDL script using a GUI tool like SSMS; however, this is problematic from an automation perspective. A better option is to use the sqlcmd utility baked into the base image. sqlcmd reads Transact-SQL statements from a command prompt and sends them to an MSSQL instance⁷. See the usage guide here. Experiment with the commands below to familiarize yourself.

# Create a MSSQL container
docker run \
    -e 'ACCEPT_EULA=Y' \
    -e 'SA_PASSWORD=<YOUR_STRONG_PASSWORD>' \
    --name sql-test \
    -p 1433:1433 \
    --rm \
    -d \
    mcr.microsoft.com/mssql/server:2019-latest

# Execute a SQL query inside the container
docker exec \
    sql-test /opt/mssql-tools/bin/sqlcmd -I -S localhost -U sa -P '<YOUR_STRONG_PASSWORD>' -q 'SELECT * FROM INFORMATION_SCHEMA.TABLES'

Invoking the DDL script is as easy as copying it to a container and sending it sqlcmd as shown below.

# Copy the DDL script created in the previous section to the container
docker cp ./schema.sql sql-test:/schema.sql

# Invoke the script using sqlcmd
docker exec sql-test /opt/mssql-tools/bin/sqlcmd -I -S localhost -U sa -P <YOUR_STRONG_PASSWORD> -d master -i /schema.sql

The result is an MSSQL container populated with a database and associated schema. Additionally, the container has guaranteed fidelity with the target. The tools required for robust automated testing and local development environments with MSSQL are present. As a bonus, installing any software (other than a container platform) is unnecessary⁸.

Conclusion

MSSQL container images, mssql-scripter, and sqlcmd make it relatively easy to build robust automated testing and local development solutions. For most situations, the process outlined in this post is more than sufficient. The major drawback to this approach is efficiency. Generating and invoking a DDL script is relatively expensive. The next post in this series outlines a process for embedding a schema directly into a container image.

TL;DR

The end-to-end process for creating MSSQL instances for test and local development that are guaranteed to have fidelity with their target environments is as follows:

1. Create an mssql-scripter container

    # Save this as Dockerfile
    FROM python:3.9-slim
   
    WORKDIR /usr/src/app
   
    RUN echo "deb http://security.debian.org/debian-security jessie/updates main" >> /etc/apt/sources.list
    RUN apt-get update && \
        apt-get install -y libicu67 libssl1.0.0 libffi-dev libunwind8 python3-dev
   
    RUN pip install --upgrade pip && \
        pip install mssql-scripter
   
    ENTRYPOINT ["mssql-scripter"]
   

    docker build -t mssql-scripter .
   

2. Generate a DDL script from the target database

    docker run --rm mssql-scripter --connection-string="<YOUR_CONNECTION_STRING>" > schema.sql
   

3. Stand up an MSSQL container instance (accessible on localhost:1433)

    docker run \
        -e 'ACCEPT_EULA=Y' \
        -e 'SA_PASSWORD=<YOUR_STRONG_PASSWORD>' \
        --name sql-test \
        -p 1433:1433 \
        --rm \
        -d \
        mcr.microsoft.com/mssql/server:2019-latest
   

4. Copy the DDL script into the running container

    docker cp ./schema.sql sql-test:/schema.sql
   

5. Invoke the DDL script using sqlcmd

    docker exec sql-test /opt/mssql-tools/bin/sqlcmd -I -S localhost -U sa -P <YOUR_STRONG_PASSWORD> -d master -i /schema.sql
   

While this is sufficient for most applications, generating and invoking a DDL script is relatively expensive. The next post in this series outlines a process for embedding a schema directly into a container image.