Using Flyway to load data conditionally per environment

Question

Does flyway support conditional execution of a script, per environment?

For example, if I have test data, can I create a test data script folder thats only loaded i

Answer 1

You can use flyway placeholder:

Configure it on your enviroment config.properties:

flyway.placeholders.tableName=MY_TABLE

flyway.placeholders.name='Mr. Test'

Then, put it on your script: INSERT INTO ${tableName} (name) VALUES (${name});

I have used the flyway.locations too, but the placeholders is simpler than locations for simple changes.

Answer 2

For future visitors, this is a solution for DB specific sql, but applies to data loading too. https://flywaydb.org/documentation/faq#db-specific-sql

You can set the flyway.locations=sql/common,sql/data property, and this can be set to different values with spring profiles(dev/test/prod), omitting the sql/data scripts on production.

Answer 3

Maven profiles did not give me the flexibility I wanted. I came up with a strategy that uses ant to merge files. This allows me to have common scripts and then included additional data depending on the deploy type eg. prod, dev.

This is my project structure:

├── build.xml
├── database.properties
├── database.properties.template
├── lib
│   └── ant-contrib-1.0b3.jar
├── pom.xml
└── sql
    ├── common
    │   ├── V1.0__.sql
    │   ├── V1.2.1__.sql
    │   └── V1.3__.sql
    ├── dev
    │   ├── V1.0__.sql
    │   └── V1.3__.sql
    └── prod
        └── V1.0__.sql

database.properties.template file in the root folder, before running this has to manually be copied to database.properties and the username and password can be entered. database.properties should be ignored by VCS so that passwords don't end up in the repo.

deployType scripts are merged into the src/main/resources/db/migrate directory, here is the ant script that does that, note that scripts that are to be merged have the same name:

<project name="data" default="prepareSql">

    <property file="database.properties" />
    <property name="destDir" value="src/main/resources/db/migration" />

    <echo message="Deploy type: ${deployType}"/>

    <taskdef resource="net/sf/antcontrib/antcontrib.properties">
      <classpath>
        <pathelement location="lib/ant-contrib-1.0b3.jar"/>
      </classpath>
    </taskdef>

    <target name="prepareSql">
        <!-- ensure the dest dir exists -->
        <mkdir dir="${destDir}"/>

        <!-- clear out the dest dir -->
        <delete>
            <fileset dir="${destDir}">
                <include name="*" />
            </fileset>
        </delete>

        <!-- append the deploy type files to the common files, delegate to the append target -->
        <foreach target="append" param="file">
            <fileset dir="sql/common" casesensitive="yes">
                <include name="*" />
            </fileset>
        </foreach>
    </target>

    <target name="append">
        <basename property="basename" file="${file}" />
        <property name="destFile" value="${destDir}/${basename}"/>
        <echo message="Appending ${file} to ${destFile}" />

        <concat destfile="${destFile}" >
            <filelist dir="sql/common" files="${basename}" />
            <filelist dir="sql/${deployType}" files="${basename}" />
        </concat>
    </target>
</project>

This ant file is executed by maven, here is the pom config:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>data</groupId>
    <artifactId>data</artifactId>
    <version>1.1-SNAPSHOT</version>
    <packaging>pom</packaging>
    <name>data</name>

    <properties>
        <sqlBaseDir>filesystem:${basedir}/src/main/resources/</sqlBaseDir>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.7</version>
                <executions>
                    <execution>
                        <id>mergeScripts</id>
                        <phase>validate</phase>
                        <inherited>false</inherited>
                        <configuration>
                            <target>
                                <ant antfile="build.xml" target="prepareSql" />
                            </target>
                        </configuration>
                        <goals>
                            <goal>run</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.flywaydb</groupId>
                <artifactId>flyway-maven-plugin</artifactId>
                <version>3.2.1</version>
                <configuration>
                    <schemas>
                        <schema>common</schema>
                    </schemas>
                    <configFile>database.properties</configFile>
                    <table>flyway</table>
                    <locations>
                        <location>filesystem:${basedir}/src/main/resources/db/migration</location>
                    </locations>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.36</version>
        </dependency>
    </dependencies>
</project>

Should you require different data for different client distributions you could add dist directories into the sql directory and they could contain deployType sub directories.

Add the dist property to the database.properties.template file and then modify the append target in the build.xml to look like this:

<target name="append">
    <basename property="basename" file="${file}" />
    <property name="destFile" value="${destDir}/${basename}"/>
    <echo message="Appending ${file} to ${destFile}" />

    <concat destfile="${destFile}" >
        <filelist dir="sql/common" files="${basename}" />
        <filelist dir="sql/${deployType}" files="${basename}" />
            <filelist dir="sql/${dist}/common" files="${basename}" />
            <filelist dir="sql/${dist}/${deployType}" files="${basename}" />
    </concat>
</target>

Answer 4

If you are using maven, you can able to achieve it very easily through maven profiles concept. Please refer the following sample

pom.xml

<plugin>
    <groupId>com.googlecode.flyway</groupId>
    <artifactId>flyway-maven-plugin</artifactId>
    <version>2.3</version>
    <configuration>
        <url>jdbc:sqlserver://${db.hostname};databaseName=${db.name}</url>
        <user>${db.username}</user>
        <password>${db.password}</password>
        <initVersion>0</initVersion>
        <initDescription>Base Migration</initDescription>
        <table>Changelog_testproject</table>
        <locations>
           <location>filesystem:${sql.file.path}</location>
        </locations>
    </configuration>
</plugin>

<profiles>
    <profile>
        <id>dev</id>
        <properties>
            <profile.name>dev</profile.name>
            <sql.file.path>${basedir}/deploy/dev/sqldelta/sqlserver</sql.file.path> 
            <db.hostname>127.0.0.1:1433</db.hostname>
            <db.name>dev</db.name>
            <db.username>dev</db.username>
            <db.password>devadmin</db.password>
        </properties>
    </profile>
    <profile>
        <id>test</id>
        <properties>
            <profile.name>test</profile.name>
            <sql.file.path>${basedir}/deploy/test/sqldelta/sqlserver</sql.file.path>  
            <db.hostname>127.0.0.1:1433</db.hostname>
            <db.name>test</db.name>
            <db.username>test</db.username>
            <db.password>testadmin</db.password>
        </properties>
    </profile>
 </profiles>

flyway flywaydb