Using Git to track mysql schema - some questions

Question

If this is recommended ?

Can I ask some git command examples about how to track versions of mysql schema?

Should we use another repository other then the on

Answer 1

The following includes a git pre-commit hook to capture mysql database/schema, given user='myuser', password='mypassword', database_name='dbase1'. Properly bubbles errors up to the git system (the exit 0's in other answers could be dangerous and may not handle error scenarios properly). Optionally, can add a database import to a post-checkout hook (when capturing all the data, not just schema), but take care given your database size. Details in bash-script comments below.

pre-commit hook:

#!/bin/bash

# exit upon error
set -e
# another way to set "exit upon error", for readability
set -o errexit

mysqldump -umyuser -pmypassword dbase1 --no-data=true > dbase1.sql

# Uncomment following line to dump all data with schema,
# useful when used in tandem for the post-checkout hook below.
# WARNING: can greatly expand your git repo when employing for
# large databases, so carefully evaluate before employing this method.
# mysqldump -umyuser -pmypassword dbase1 > dbase1.sql

git add dbase1.sql

(optional) post-checkout hook:

#!/bin/bash
# mysqldump (above) is presumably run without '--no-data=true' parameter.
set -e
mysql -umyuser -pmypassword dbase1 < dbase1.sql

Versions of apps, OS I'm running:

root@node1 Dec 12 22:35:14 /var/www# mysql --version
mysql  Ver 14.14 Distrib 5.1.54, for debian-linux-gnu (x86_64) using readline 6.2
root@node1 Dec 12 22:35:19 /var/www# git --version
git version 1.7.4.1
root@node1 Dec 12 22:35:22 /var/www# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 11.04
Release:        11.04
Codename:       natty
root@node1 Dec 12 22:35:28 /var/www#

Answer 2

IMO the best approach is described here: http://viget.com/extend/backup-your-database-in-git. For your convenience I repeat the most important pieces here.

The trick is to use mysqldump --skip-extended-insert, which creates dumps that can be better tracked/diffed by git.

There are also some hints regarding the best repository configuration in order to reduce disk size. Copied from here:

• core.compression = 9 : Flag for gzip to specify the compression level for blobs and packs. Level 1 is fast with larger file sizes, level 9 takes more time but results in better compression.
• repack.usedeltabaseoffset = true : Defaults to false for compatibility reasons, but is supported with Git >=1.4.4.
• pack.windowMemory = 100m : (Re)packing objects may consume lots of memory. To prevent all your resources go down the drain it's useful to put some limits on that. There is also pack.deltaCacheSize.
• pack.window = 15 : Defaults to 10. With a higher value, Git tries harder to find similar blobs.
• gc.auto = 1000 : Defaults to 6700. As indicated in the article it is recommended to run git gc every once in a while. Personally I run git gc --auto everyday, so only pack things when there's enough garbage. git gc --auto normally only triggers the packing mechanism when there are 6700 loose objects around. This flag lowers this amount.
• gc.autopacklimit = 10: Defaults to 50. Every time you run git gc, a new pack is generated of the loose objects. Over time you get too many packs which waste space. It is a good idea to combine all packs once in a while into a single pack, so all objects can be combined and deltified. By default git gc does this when there are 50 packs around. But for this situation a lower number may be better.

Old versions can be pruned via:

git rebase --onto master~8 master~7

(copied from here)

Answer 3

(shameless plug)

The dbvc commandline tool allows you to manage your database schema updates in your repository.

It creates and uses a table _dbvc in the database which holds a list of the updates that are run. You can easily run the updates that haven't been apply to your database schema yet.

The tool uses git to determine the correct order of executing the updates.

DBVC usage

Show a list of commands

dbvc help

Show help on a specific command

dbvc help init

Initialise DBVC for an existing database.

dbvc init

Create a database dump. This is used to create the DB on a new environment.

mysqldump foobar > dev/schema.php

Create the DB using the schema.

dbvc create

Add an update file. These are used to update the DB on other environments.

echo 'ALTER TABLE `foo` ADD COLUMN `status` BOOL DEFAULT 1;' > dev/updates/add-status-to-foo.sql

Mark an update as already run.

dbvc mark add-status-to-foo

Show a list of updates that need to be run.

dbvc status

Show all updates with their status.

dbvc status --all

Update the database.

dbvc update

Answer 4

As brilliant as it sounds (the idea did occur to me as well), when I tried to implement it, I hit a wall. In theory, by using the --skip-extended-insert flag, despite initial dump would be big, the diffs between daily dumps should be minimal, hence the size increase over time of the repository could be assumed to be minimal as well, right? Wrong!

Git stores shapshots, not diffs, which means on each commit, it will take the entire dump file, not just the diff. Moreover, since the dump with --skip-extended-instert will use all field names on every single insert line, it will be huge compared to a dump done without --skip-extended-instert. This results in an explosion in size, the exact opposite what one would expect.

In my case, with a ~300MB sql dump, the repository went to gigabytes in days. So, what did I do? I first tried the same thing, only remove --skip-extended-instert, so that dumps will be smaller, and snapshots would be proportionally smaller as well. This approach held for a while, but in time it became unusable as well.

Still, the diff usage with --skip-extended-insert actually still seemed like a good idea, only, now I try to use subversion instead of git. I know, compared to git, svn is ancient history, yet it seems to work better, since it actually does use diffs instead of snapshots.

So in short, I believe best solution is doing the above, but with subversion instead of git.

Answer 5

If you're just tracking the schema, put all of the CREATE statements into one .sql file, and add the file to git.

$> mkdir myschema && cd myschema
$> git init
$> echo "CREATE TABLE ..." > schema.sql
$> git add schema.sql
$> git commit -m "Initial import"

Answer 6

Assuming you have a git repo already, do the following in a shell script or whatever:

#!/bin/bash -e
# -e means exit if any command fails
DBHOST=dbhost.yourdomain.com
DBUSER=dbuser
DBPASS=dbpass # do this in a more secure fashion
DBNAME=dbname
GITREPO=/path/to/git/repo
cd $GITREPO
mysqldump -h $DBHOST -u $DBUSER -p$DBPASS -d $DBNAME > $GITREPO/schema.sql # the -d flag means "no data"
git add schema.sql
git commit -m "$DBNAME schema version $(`date`)"
git push # assuming you have a remote to push to

Then start this script on a daily basis from a cron job or what have you.

EDIT: By placing a script in $gitdir/hooks/pre-commit (the name is important), the script will be executed before every commit. This way the state of the DB schema is captured for each commit, which makes sense. If you automatically run this sql script every time you commit, you will blow away your database, which does not make sense.

#!/bin/sh

This line specifies that it's a shell script.

mysqldump -u DBUSER -pDBPASSWORD  DATABASE --no-data=true> SQLVersionControl/vc.sql

This is the same as in my answer above; taking the DDL only from the database and storing it in a file.

git add SQLVersionControl/vc.sql

This adds the SQL file to every commit made to your repository.

exit 0

This exits the script with success. This is possibly dangerous. If mysqldump or git add fails, you may blow away something you wanted to keep.