How to gracefully handle “Mysql2::Error: Invalid date” in ActiveRecord?

Question

I'm building a Rails 3.2 app upon a legacy database which also has some broken records in different tables. One of the issues giving the most headache is that it includes invalid dates.

I've setup a sandbox which I manually fixed one time to get my code working. Now it's time for deployment. For this reason, the sandbox is reset every night and copied from the live database, ferret indexes are rebuilt, and migrations are re-applied. We are going to deploy to the sandbox often to get in the last fixes before deploying to the live setup.

As the legacy PHP app and this new Rails app need to run in parallel for a few weeks to months, we cannot simply one-time-fix the dates (Update: just for clarification, that means they run on the same database at the same time). I need a way to automate this, maybe with a migration or rake task (I'd go for the latter).

But the problem is: ActiveRecord chokes on loading such records so I have no way to investigate the record and fix the dates by some hardcoded assumptions made in ruby code.

A second problem is that the legacy database has inconsistencies because the PHP code did not use transactions and some code paths are broken and left orphans and broken table constraints behind. I will deal with that as they occur, most of them is already taken care of in the models. First problem goes with the dates.

How would you usually fix this? Maybe there's even some magic gem out there which supports migrating legacy databases with broken records by intercepting exceptions and running some try-to-fix code...

The migration path uses MySQL, and three production environments (stable with live database, staging with the same database, and sandbox with a database clone reset every night). We decided against doing a one-time data mapping / migration because we cannot replace the complete legacy application in one step (it consists of a CMS with about 50000 articles, hundreds of topics, huge file database with images and downloads, supporting about 10 websites, about 12 years of data and work, messy PHP code from different programming skills, duplicated code from different migration stages, pulling in RSS content from partner sites to mix articles/posts from there into the article timelines in our own application's topics, and a lot more fun stuff...

First step is to migrate the backend application to get a consistent admin and publishing interface. The legacy frontend applications still need to write to the database (comments and other content created by visitors). So the process of fixing the database must be able to run unattended on a regular basis.

We already have fixes in place that gracefully handle broken model dependencies in belongs_to and has_many. Paperclip integration has been designed to work with all the fantastic filename mappings invented. And the airbrake gem reports all application crashes to our redmine installation so we get a quick overview of all the left quirks.

The legacy applications have already been modified to work with the latest MySQL version and has been migrated to a current MySQL database server.

Answer 1

I had the same problem. The solution was to tell mysql2 not to perform casting, like this:

client.query(sql, cast: false).each do |row|
  row['some_date'] = Date.parse(row['some_date']) rescue(nil)
end

See mysql2 documentation for details on how to build client object. If required, access rails db config via ActiveRecord::Base.configurations.

Answer 2

Create a data import rake task that does all the conversions and fixes you need (including the data parsing and fixing), and run it every time you get a fresh update from the legacy app. The task can use raw SQL (look-up "execute" and "exec_query" methods), it doesn't have to work with models. This will be your magical "gem" that you were looking for. Obviously, you cannot have a one-fits-all tool for that, as every case of broken data is unique. But just don't create kludges in your new code base.

Answer 3

Similar to: Rails: How to handle existing invalid dates in database? and also without correct answer so I repost my solution below.

I think the simplest solution that worked for me was to set in database.yml file write cast: false, e.g. for development section

development
  <<: *default
  adapter: mysql2    
  (... some other settings ...)
  cast: false

Answer 4

I believe it will solve your problem Date.parse()

e.g. Date.parse(foo.created_at)