For anyone searching for this. I did find Sidekiq now has a setting for this out of the box. All you
need to do is add Sidekiq.remove_delay! to config/initializers/sidekiq.rb
This is described here:
https://github.com/mperham/sidekiq/wiki/Delayed-Extensions