How to batch resize millions of images to fit a max width and height?

Question

The situation

I\'m looking for a way to batch-resize approximately 15 million images of different file types to fit a certain bounding box resolution (in this case

Answer 1

The best way I found to convert millions of images like these is by creating a simple bash script which starts converting all the images it finds, like the one listed below:

To edit this bash script, I use nano if you don't have nano: "apt-get install nano" for Ubuntu/Debian or "yum install nano" for CentOS/CloudLinux.. for other distributions: use Google) but you're free to use any editor you want.

Bash script

First, create the bash script by starting your favorite editor (mine's nano):

nano -w ~/imgconv.sh

Then fill it with this content:

#!/bin/bash
find ./ -type f -iname "*.jpeg" -exec mogrify -verbose -format jpeg -layers Dispose -resize 1024\>x1024\> -quality 75% {} +
find ./ -type f -iname "*.jpg" -exec mogrify -verbose -format jpg -layers Dispose -resize 1024\>x1024\> -quality 75% {} +
find ./ -type f -iname "*.png" -exec mogrify -verbose -format png -alpha on -layers Dispose -resize 1024\>x1024\> {} +

Then all you need to do is make it executable with chmod +x ~/imgconv.sh and run it from the main images directory where you want to resize the images in all subdirectories:

cd /var/www/webshop.example.com/public_html/media/
~/imgconv.sh

That should start the conversion process.

Explanation

The way the script works is that it uses find to find the file with extension .jpeg of any capitalization and then runs a command:

find ./ -type f -iname "*.jpeg" -exec <COMMAND> {} +

.. and then execute the appropriate convert job using the "-exec {} +" parameter:

mogrify -verbose -format jpeg -layers Dispose -resize 1024\>x1024\> -quality 75% <### the filename goes here, in this case *.jpeg ###>

If you're working with files older than today and you want to prevent re-doing files you've already converted today, you could even tell the 'find' command only convert the files older than today by adding the option -mtime +1 like so:

#!/bin/bash
find ./ -type f -mtime +1 -iname "*.jpeg" -exec mogrify -verbose -format jpeg -layers Dispose -resize 1024\>x1024\> -quality 75% {} +
find ./ -type f -mtime +1 -iname "*.jpg" -exec mogrify -verbose -format jpg -layers Dispose -resize 1024\>x1024\> -quality 75% {} +
find ./ -type f -mtime +1 -iname "*.png" -exec mogrify -verbose -format png -alpha on -layers Dispose -resize 1024\>x1024\> {} +

Performance

A really simple way to use more cores to perform this process is to fork each job to the background by adding a & after each line. Another way would be to use GNU Parallel, especially with the -X parameter as it will use all your CPU cores and get the job done many times quicker.

But no matter what kind of parallelization technique you'll be using, be sure only to do that on your own system and not on a shared disk system where your production platform resides, since going for maximum performance will bog down your hardware or hypervisor performance.

This job is going to take a while, so be sure to set up a screen or a terminal without timeout/noop packets beforehand. On my system, it churned through about 5000 files per minute, so the entire job should take less than ~50-60 hours... sounds like a fine job to run over the weekend.

Just be sure to separate all file extensions from each other by writing separate commands; Piling all options on top of each other and having 'mogrify' using all options for all image formats won't work.

ImageMagick is a powerful tool in the right hands.