Uploading files to s3 using s3cmd in parallel

Question

I\'ve got a whole heap of files on a server, and I want to upload these onto S3. The files are stored with a .data extension, but really they\'re just a bunch of jpegs,pngs,zips

Answer 1

You are clearly skilled in writing shell, and extremely close to a solution:

s3upload_single() {
    n=$1
    data=".data" 
    extension=`file $n | cut -d ' ' -f2 | awk '{print tolower($0)}'` 
    mimetype=`file --mime-type $n | cut -d ' ' -f2`
    fullpath=`readlink -f $n`

    changed="${fullpath/.data/.$extension}"

    filePathWithExtensionChanged=${changed#*internal_data}

    s3upload="s3cmd put -m $mimetype --acl-public $fullpath s3://tff-xenforo-data"$filePathWithExtensionChanged     

    response=`$s3upload`
    echo $response 
}
export -f s3upload_single
find -name "*.data" | parallel s3upload_single