Shortest possible encoded string with decode possibility (shorten url) using only PHP

Question

I\'m looking for a method that encodes an string to shortest possible length and lets it be decodable (pure PHP, no SQL). I have working sc

Answer 1

There are many ways to shorten urls. You can look up how other services, like TinyUrl, shorten their urls. Here is a good article on hashes and shortening urls: http://blog.codinghorror.com/url-shortening-hashes-in-practice/

You can use the php function mhash() to apply hashes to strings: http://php.net/manual/en/function.mhash.php

And if you scroll down to "Available Hashes" on the mhash website, you can see what hashes you can use in the function (although I would check what php versions have which functions):http://mhash.sourceforge.net/mhash.3.html

Answer 2

Theory

In theory we need a short input character set and a large output character set. I will demonstrate it by the following example. We have the number 2468 as integer with 10 characters (0-9) as character set. We can convert it to the same number with base 2 (binary number system). Then we have a shorter character set (0 and 1) and the result is longer: 100110100100

But if we convert to hexadecimal number (base 16) with a character set of 16 (0-9 and A-F). Then we get a shorter result: 9A4

Practice

So in your case we have the following character set for the input:

$inputCharacterSet = "0123456789abcdefghijklmnopqrstuvwxyz=/-.&";

In total 41 characters: Numbers, lower cases and the special chars = / - . &

The character set for output is a bit tricky. We want use URL save characters only. I've grabbed them from here: Characters allowed in GET parameter

So our output character set is (73 characters):

$outputCharacterSet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz~-_.!*'(),$";

Numbers, lower AND upper cases and some special chars.

We have more characters in our set for the output than for the intput. Theory says we can short our input string. CHECK!

Coding

Now we need an encode function from base 41 to base 73. For that case I don't know a PHP function. Luckily we can grab the function 'convBase' from here: http://php.net/manual/de/function.base-convert.php#106546 (if someone knows a smarter function let me know)

<?php
function convBase($numberInput, $fromBaseInput, $toBaseInput)
{
    if ($fromBaseInput==$toBaseInput) return $numberInput;
    $fromBase = str_split($fromBaseInput,1);
    $toBase = str_split($toBaseInput,1);
    $number = str_split($numberInput,1);
    $fromLen=strlen($fromBaseInput);
    $toLen=strlen($toBaseInput);
    $numberLen=strlen($numberInput);
    $retval='';
    if ($toBaseInput == '0123456789')
    {
        $retval=0;
        for ($i = 1;$i <= $numberLen; $i++)
            $retval = bcadd($retval, bcmul(array_search($number[$i-1], $fromBase),bcpow($fromLen,$numberLen-$i)));
        return $retval;
    }
    if ($fromBaseInput != '0123456789')
        $base10=convBase($numberInput, $fromBaseInput, '0123456789');
    else
        $base10 = $numberInput;
    if ($base10<strlen($toBaseInput))
        return $toBase[$base10];
    while($base10 != '0')
    {
        $retval = $toBase[bcmod($base10,$toLen)].$retval;
        $base10 = bcdiv($base10,$toLen,0);
    }
    return $retval;
}

Now we can short the url. The final code is:

$input = 'img=/dir/dir/hi-res-img.jpg&w=700&h=500';
$inputCharacterSet = "0123456789abcdefghijklmnopqrstuvwxyz=/-.&";
$outputCharacterSet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz~-_.!*'(),$";
$encoded = convBase($input, $inputCharacterSet, $outputCharacterSet);
var_dump($encoded); // string(34) "BhnuhSTc7LGZv.h((Y.tG_IXIh8AR.$!t*"
$decoded = convBase($encoded, $outputCharacterSet, $inputCharacterSet);
var_dump($decoded); // string(39) "img=/dir/dir/hi-res-img.jpg&w=700&h=500"

The encoded string has only 34 characters.

Optimizations

You can optimize the count of characters by

• reduce the length of input string. Do you really need the overhead of url parameter syntax? Maybe you can format your string as follows:

  $input = '/dir/dir/hi-res-img.jpg,700,500';

  This reduces the input itself AND the input character set. Your reduced input character set is then:

  $inputCharacterSet = "0123456789abcdefghijklmnopqrstuvwxyz/-.,";

  Final output:

  string(27) "E$AO.Y_JVIWMQ9BB_Xb3!Th*-Ut"

  string(31) "/dir/dir/hi-res-img.jpg,700,500"

• reducing the input character set ;-). Maybe you can exclude some more characters? You can encode the numbers to characters first. Then your input character set can be reduced by 10!

• increase your output character set. So the given set by me is googled within 2 minutes. Maybe you can use more url save characters. No idea... Maybe someone has a list

Security

Heads up: There is no cryptographically logic in the code. So if somebody guesses the character sets, he can decode the string easily. But you can shuffle the character sets (once). Then it is a bit harder for the attacker, but not really safe. Maybe its enough for your use case anyway.

Answer 3

Short words about "security"

You simply won't be able to secure your link if there is no "secret password" stored somewhere: as long as the URI carries all information to access your resource, then it will be decodable and your "custom security" (they are opposite words btw) will be broken easily.

You can still put a salt in your PHP code (like $mysalt="....long random string...") since I doubt you want an eternal security (such approach is weak because you cannot renew the $mysalt value, but in your case, few years security sounds sufficient, since anyway, a user can buy one picture and share it elsewhere, breaking any of your security mechanism).

If you want to have a safe mechanism, use a well-known one (as a framework would carry), along with authentication and user rights management mechanism (so you can know who's looking for your image, and whether they are allowed to).

Security has a cost, if you don't want to afford its computing & storing requirements, then forget about it.

---

Secure by signing the URL

If you want to avoid users easy by-passing and get full res picture, then you may just sign the URI (but really, for safety, use something that already exist instead of that quick draft example below):

$salt = '....long random stirng...';
$params = array('img' => '...', 'h' => '...', 'w' => '...');
$p = http_build_query($params);
$check = password_hash($p, PASSWORD_BCRYPT, array('salt' => $salt, 'cost' => 1000);
$uri = http_build_query(array_merge($params, 'sig' => $check));

Decoding:

$sig = $_GET['sig'];
$params = $_GET;
unset($params['sig']);

// Same as previous
$salt = '....long random stirng...';
$p = http_build_query($params);
$check = password_hash($p, PASSWORD_BCRYPT, array('salt' => $salt, 'cost' => 1000);
if ($sig !== $check) throw new DomainException('Invalid signature');

See http://php.net/manual/fr/function.password-hash.php

---

Shorten smartly

"Shortening" with a generic compression algorithm is useless here because the headers will be longer than the URI, so it will almost never shorten it.

If you want to shorten it, be smart: don't give the relative path (/dir/dir) if it's always the same (or give it only if it's not the main one). Don't give the extension if it's always the same (or give it when it's not png if almost everything is in png). Don't give the height because the image carries the aspect ratio: you only need the width. Give it in x100px if you do not need a pixel-accurate width.

Answer 4

In your question you state that it should be pure PHP and not use a database, and there should be a possibility to decode the strings. So bending the rules a bit:

• The way I am interpreting this question is that we don't care about security that much but, we do want the shortest hashes that lead back to images.
• We can also take "decode possibility" with a pinch of salt by using a one way hashing algorithm.
• We can store the hashes inside a JSON object, then store the data in a file, so all we have to do at the end of the day is string matching

```

class FooBarHashing {

    private $hashes;

    private $handle;

    /**
     * In producton this should be outside the web root
     * to stop pesky users downloading it and geting hold of all the keys.
     */
    private $file_name = './my-image-hashes.json';

    public function __construct() {
        $this->hashes = $this->get_hashes();
    }

    public function get_hashes() {
        // Open or create a file.
        if (! file_exists($this->file_name)) {
            fopen($this->file_name, "w");
        }
        $this->handle = fopen($this->file_name, "r");


        $hashes = [];
        if (filesize($this->file_name) > 0) {
            $contents = fread($this->handle, filesize($this->file_name));
            $hashes = get_object_vars(json_decode($contents));
        }

        return $hashes;
    }

    public function __destroy() {
        // Close the file handle
        fclose($this->handle);
    }

    private function update() {
        $handle = fopen($this->file_name, 'w');
        $res = fwrite($handle, json_encode($this->hashes));
        if (false === $res) {
            //throw new Exception('Could not write to file');
        }

        return true;
    }

    public function add_hash($image_file_name) {
        $new_hash = md5($image_file_name, false);

        if (! in_array($new_hash, array_keys($this->hashes) ) ) {
            $this->hashes[$new_hash] =  $image_file_name;
            return $this->update();
        }

        //throw new Exception('File already exists');
    }

    public function resolve_hash($hash_string='') {
        if (in_array($hash_string, array_keys($this->hashes))) {
            return $this->hashes[$hash_string];
        }

        //throw new Exception('File not found');
    }
}

```

Usage example:

<?php
// Include our class
require_once('FooBarHashing.php');
$hashing = new FooBarHashing;

// You will need to add the query string you want to resolve first.
$hashing->add_hash('img=/dir/dir/hi-res-img.jpg&w=700&h=500');

// Then when the user requests the hash the query string is returned.
echo $hashing->resolve_hash('65992be720ea3b4d93cf998460737ac6');

So the end result is a string that is only 32 chars long, which is way shorter than the 52 we had before.

Answer 5

I suspect that you will need to think more about your method of hashing if you don't want it to be decodable by the user. The issue with base64 is that a base64 string looks like a base64 string. There's a good chance that someone that's savvy enough to be looking at your page source will probably recognise it too.

Part one:

a method that encodes an string to shortest possible length

If you're flexible on your URL vocab/characters, this will be a good starting place. Since gzip makes a lot of its gains using back references, there is little point as the string is so short.

Consider your example - you've only saved 2 bytes in the compression, which are lost again in base64 padding:

Non-gzipped: string(52) "aW1nPS9kaXIvZGlyL2hpLXJlcy1pbWcuanBnJnc9NzAwJmg9NTAw"

Gzipped: string(52) "y8xNt9VPySwC44xM3aLUYt3M3HS9rIJ0tXJbcwMDtQxbUwMDAA=="

If you reduce your vocab size, this will naturally allow you better compression. Let's say we remove some redundant information

Take a look at the functions:

function compress($input, $ascii_offset = 38){
    $input = strtoupper($input);
    $output = '';
    //We can try for a 4:3 (8:6) compression (roughly), 24 bits for 4 chars
    foreach(str_split($input, 4) as $chunk) {
        $chunk = str_pad($chunk, 4, '=');

        $int_24 = 0;
        for($i=0; $i<4; $i++){
            //Shift the output to the left 6 bits
            $int_24 <<= 6;

            //Add the next 6 bits
            //Discard the leading ascii chars, i.e make
            $int_24 |= (ord($chunk[$i]) - $ascii_offset) & 0b111111;
        }

        //Here we take the 4 sets of 6 apart in 3 sets of 8
        for($i=0; $i<3; $i++) {
            $output = pack('C', $int_24) . $output;
            $int_24 >>= 8;
        }
    }

    return $output;
}

And

function decompress($input, $ascii_offset = 38) {

    $output = '';
    foreach(str_split($input, 3) as $chunk) {

        //Reassemble the 24 bit ints from 3 bytes
        $int_24 = 0;
        foreach(unpack('C*', $chunk) as $char) {
            $int_24 <<= 8;
            $int_24 |= $char & 0b11111111;
        }

        //Expand the 24 bits to 4 sets of 6, and take their character values
        for($i = 0; $i < 4; $i++) {
            $output = chr($ascii_offset + ($int_24 & 0b111111)) . $output;
            $int_24 >>= 6;
        }
    }

    //Make lowercase again and trim off the padding.
    return strtolower(rtrim($output, '='));
}

What's going on there is basically a removal of redundant information, followed by the compression of 4 bytes into 3. This is achieved by effectively having a 6-bit subset of the ascii table. This window is moved so that the offset starts at useful characters and includes all the characters you're currently using.

With the offset I've used, you can use anything from ASCII 38 to 102. This gives you a resulting string of 30 bytes, that's a 9-byte (24%) compression! Unfortunately, you'll need to make it URL-safe (probably with base64), which brings it back up to 40 bytes.

I think at this point, you're pretty safe to assume that you've reached the "security through obscurity" level required to stop 99.9% of people. Let's continue though, to the second part of your question

so the user can't guess how to get the larger image

It's arguable that this is already solved with the above, but what you need to do is pass this through a secret on the server, preferably with php openssl. The following code shows the complete usage flow of functions above and the encryption:

$method = 'AES-256-CBC';
$secret = base64_decode('tvFD4Vl6Pu2CmqdKYOhIkEQ8ZO4XA4D8CLowBpLSCvA=');
$iv = base64_decode('AVoIW0Zs2YY2zFm5fazLfg==');

$input = 'img=/dir/dir/hi-res-img.jpg&w=700&h=500';
var_dump($input);

$compressed = compress($input);
var_dump($compressed);

$encrypted = openssl_encrypt($compressed, $method, $secret, false, $iv);
var_dump($encrypted);

$decrypted = openssl_decrypt($encrypted, $method, $secret, false, $iv);
var_dump($decrypted);

$decompressed = decompress($compressed);
var_dump($decompressed);

The output of this script is the following:

string(39) "img=/dir/dir/hi-res-img.jpg&w=700&h=500"
string(30) "<��(��tJ��@�xH��G&(�%��%��xW"
string(44) "xozYGselci9i70cTdmpvWkrYvGN9AmA7djc5eOcFoAM="
string(30) "<��(��tJ��@�xH��G&(�%��%��xW"
string(39) "img=/dir/dir/hi-res-img.jpg&w=700&h=500"

You'll see the whole cycle: compression > encryption > base64 encode/decode > decryption > decompression. The output of this would be as close as possible as you could really get, at near the shortest length you could get.

Everything aside, I feel obliged to conclude this with the fact that it is theoretical only, and this was a nice challenge to think about. There are definitely better ways to achieve your desired result - I'll be the first to admit that my solution is a little bit absurd!

Answer 6

I'm afraid, you won't be able to shorten the query string better than any known compression algorithm. As already mentioned, a compressed version will be shorter by a few (around 4-6) characters than the original. Moreover, the original string can be decoded relatively easy (opposed to decoding sha1 or md5, for instance).

I suggest shortening URLs by means of Web server configuration. You might shorten it further by replacing image path with an ID (store ID-filename pairs in a database).

For example, the following Nginx configuration accepts URLs like /t/123456/700/500/4fc286f1a6a9ac4862bdd39a94a80858, where

• the first number (123456) is supposed to be an image ID from database;
• 700 and 500 are image dimentions;
• the last part is an MD5 hash protecting from requests with different dimentions.

# Adjust maximum image size
# image_filter_buffer 5M;

server {
  listen          127.0.0.13:80;
  server_name     img-thumb.local;

  access_log /var/www/img-thumb/logs/access.log;
  error_log /var/www/img-thumb/logs/error.log info;

  set $root "/var/www/img-thumb/public";

  # /t/image_id/width/height/md5
  location ~* "(*UTF8)^/t/(\d+)/(\d+)/(\d+)/([a-zA-Z0-9]{32})$" {
    include        fastcgi_params;
    fastcgi_pass   unix:/tmp/php-fpm-img-thumb.sock;
    fastcgi_param  QUERY_STRING image_id=$1&w=$2&h=$3&hash=$4;
    fastcgi_param  SCRIPT_FILENAME /var/www/img-thumb/public/t/resize.php;

    image_filter resize $2 $3;
    error_page 415 = /empty;

    break;
  }

  location = /empty {
    empty_gif;
  }

  location / { return 404; }
}

The server accepts only URLs of specified pattern, forwards request to /public/t/resize.php script with modified query string, then resizes the image generated by PHP with image_filter module. In case of error, returns an empty GIF image.

The image_filter is optional, it is included only as an example. Resizing can be performed fully on PHP side. With Nginx, it is possible to get rid of PHP part, by the way.

The PHP script is supposed to validate the hash as follows:

// Store this in some configuration file.
$salt = '^sYsdfc_sd&9wa.';

$w = $_GET['w'];
$h = $_GET['h'];

$true_hash = md5($w . $h . $salt . $image_id);
if ($true_hash != $_GET['hash']) {
  die('invalid hash');
}

$filename = fetch_image_from_database((int)$_GET['image_id']);
$img = imagecreatefrompng($filename);
header('Content-Type: image/png');
imagepng($img);
imagedestroy($img);