How to generate a predictable shuffling of a sequence without generating the whole sequence in advance?

Question

The following python code describes exactly what I want to achieve for a sequence of arbitrary size (population):

import random
fixed_seed = 1 #generate the          


        

Answer 1

First off, note that this isn't a random sequence. It only generates a single, fixed, repeating sequence, and the seed chooses where in the sequence you start. That's the same as all PRNGs, of course, but normally the cycle of a PRNG is much larger than 16-bit or 32-bit. The way you've described using it, the cycle length is equal to the number of items you're iterating over, so all it'll do is take a single "shuffled" order and change where you start. The "seed" value is more like a starting index than a seed.

It's not the most satisfactory answer, but it's probably practical: you can pad the length to the next power of two, and skip any indexes above the actual maximum. Thus, if you have 5000 items, generate a sequence over 8192 items, and discard any results between [5000,8191]. The overhead from this sounds ugly, but in perspective it's not that bad: since this can at most double the length of the list, on average you'll have to discard one out of two results, so the worst-case average overhead is doubling the amount of work.

The following code demonstrates this (as well as showing a cleaner way to implement it). The third parameter to MaxLengthLFSR, if given, is the actual maximum value. You'd probably want to fill in TAP_MASKS for a larger number of sizes and then choose the smallest register size that fits the requested sequence length; here we just use the one requested, which works but will cause much more overhead if the length of the sequence is much larger than it needs to be.

TAP_MASKS = { # only one needed, but I included 3 to make the code more useful
    10: 0x00000240, # taps at 10, 7
    16: 0x0000B400, # taps at 16, 14, 13, 11
    32: 0xE0000200, # taps at 32, 31, 30, 10
}

def MaxLengthLFSR(next_val, reglen, max_value=None):
    """Iterate values from seed in max-length LFSR using Galois configuration."""
    # Ensure that max_value isn't 0, or we'll infinitely loop without yielding any values.
    if max_value is not None:
        assert max_value > 0

    while True:
        if max_value is None or next_val < max_value:
            yield next_val

        lsb = next_val & 1
        next_val = next_val >> 1
        if lsb == 1:
            mask = TAP_MASKS[reglen]
            next_val ^= mask

sample_count = 5 # demonstration number
num_retries = 3  # just enough to show the repeatable behaviour
for trynum in xrange(num_retries):
    it = MaxLengthLFSR(1, 16, 2000)
    seq = []
    for x in xrange(sample_count):
        seq.append(next(it))
    seq = [str(x) for x in seq]
    print "try %s: %s..." % (trynum + 1, ", ".join(seq))

Answer 2

I've actually written about this before: Secure Permutations with Block Ciphers. In a nutshell:

1. Yes, you can use an LFSR to generate permutations with a length that's a power of 2. You can also use any block cipher. With a block cipher, you can also find the element at index n, or the index for element n.
2. To generate a permutation with arbitrary length l, create one with the smallest power of 2 length greater than l. When you want to find the nth permutation element, apply the permutation function, and if it generates a number outside the desired range, apply it again; repeat until the number is in the acceptable range.

The number of iterations required for step 2 will average no more than 2; the worst case is high, but extremely unlikely to occur.