Linux: create random directory/file hierarchy

Question

For testing a tool I need a directory with a whole bunch of different Office files in a deep nested structure. I already have the files in a directory, but now need to creat

Answer 1

You can use bash brace-expansion:

mkdir -p {a,b}/{e,f,g}/{h,i,j}

---

├───a
│   ├───e
│   │   ├───h
│   │   ├───i
│   │   └───j
│   ├───f
│   │   ├───h
│   │   ├───i
│   │   └───j
│   └───g
│       ├───h
│       ├───i
│       └───j
└───b
    ├───e
    │   ├───h
    │   ├───i
    │   └───j
    ├───f
    │   ├───h
    │   ├───i
    │   └───j
    └───g
        ├───h
        ├───i
        └───j

Answer 2

This is a script that generate a random dir structure :

#!/bin/bash

# Decimal ASCII codes (see man ascii)
ARR=( {48..57} {65..90} {97..122} )

# Array count
arrcount=${#ARR[@]}

# return a random string
get_rand_dir(){
    for ((i=1; i<$((RANDOM%30)); i++)) {
        printf \\$(printf '%03o' ${ARR[RANDOM%arrcount]});
    }
}

dir=/tmp/

# appending random characters to make a hierarchy
for ((i=0; i<$((RANDOM%100)); i++)) {
    dir+="$(get_rand_dir)/"
}

echo $dir
mkdir -p "$dir"

oldir=$(echo "$dir" | cut -d '/' -f1-3)

while [[ $dir ]]; do
    dir=${dir%/*}
    cd $dir
    for ((i=0; i<$((RANDOM%100)); i++)) {
        mkdir &>/dev/null -p $(get_rand_dir)
    }
done

tree "$oldir"

OUTPUT

/tmp/x
├── egeDVPW
├── iOkr
├── l
├── o1gye8uF
├── q
│   ├── 4Dlrfagv
│   ├── 4Yxmoqf
│   ├── 8LkyIrXA
│   ├── 8m9kse8s
│   ├── aV
│   ├── in
│   │   ├── 12zdLso68HWlPK
│   │   │   ├── C
│   │   │   ├── DOYt8wUW
│   │   │   ├── FXP
│   │   │   ├── hFLem8
│   │   │   ├── hhHIv
│   │   │   ├── iD87kxs54x04
│   │   │   ├── oFM
│   │   │   ├── OjFT

Now you can create an array of dirs :

shopt -s globstar # require bash4
dirs=( /tmp/x/** )
printf '%s\n' ${dirs[@]}

and populate dirs with files randomly. You have enough examples to do so. I've done the most hard work.

Answer 3

Thanks to all who posted here; it turns out, it wasn't really trivial to escape filenames with special characters, so I built my own script based on those here; here is how it behaves with special character filenames:

$ ~/rndtree.sh ./rndpath 0 3 1
Warning: will create random tree at: ./rndpath
Proceed (y/n)? y
Removing old outdir ./rndpath
mkdir -p ./rndpath/";"/{")?DxVBBJ{w2","L,|+","^VC)Vn.6!"}/"D+,IFJ( LN"
> > > > > > > > > > > 
./rndpath
└── [       4096]  ;
    ├── [       4096]  )?DxVBBJ{w2
    │   ├── [       4096]  D+,IFJ( LN
    │   │   └── [        929]  r2.bin
    │   ├── [       8557]  %3fsaG# Rl;ffXf.bin
    │   └── [      19945]  Dzk .bin
    ├── [       4096]  L,|+
    │   ├── [       4096]  D+,IFJ( LN
    │   │   ├── [       2325]  6Qg#pe5j'&ji49oqrO.bin
    │   │   ├── [      16345]  #?.bin
    │   │   └── [       2057]  Uz-0XtLVWz#}0lI.bin
    │   ├── [       2543]  bbtA-^s22vdTu.bin
    │   └── [      10848]  K46+kh7L9.bin
    ├── [       4096]  ^VC)Vn.6!
    │   ├── [       4096]  D+,IFJ( LN
    │   ├── [      10502]  8yY,MqZ ^5+_SA^.r4{.bin
    │   └── [      17628]  ipO"|69.bin
    └── [      12376]  a2Y% }G1.qDir.bin

7 directories, 11 files
total bytes: 136823 ./rndpath

and here with a safe subset of ASCII:

$ ~/rndtree.sh ./rndpath 1 3 1
Warning: will create random tree at: ./rndpath
Proceed (y/n)? y
Removing old outdir ./rndpath
mkdir -p ./rndpath/"48SLS"/{"nyG","jIC6vj"}/{"PSLd5tMn","V R"}
> > > > > > > 
./rndpath
├── [       4096]  48SLS
│   ├── [       4096]  jIC6vj
│   │   ├── [       4096]  PSLd5tMn
│   │   ├── [       4096]  V R
│   │   │   ├── [        922]  lg.bin
│   │   │   └── [       9600]  VVYG.bin
│   │   ├── [      10883]  B7nt06p.bin
│   │   └── [      19339]  g5uT i.bin
│   ├── [       4096]  nyG
│   │   ├── [       4096]  PSLd5tMn
│   │   └── [       4096]  V R
│   │       └── [       6128]  1tfLR.bin
│   └── [       5448]  Jda.bin
└── [      18196]  KwEXu2O2H9s.bin

Spaces should be handled in both cases - however, note that subdirectory names repeat (while filenames do not).

The script rndtree.sh:

#!/usr/bin/env bash

# http://stackoverflow.com/questions/13400312/linux-create-random-directory-file-hierarchy
# Decimal ASCII codes (see man ascii); added space
AARR=( 32 {48..57} {65..90} {97..122} )
# Array count
aarrcount=${#AARR[@]}

if [ "$1" == "" ] ; then
  OUTDIR="./rndpath" ;
else
  OUTDIR="$1" ;
fi

if [ "$2" != "" ] ; then
  ASCIIONLY="$2" ;
else
  ASCIIONLY=1 ;
fi

if [ "$3" != "" ] ; then
  DIRDEPTH="$3" ;
else
  DIRDEPTH=3 ;
fi

if [ "$4" != "" ] ; then
  MAXFIRSTLEVELDIRS="$4" ;
else
  MAXFIRSTLEVELDIRS=2 ;
fi

if [ "$5" != "" ] ; then
  MAXDIRCHILDREN="$5" ;
else
  MAXDIRCHILDREN=4 ;
fi

if [ "$6" != "" ] ; then
  MAXDIRNAMELEN="$6" ;
else
  MAXDIRNAMELEN=12 ;
fi

if [ "$7" != "" ] ; then
  MAXFILECHILDREN="$7" ;
else
  MAXFILECHILDREN=4 ;
fi

if [ "$8" != "" ] ; then
  MAXFILENAMELEN="$8" ;
else
  MAXFILENAMELEN=20 ;
fi

if [ "$9" != "" ] ; then
  MAXFILESIZE="$9" ;
else
  MAXFILESIZE=20000 ;
fi

MINDIRNAMELEN=1
MINFILENAMELEN=1
MINDIRCHILDREN=1
MINFILECHILDREN=0
MINFILESIZE=1
FILEEXT=".bin"
VERBOSE=0 #1

get_rand_dirname() {
  if [ "$ASCIIONLY" == "1" ]; then
    for ((i=0; i<$((MINDIRNAMELEN+RANDOM%MAXDIRNAMELEN)); i++)) {
      printf \\$(printf '%03o' ${AARR[RANDOM%aarrcount]});
    }
  else
    cat /dev/urandom | tr -dc '[ -~]' | tr -d '[$></~:`\\]' | head -c$((MINDIRNAMELEN + RANDOM % MAXDIRNAMELEN)) | sed 's/\(["]\)/\\\1/g'
  fi
  #echo -e " " # debug last dirname space
}

get_rand_filename() {
  if [ "$ASCIIONLY" == "1" ]; then
    for ((i=0; i<$((MINFILENAMELEN+RANDOM%MAXFILENAMELEN)); i++)) {
      printf \\$(printf '%03o' ${AARR[RANDOM%aarrcount]});
    }
  else
    # no need to escape double quotes for filename
    cat /dev/urandom | tr -dc '[ -~]' | tr -d '[$></~:`\\]' | head -c$((MINFILENAMELEN + RANDOM % MAXFILENAMELEN)) #| sed 's/\(["]\)/\\\1/g'
  fi
  printf "%s" $FILEEXT
}


echo "Warning: will create random tree at: $OUTDIR"
[ "$VERBOSE" == "1" ] && echo "  MAXFIRSTLEVELDIRS $MAXFIRSTLEVELDIRS ASCIIONLY $ASCIIONLY DIRDEPTH $DIRDEPTH MAXDIRCHILDREN $MAXDIRCHILDREN MAXDIRNAMELEN $MAXDIRNAMELEN MAXFILECHILDREN $MAXFILECHILDREN MAXFILENAMELEN $MAXFILENAMELEN MAXFILESIZE $MAXFILESIZE"

read -p "Proceed (y/n)? " READANS
if [ "$READANS" != "y" ]; then
  exit
fi

if [ -d "$OUTDIR" ]; then
  echo "Removing old outdir $OUTDIR"
  rm -rf "$OUTDIR"
fi

mkdir "$OUTDIR"

if [ $MAXFIRSTLEVELDIRS -gt 0 ]; then
  NUMFIRSTLEVELDIRS=$((1+RANDOM%MAXFIRSTLEVELDIRS))
else
  NUMFIRSTLEVELDIRS=0
fi



# create directories
for (( ifl=0;ifl<$((NUMFIRSTLEVELDIRS));ifl++ )) {
  FLDIR="$(get_rand_dirname)"
  FLCHILDREN="";
  for (( ird=0;ird<$((DIRDEPTH-1));ird++ )) {
    DIRCHILDREN=""; MOREDC=0;
    for ((idc=0; idc<$((MINDIRCHILDREN+RANDOM%MAXDIRCHILDREN)); idc++)) {
      CDIR="$(get_rand_dirname)" ;
      # make sure comma is last, so brace expansion works even for 1 element? that can mess with expansion math, though
      if [ "$DIRCHILDREN" == "" ]; then DIRCHILDREN="\"$CDIR\"" ;
      else DIRCHILDREN="$DIRCHILDREN,\"$CDIR\"" ; MOREDC=1 ; fi
    }
    if [ "$MOREDC" == "1" ] ; then
      if [ "$FLCHILDREN" == "" ]; then FLCHILDREN="{$DIRCHILDREN}" ;
      else FLCHILDREN="$FLCHILDREN/{$DIRCHILDREN}" ; fi
    else
      if [ "$FLCHILDREN" == "" ]; then FLCHILDREN="$DIRCHILDREN" ;
      else FLCHILDREN="$FLCHILDREN/$DIRCHILDREN" ; fi
    fi
  }
  DIRCMD="mkdir -p $OUTDIR/\"$FLDIR\"/$FLCHILDREN"
  eval "$DIRCMD"
  echo "$DIRCMD"
}

# now loop through all directories, create random files inside
# note printf '%q' escapes to preserve spaces; also here
# escape, and don't wrap path parts in double quotes (e.g. | sed 's_/_"/"_g');
# note then we STILL have to eval to use it!
# but now ls "$D" works, so noneed for QD
# unfortunately backslashes can make '%q' barf - prevent them
find "$OUTDIR" -type d | while IFS= read D ; do
  QD="$(printf '%q' "$(echo "$D")" )" ;
  [ "$VERBOSE" == "1" ] && echo "$D"; #echo "$QD"; ls -la "$D"; #eval "ls -la $QD";
  for ((ifc=0; ifc<$((MINFILECHILDREN+RANDOM%MAXFILECHILDREN)); ifc++)) {
    CFILE="$(get_rand_filename)" ;
    echo -n '> '
    [ "$VERBOSE" == "1" ] && echo "$D"/"$CFILE"
    cat /dev/urandom \
    | head -c$((MINFILESIZE + RANDOM % MAXFILESIZE)) \
    > "$D"/"$CFILE"
  }
done

echo
tree -a --dirsfirst -s "$OUTDIR"
echo "total bytes: $(du -bs $(echo "$OUTDIR"))"

Answer 4

I wasn't too happy with the given answers, so I came up with my own. The following takes my input files and uses /dev/urandom to gather 10 to 256 printable chars, puts in a few more directory separators, creates the directory hierarchy and places a file in it.

Using urandom creates some really weird directory names which is good for my purpose. I'm sure a real Unix guru could simplify this even more. The dir building could probably be done in a single awk command for example.

#!/bin/bash
INDIR='files';

IFS=$'\n'
for FILE in `ls $INDIR/*`; do
    DIR=`cat /dev/urandom | \
         tr -dc '[ -~]' | \
         tr 'ABCDEF\\\\' '///////' | \
         head -c$((10 + $RANDOM % 256))`

    mkdir -p $DIR
    cp $FILE $DIR
done