In Bash, how to find the lowest-numbered unused file descriptor?

Question

In a Bash-script, is it possible to open a file on \"the lowest-numbered file descriptor not yet in use\"?

I have looked around for how to do this, but it seems that Bas

Answer 1

If it is on Linux, you can always read the /proc/self/fd/ directory to find out the used file descriptors.

Answer 2

I know this thread is old, but believe that the best answer is missing, and would be useful to others like me who come here searching for a solution.

Bash and Zsh have built in ways to find unused file descriptors, without having to write scripts. (I found no such thing for dash, so the above answers may still be useful.)

Note: this finds the lowest unused file descriptor > 10, not the lowest overall.

$ man bash /^REDIRECTION (paragraph 2)
$ man zshmisc /^OPENING FILE DESCRIPTORS

Example works with bsh and zsh.

Open an unused file descriptor, and assign the number to $FD:

$ exec {FD}>test.txt
$ echo line 1 >&$FD
$ echo line 2 >&$FD
$ cat test.txt
line 1
line 2
$ echo $FD
10  # this number will vary

Close the file descriptor when done:

$ exec {FD}>&-

The following shows that the file descriptor is now closed:

$ echo line 3 >&$FD
bash: $FD: Bad file descriptor
zsh: 10: bad file descriptor

Answer 3

I revised my original answer and now have a one line solution for the original post.
The following function could live in a global file or sourced script (e.g. ~/.bashrc):

# Some error code mappings from errno.h
readonly EINVAL=22   # Invalid argument
readonly EMFILE=24   # Too many open files

# Finds the lowest available file descriptor, opens the specified file with the descriptor
# and sets the specified variable's value to the file descriptor.  If no file descriptors
# are available the variable will receive the value -1 and the function will return EMFILE.
#
# Arguments:
#   The file to open (must exist for read operations)
#   The mode to use for opening the file (i.e. 'read', 'overwrite', 'append', 'rw'; default: 'read')
#   The global variable to set with the file descriptor (must be a valid variable name)
function openNextFd {
    if [ $# -lt 1 ]; then
        echo "${FUNCNAME[0]} requires a path to the file you wish to open" >&2
        return $EINVAL
    fi

    local file="$1"
    local mode="$2"
    local var="$3"

    # Validate the file path and accessibility
    if [[ "${mode:='read'}" == 'read' ]]; then
        if ! [ -r "$file" ]; then
            echo "\"$file\" does not exist; cannot open it for read access" >&2
            return $EINVAL
        fi
    elif [[ !(-w "$file") && ((-e "$file") || !(-d $(dirname "$file"))) ]]; then
        echo "Either \"$file\" is not writable (and exists) or the path is invalid" >&2
        return $EINVAL
    fi

    # Translate mode into its redirector (this layer of indirection prevents executing arbitrary code in the eval below)
    case "$mode" in
        'read')
            mode='<'
            ;;
        'overwrite')
            mode='>'
            ;;
        'append')
            mode='>>'
            ;;
        'rw')
            mode='<>'
            ;;
        *)
            echo "${FUNCNAME[0]} does not support the specified file access mode \"$mode\"" >&2
            return $EINVAL
            ;;
    esac

    # Validate the variable name
    if ! [[ "$var" =~ [a-zA-Z_][a-zA-Z0-9_]* ]]; then
        echo "Invalid variable name \"$var\" passed to ${FUNCNAME[0]}" >&2
        return $EINVAL
    fi

    # we'll start with 3 since 0..2 are mapped to standard in, out, and error respectively
    local fd=3
    # we'll get the upperbound from bash's ulimit
    local fd_MAX=$(ulimit -n)
    while [[ $fd -le $fd_MAX && -e /proc/$$/fd/$fd ]]; do
        ((++fd))
    done

    if [ $fd -gt $fd_MAX ]; then
        echo "Could not find available file descriptor" >&2
        $fd=-1
        success=$EMFILE
    else
        eval "exec ${fd}${mode} \"$file\""
        local success=$?
        if ! [ $success ]; then
            echo "Could not open \"$file\" in \"$mode\" mode; error: $success" >&2
            fd=-1
        fi
    fi

    eval "$var=$fd"
    return $success;
}

One would use the foregoing function as follows to open files for input and output:

openNextFd "path/to/some/file" "read" "inputfile"
# opens 'path/to/some/file' for read access and stores
# the descriptor in 'inputfile'

openNextFd "path/to/other/file" "overwrite" "log"
# truncates 'path/to/other/file', opens it in write mode, and
# stores the descriptor in 'log'

And one would then use the preceding descriptors as usual for reading and writing data:

read -u $inputFile data
echo "input file contains data \"$data\"" >&$log

Answer 4

In Basile Starynkevitch's answer to this question, on Nov 29 2011, he writes:

If it is on Linux, you can always read the /proc/self/fd/ directory to find out the used file descriptors.

Having done several experiments based on reading the fd directory, I have arrived at the following code, as the "closest match" to what I was looking for. What I was looking for was actually a bash one-liner, like

my_file_descriptor=$(open_r /path/to/a/file)

which would find the lowest, unused file descriptor AND open the file on it AND assign it to the variable. As seen in the code below, by introducing the function "lowest_unused_fd", I at least get a "two-liner" (FD=$(lowest_unused_fd) followed by eval "exec $FD<$FILENAME") for the task. I have NOT been able to write a function that works like (the imaginary) "open_r" above. If someone knows how to do that, please step forward! Instead, I had to split the task into two steps: one step to find the unused file descriptor and one step to open the file on it. Also note that, to be able to place the find step in a function ("lowest_unused_fd") and have its stdout assigned to FD, I had to use "/proc/$$/fd" instead of "/proc/self/fd" (as in Basile Starynkevitch's suggestion), since bash spawns a subshell for the execution of the function.

#!/bin/bash

lowest_unused_fd () {
    local FD=0
    while [ -e /proc/$$/fd/$FD ]; do
        FD=$((FD+1))
    done
    echo $FD
}

FILENAME="/path/to/file"

#  Find the lowest, unused file descriptor
#+ and assign it to FD.
FD=$(lowest_unused_fd)

# Open the file on file descriptor FD.
if ! eval "exec $FD<$FILENAME"; then
    exit 1
fi

# Read all lines from FD.
while read -u $FD a_line; do
    echo "Read \"$a_line\"."
done

# Close FD.
eval "exec $FD<&-"

Answer 5

I needed to support both bash v3 on Mac and bash v4 on Linux and the other solutions require either bash v4 or Linux, so I came up with a solution that works for both, using /dev/fd.

find_unused_fd() {
  local max_fd=$(ulimit -n)
  local used_fds=" $(/bin/ls -1 /dev/fd | sed 's/.*\///' | tr '\012\015' '  ') "
  local i=0
  while [[ $i -lt $max_fd ]]; do
    if [[ ! $used_fds =~ " $i " ]]; then
      echo "$i"
      break
    fi
    (( i = i + 1 ))
  done
}

For example to dup stdout, you can do:

newfd=$(find_unused_fd)
eval "exec $newfd>&1"

Answer 6

Apple Mac OS X is not Linux. I don't see any '/proc' file system on OS X.

I guess one answer is to use "zsh", but I want to have a script that works on both OS X (aka BSD) and Linux in "bash". So, here I am, in the year 2020, with the latest version of OS X, which at this moment is Catalina, and I realize that Apple seems to have abandoned maintenance of Bash long ago; apparently in favor of Zsh.

Here is my multi-OS solution to find the lowest unused file descriptor on Apple Mac OS X or Linux. I created an entire Perl script, and in-lined it into the Shell script. There must be a better way, but for now, this works for me.

lowest_unused_fd() {
  # For "bash" version 4.1 and higher, and for "zsh", this entire function  
  # is replaced by the more modern operator "{fd}", used like this:
  #    exec {FD}>myFile.txt; echo "hello" >&$FD;
  if [ $(uname) = 'Darwin' ] ; then
    lsof -p $$ -a -d 0-32 | perl -an \
      -e 'BEGIN { our @currentlyUsedFds; };' \
      -e '(my $digits = $F[3]) =~ s/\D//g;' \
      -e 'next if $digits eq "";' \
      -e '$currentlyUsedFds[$digits] = $digits;' \
      -e 'END { my $ix; 
            for( $ix=3; $ix <= $#currentlyUsedFds; $ix++) {  
              my $slotContents = $currentlyUsedFds[$ix];
              if( !defined($slotContents) ) { 
                last; 
              } 
            } 
            print $ix;
          }' ;
  else 
    local FD=3
    while [ -e /proc/$$/fd/$FD ]; do
      FD=$((FD+1))
    done
    echo $FD
  fi;
}

The -an options to Perl tells it to (-n) run an implied while() loop that reads the file line by line and (-a) auto-split it into an array of words which, by convention, is named @F. The BEGIN says what to do before that while() loop, and the END says what to do after. The while() loop picks out field [3] of each line, reduces it to just its leading digits, which is a port number, and saves that in an array of port numbers that are currently in use, and therefore are unavailable. The END block then finds the lowest integer whose slot is not occupied.

Update: After doing all that, I actually am not using this in my own code. I realized that the answer from KingPong and Bruno Bronsky is far more elegant. However, I will leave this answer in place; it might be interesting to somebody.