Parallel Iterating IP Addresses in Bash

Question

I\'m dealing with a large private /8 network and need to enumerate all webservers which are listening on port 443 and have a specific version stated in their HTTP HEADER res

Answer 1

Shorter:

mycurl() {
    curl --head https://${1}:443 | grep -iE "(Server\:\ Target)" > ${1}_info.txt;  
}
export -f mycurl
parallel -j0 --tag mycurl {1}.{2}.{3}.{4} ::: {10..10} ::: {0..255} ::: {0..255} ::: {0..255}

Slightly different using --tag instead of many _info.txt-files:

parallel -j0 --tag curl --head https://{1}.{2}.{3}.{4}:443 ::: {10..10} ::: {0..255} ::: {0..255} ::: {0..255} | grep -iE "(Server\:\ Target)" > info.txt

Fan out to run more than 500 in parallel:

parallel echo {1}.{2}.{3}.{4} ::: {10..10} ::: {0..255} ::: {0..255} ::: {0..255} | \
  parallel -j100 --pipe -N1000 --load 100% --delay 1 parallel -j250 --tag -I ,,,, curl --head https://,,,,:443 | grep -iE "(Server\:\ Target)" > info.txt

This will spawn up to 100*250 jobs, but will try to find the optimal number of jobs where there is no idle time for any of the CPUs. On my 8 core system that is 7500. Make sure you have RAM enough to run the potential max (25000 in this case).