gnu-parallel | 易学教程

gnu parallel: output each job to a different file

阅读更多关于 gnu parallel: output each job to a different file

问题 I am trying to process so text files with awk using the parallel command as a shell script, but haven't been able to get it to output each job to a different file If i try: seq 10 | parallel awk \''{ if ( $5 > 0.4 ) print $2}'\' file{}.txt > file{}.out It outputs to the file file{}.out instead of file1.out , file2.out , etc. The tutorial and man pages also suggest that I could use --files , but it just prints to stdout : seq 10 | parallel awk \''{ if ( $5 > 0.4 ) print $2}'\' file{}.txt -

export function from zsh to bash for use in gnu parallel

阅读更多关于 export function from zsh to bash for use in gnu parallel

问题 How do I export a function from zsh, so that I can use it in gnu parallel? example: function my_func(){ echo $1;} export -f my_func parallel "my_func {}" ::: 1 2 in bash will output 1 2 whereas in zsh it will output error messages /bin/bash: my_func: command not found /bin/bash: my_func: command not found 回答1: zsh does not have a concept of exporting functions. export -f somefunc will print the function definition, it will not export a function. Instead, you can rely on the fact that bash

Executing bash convert commands asynchronously

阅读更多关于 Executing bash convert commands asynchronously

问题 I have a loop that cycles through a bunch of images that I need to edit with Imagemagick. Trouble is, it's slow to have to wait for each image to be edited before editing the next one. I want to execute each of the convert commands asynchronously not waiting for the last one to finish.Is this possible with bash? Here's a basic overview of what I have: for img in * do convert $img **more params here** done I want to execute that convert command asynchronously. So I can convert all the images

Perl command inside GNU parallel?

阅读更多关于 Perl command inside GNU parallel?

问题 I am trying to run this in parallel: parallel perl -pe '!/^step/ && s/(\S+)/sprintf("%.2e", $1)/ge' {} > {}.fix ::: * That is, I want to execute the perl command on all files in the current directory, in parallel. This is not working, but I have no idea why. Comment: The perl command is fixing floating-point numbers in tables. See Replacing precision of floating point numbers in existing file. 回答1: In Bash you can make a function: doit() { perl -pe '!/^step/ && s/(\S+)/sprintf("%.2e", $1)/ge'

gnu parallel to parallelize a for loop

阅读更多关于 gnu parallel to parallelize a for loop

问题 I have seen several questions about this topic, but I lack the ability to translate this to my specific problem. I have a for loop that loops through sub directories and then executes a .sh script on a compressed text file inside each directory. I want to parallelize this process, but I'm struggling to apply gnu parallel. Here is my loop: for d in ./*/ ; do (cd "$d" && script.sh); done I understand I need to input a list into parallel, so i have been trying this: ls -d */ | parallel cd &&

GNU parallel with rsync

阅读更多关于 GNU parallel with rsync

问题 I'm trying to run some instances of rsync in parallel using ssh with GNU parallel . The command I'm running is like this: find /tmp/tempfolder -type f -name 'chunck.*' | sort | parallel --gnu -j 4 -v ssh -i access.pem user@server echo {}\; rsync -Havessh -auz -0 --files-from={} ./ user@server:/destination/path /tmp/tempfolder contains files with the prefix chunck and they contain the actual file lists. With this command, I got the 4 calls for rsync alright, but they take a while to start

Executing bash convert commands asynchronously

阅读更多关于 Executing bash convert commands asynchronously

I have a loop that cycles through a bunch of images that I need to edit with Imagemagick. Trouble is, it's slow to have to wait for each image to be edited before editing the next one. I want to execute each of the convert commands asynchronously not waiting for the last one to finish.Is this possible with bash? Here's a basic overview of what I have: for img in * do convert $img **more params here** done I want to execute that convert command asynchronously. So I can convert all the images at once. Is this possible with bash? If you have many images, running hundreds/thousands of convert

split STDIN to multiple files (and compress them if possible)

阅读更多关于 split STDIN to multiple files (and compress them if possible)

问题 I have program (gawk) that outputs stream of data to its STDOUT. The data processed is literally 10s of GBs. I don't want to persist it in a single file but rather split it into chunks and potentially apply some extra processing (like compression) to each before saving. my data is a sequence of records and I don't want splitting to cut record in half. Each record matches the following regexp: ^\{index.+?\}\}\n\{.+?\}$ or for simplicity can assume that two rows (first uneven then even when

How to Install GNU Parallel on Windows 10 using git-bash

阅读更多关于 How to Install GNU Parallel on Windows 10 using git-bash

Has anyone been able to successfully use GNU Parallel on Windows 10 with git-bash? Is it possible? - If so, how? Background: I'm having trouble installing GNU Parallel and using it, and it got me thinking - maybe git-bash is holding me back? I'm sure if I installed Ubuntu through WSL I wouldn't have any problems running GNU Parallel. But I wanted to know if I could do this in git-bash first. I just installed git-bash on a Microsoft Windows 10 machine and had no problems installing GNU Parallel. It is by no means well tested on git-bash, but basic functionality clearly works. I'm having trouble

Calling GNU parallel from GNU parallel

阅读更多关于 Calling GNU parallel from GNU parallel

问题 What's the proper way to nest calls to GNU parallel ? Silly example: seq 1 100 | parallel echo {} | parallel seq {} 1000 My understanding is that on an 8-CPU box, each parallel would launch 8 jobs for a total of 64 jobs. If you're calling something more substantial than seq this could potentially overload the box. Is there a way to limit the number of jobs but still make full use of parallelism? 回答1: Use -j to limit either the outer or the inner parallel: seq 1 100 | parallel -j1 "echo {} |