View previous topic :: View next topic |
Author |
Message |
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Mon Feb 09, 2009 8:11 pm Post subject: Simplest way to threading support for bash scripts? |
|
|
Basically what I'm looking is something like this: Code: | thread 4 "command1 opts" "command2 opts" ... | That would run all the commands specified in the command line simulatenously, but not more than four at the same time.
Or is there any simple way to implement this feature to for -loop?
Mainly I need this when I convert many files from a format to another.
Thanks for advance! :) _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Sadako Advocate
Joined: 05 Aug 2004 Posts: 3792 Location: sleeping in the bathtub
|
Posted: Mon Feb 09, 2009 10:08 pm Post subject: |
|
|
I don't know of a simple way to do this, but you can background a process (or a group of processes, in your case just put all the commands you'd like run in each "thread" in a shell function and background the function), do other stuff, and then wait for the backgrounded process/function to complete before spawning a new "thread".
edit: thinking more about waht you specifically asked for, a shell script would be workable, but how would you deliminate between a commands' arguments and subsequent commands to be run in parallel?
enclosing each "thread" with double quotes seems kinda unwieldy... _________________ "You have to invite me in"
Last edited by Sadako on Mon Feb 09, 2009 10:16 pm; edited 1 time in total |
|
Back to top |
|
|
easy target Tux's lil' helper
Joined: 09 Jan 2008 Posts: 134
|
Posted: Mon Feb 09, 2009 10:15 pm Post subject: |
|
|
I don't find an easy way...
You may create two levels of loops:
1. Main -- 4 iterations -- call a command with & to make it run in parallel
2. Nested -- actually doing the work
Drawback is that you have to split input data somehow...
EDIT: Or use make(1), write a recipe and just run make -j4 |
|
Back to top |
|
|
Sadako Advocate
Joined: 05 Aug 2004 Posts: 3792 Location: sleeping in the bathtub
|
Posted: Mon Feb 09, 2009 10:46 pm Post subject: |
|
|
Messy, but this seems to work, however you can only have one process per thread, "&&" and ";" will be passed as arguments to the commands if they're within the double quotes. Code: | #!/bin/bash
THREADS="4"
while [ $# != 0 ]; do
sleep 0.5
[ "$(jobs | wc -l)" -lt "${THREADS}" ] && $1 & shift
done
wait
exit | It's easier to just set the nember of threads in the script, but if the THREADS env variable is already defined it'll use that value instead (ie start it with "THREADS=6 ./foo.sh "command1 args" "command2 args" etc).
I'm sure someone else can come up with a better version.
edit: cleaned up some; Code: | #!/bin/bash
THREADS="4"
while [ $# -gt 0 ]; do
JOBS=($(jobs -p))
if [ "${#JOBS[@]}" -lt "${THREADS}" ]; then
$1 & shift
else
sleep 1
fi
done
wait
exit |
_________________ "You have to invite me in" |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 10, 2009 4:51 am Post subject: |
|
|
Hopeless wrote: | enclosing each "thread" with double quotes seems kinda unwieldy... | Tha't very true.
Best I could imagine was a new loop family in bash ie. forthreads <number of max threads> ARG in $VAR; do blah; done _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
UberLord Retired Dev
Joined: 18 Sep 2003 Posts: 6835 Location: Blighty
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 10, 2009 4:29 pm Post subject: |
|
|
Thanks. I think I can work something out from the code you all posted. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Sadako Advocate
Joined: 05 Aug 2004 Posts: 3792 Location: sleeping in the bathtub
|
Posted: Tue Feb 10, 2009 4:43 pm Post subject: |
|
|
UberLord wrote: | No need to use bashisms here
Code: | #!/bin/sh
THREADS="4"
while [ $# -gt 0 ]; do
JOBS=$(jobs -p | wc -l)
if [ "${JOBS}" -lt "${THREADS}" ]; then
$1 & shift
else
sleep 1
fi
done
wait
exit |
| Yeah, I know, I usually try making shell scripts posix-compliant if possible, but I just thought the "wc -l" was a little ugly, especially when you can do the same with bash internals... _________________ "You have to invite me in" |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 10, 2009 5:28 pm Post subject: |
|
|
Hopeless wrote: | Yeah, I know, I usually try making shell scripts posix-compliant if possible, but I just thought the "wc -l" was a little ugly, especially when you can do the same with bash internals... | Same here. I try to use bash internals as much as possible if it does not affect performance too much. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 10, 2009 7:16 pm Post subject: |
|
|
Now The problem is that if user interrupts the process it won't kill the running threads.
So when main loop gets term or kill signal it must also kill every thread.
I guess it can be archived by getting each thread's pid, then via kill $PIDS to stop them all. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Sadako Advocate
Joined: 05 Aug 2004 Posts: 3792 Location: sleeping in the bathtub
|
Posted: Tue Feb 10, 2009 7:22 pm Post subject: |
|
|
Zucca wrote: | Now The problem is that if user interrupts the process it won't kill the running threads.
So when main loop gets term or kill signal it must also kill every thread.
I guess it can be archived by getting each thread's pid, then via kill $PIDS to stop them all. | Read up on the "trap" command? _________________ "You have to invite me in" |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 10, 2009 8:20 pm Post subject: |
|
|
Hopeless wrote: | Zucca wrote: | Now The problem is that if user interrupts the process it won't kill the running threads.
So when main loop gets term or kill signal it must also kill every thread.
I guess it can be archived by getting each thread's pid, then via kill $PIDS to stop them all. | Read up on the "trap" command? | I read the manpage and modified the code a bit: Code: | #!/bin/bash
THREADS="4"
PIDS=""
while [ $# -gt 0 ]; do
JOBS=($(jobs -p))
if [ "${#JOBS[@]}" -lt "${THREADS}" ]; then
$1 &
PIDS="${PIDS}$! "
trap 'kill $PIDS; exit' SIGINT SIGTERM SIGKILL
shift
else
sleep 1
fi
done
wait
exit #why this exit here? |
_________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
UberLord Retired Dev
Joined: 18 Sep 2003 Posts: 6835 Location: Blighty
|
Posted: Tue Feb 10, 2009 10:30 pm Post subject: |
|
|
Hopeless wrote: | Yeah, I know, I usually try making shell scripts posix-compliant if possible, but I just thought the "wc -l" was a little ugly, especially when you can do the same with bash internals... |
Code: | #!/bin/sh
THREADS="4"
count_jobs()
{
set -- $(jobs -p)
echo $#
}
while [ $# -gt 0 ]; do
JOBS=$(count_jobs)
if [ "${JOBS}" -lt "${THREADS}" ]; then
$1 & shift
else
sleep 1
fi
done
wait
|
wc removed, still don't need bash _________________ Use dhcpcd for all your automated network configuration needs
Use dhcpcd-ui (GTK+/Qt) as your System Tray Network tool |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Wed Feb 11, 2009 4:56 am Post subject: |
|
|
I feel stupid...
Doesn't kill every background jobs in current session/shell/howimsupposedtosaythat?
EDIT: current shell environment _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Akkara Bodhisattva
Joined: 28 Mar 2006 Posts: 6702 Location: &akkara
|
Posted: Wed Feb 11, 2009 5:22 am Post subject: |
|
|
If you are doing a lot of the same kinds of commands, look into xargs, in particular the --max-procs=N option.
Zucca wrote: | Mainly I need this when I convert many files from a format to another. |
Are they audio files? I had written a script that might help you. Although that script can be improved taking some suggestions from UberLord's code. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Wed Feb 11, 2009 3:20 pm Post subject: |
|
|
Akkara wrote: | If you are doing a lot of the same kinds of commands, look into xargs, in particular the --max-procs=N option.
Zucca wrote: | Mainly I need this when I convert many files from a format to another. |
Are they audio files? I had written a script that might help you. Although that script can be improved taking some suggestions from UberLord's code. | This is what I'm coding... I think I will change the name and include support for other formats. Anyway the main idea is to be able to convert flac files to other formats (mainly for mp3-players and mobile phones).
EDIT: I should have looked your link first. :D I guess I can l stop my development. But I think my appoarch with LAMEOPTS is bit more userfriendly..? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Sat Feb 21, 2009 4:50 pm Post subject: Number of threads? |
|
|
What's the best way to count the number of CPUs?
I came up with this little&dirty idea: Code: | let "THREADS=`find /sys/devices/system/cpu/ -name 'cpu[0-9]*' | wc -l`+1" |
It sets THREADS variable to number of CPUs+1. I guess that ain't POSIX compatible? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
hielvc Advocate
Joined: 19 Apr 2002 Posts: 2805 Location: Oceanside, Ca
|
Posted: Sun Feb 22, 2009 12:55 am Post subject: |
|
|
For your number of cpu's, cores, and or hyperthreaeds Code: | hielvc@hielvc-laptop:~$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 28
model name : Intel(R) Atom(TM) CPU N270 @ 1.60GHz
stepping : 2
cpu MHz : 800.000
cache size : 32 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pebs bts pni monitor ds_cpl est tm2 ssse3 xtpr lahf_lm
bogomips : 3191.96
clflush size : 64
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 28
model name : Intel(R) Atom(TM) CPU N270 @ 1.60GHz
stepping : 2
cpu MHz : 800.000
cache size : 32 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx constant_tsc pebs bts pni monitor ds_cpl est tm2 ssse3 xtpr lahf_lm
bogomips : 3192.02
clflush size : 64
power management:
awk '{ if ($1 ~ /processor/) proc = $3 + 1 } END {print proc}' /proc/cpuinfo
2
| an eeepc 900a with hyperthreading _________________ An A-Z Index of the Linux BASH command line |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Sun Feb 22, 2009 6:26 pm Post subject: |
|
|
Or just: Code: | egrep '^processor' /proc/cpuinfo | wc -l |
But I think awk is more POSIX? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Mon Feb 23, 2009 8:16 pm Post subject: Troubles killing all the background processes |
|
|
I have troubles killing all the background processes...
I run commands in the while -loop like this: Code: | $CMD &> /dev/null && echo "Finished: `basename $FLAC` --> `basename $OUTF`" || echo "Failed: `basename $FLAC` --> `basename $OUTF`" & |
Variable CMD obviously changes depending what options user gives at the command line.
CMD is actually a function. Here's an example: Code: | function toogg {
get_tags
flac --totally-silent -cd "$FLAC" | oggenc -o "$OUTF" "${ENCOPTS[@]}" --artist "$ARTIST" --title "$TITLE" --album "$ALBUM" --tracknum "$TRACKNUMBER" --date "$YEAR" --genre "$GENRE" -
} | The problem is that function spans a new bash process and when I hit ctrl+c to send SIGINT, my 'kill $(jobs -p)' kills only the bash processes and leaves flac and oggenc (in this case) processes still running.
So how do I get the pids of every background process under the current shell? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3309 Location: Rasi, Finland
|
Posted: Tue Feb 24, 2009 8:05 pm Post subject: |
|
|
Sorry for double(EDIT: triple)-posting, but I found one way to do it: Code: | trap "kill 0" SIGINT |
_________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
|