Using Parallel Instead of For Loops

For loops are an addiction of mine, I use them all day every day. Any time you have a tool that does one thing well but doesn't support multiple inputs or inputs from a file, I use a bash for loop. Unfortunately for loops work sequentially, one after the other. Once process runs, finishes, exits, and the next process starts, finishes, exits and so on.

Many times I've come across a tool or process that just hangs, and as a result hangs all the later processes as well. In situations where I think that is likely to happen, I'll use parallel.

Ok, so lets make a for loop that resolves the MX records of google.com

for i in $(host google.com | grep 'mail is' | cut -d ' ' -f7); do printf $i:; host $i | grep 'has address' | cut -d ' ' -f4; done
alt1.aspmx.l.google.com.:74.125.192.27
alt2.aspmx.l.google.com.:74.125.141.27
aspmx.l.google.com.:209.85.147.26
alt4.aspmx.l.google.com.:209.85.203.26
alt3.aspmx.l.google.com.:64.233.190.27

Great, nothing fancy there. Now lets say for some reason one iteration of that for loop is hanging and lets pretend we are using a tool (not "host") that has ridiculous timeouts (e.g. nikto on default), wouldn't it be great to run several all at the same time in groups and as one finishes it's spot in the group the next iteration populates it's place? yeah, thats what parallel does. Let's change that for loop to use parallel instead:

host google.com | grep 'mail is' | cut -d ' ' -f7 | parallel -j 5 -I{} -r "printf {}:; host {} | grep 'has address' | cut -d ' ' -f4"
alt2.aspmx.l.google.com.:209.85.202.27
alt3.aspmx.l.google.com.:108.177.15.27
aspmx.l.google.com.:209.85.201.27
alt4.aspmx.l.google.com.:74.125.136.27
alt1.aspmx.l.google.com.:173.194.68.27

This shows you how to send piped bash commands to parallel, instead of just single processes. In this way, it functions very similarly to the classic "while read line" looping structure.

BONUS:
The same command using xargs (very similar, works on OSX & nix):
host google.com | grep 'mail is' | cut -d ' ' -f7 | xargs -I {} sh -c "printf {}:; host {} | grep 'has address' | cut -d ' ' -f4"
aspmx.l.google.com.:209.85.232.27
alt3.aspmx.l.google.com.:64.233.167.27
alt2.aspmx.l.google.com.:74.125.24.27
alt1.aspmx.l.google.com.:64.233.186.27
alt4.aspmx.l.google.com.:74.125.136.26