Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

A bash script to retry large compilations via 'make' UNIX
When compiling large programs/libraries, a segmentation fault can happen from the compiler. This can be frustrating when it takes several hours to compile everything since you have to start 'make' manually until it works. I found out an elegant way to automatically start the compilation until the job is completed. In bash, the return value of the last executed program is stored in the $? variable. For example:
$ mkdir existingdir 
$ cd existingdir 
$ echo $? 
0 
$ cd nonexistingdir 
-bash: cd: nonexistingdir: No such file or directory 
$ echo $? 
1
Note that a successfully run program usually returns 0. That being said, it is easy to check whether your building program (e.g. make) succeeded, and if not, to start it back up. I used the following bash loop:
$ while [ $? != 0 ] ; do make ; done
Of course, if the program/library doesn't compile (for other reasons than a segmentation fault by the compiler), this will create an infinite loop that you might want to avoid. I therefore wrote a small script that tries to build five times before stopping:
#!/bin/bash 
NBTRIES=0 
# run invalid command first 
cd /dev/null 
while [[ $? != 0 && $NBTRIES < 5 ]] 
do 
  let NBTRIES=NBTRIES+1 
  make 
done 
After running this script, I can go to sleep without any worries ;)
    •    
  • Currently 1.00 / 5
  • 1
  • 2
  • 3
  • 4
  • 5
  (1 vote cast)
 
[6,091 views]  

A bash script to retry large compilations via 'make' | 8 comments | Create New Account
Click here to return to the 'A bash script to retry large compilations via 'make'' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
A bash script to retry large compilations via 'make'
Authored by: etrepum on Jun 25, '04 11:11:20AM

Wow that is a crazy idea. What the hell are you compiling that brings out the bugs in gcc?



[ Reply to This | # ]
A bash script to retry large compilations via 'make'
Authored by: LC on Jun 25, '04 11:14:14AM
until make ; do echo "make failed. Retrying ... -C to interrupt" ; sleep 1 ; done

[ Reply to This | # ]
A bash script to retry large compilations via 'make'
Authored by: LC on Jun 25, '04 11:16:04AM

oops ... that php stripper removed my &lt; &gt; again ...
s.b. <CTRL>-C
Larry



[ Reply to This | # ]
You've got hardware problems.
Authored by: TrumpetPower! on Jun 25, '04 01:00:46PM

If you're getting segfaults and re-starting ``make'' lets you continue, you've almost certainly got bad hardware. If gcc is dying with SIG11 errors, you can remove the ``almost.'' In fact, repeatedly compiling known-good code is a classic hardware test.

RAM is often cited as the most likely culprit, but it could also be the CPU, motherboard interconnects, a bad hard disk or hard disk cable, bad power supply....

I sure hope you're regularly making true archival backups. Don't be surprised if, before long, you're looking for a six-month old copy of some file, because it corrupted when you saved it five months ago.

Cheers,

b&



[ Reply to This | # ]
You've got hardware problems.
Authored by: wgscott on Jun 25, '04 02:15:26PM

I've had seg faults with g77 v. 3.4 that are reproducible, but I haven't got around them by retrying. Sometimes lowering the optimization helps.



[ Reply to This | # ]
You've got hardware problems.
Authored by: greck on Jun 25, '04 09:48:08PM

This is exactly what I was about to say... every single time (and I've been doing this a pretty long time) I've seen gcc die on an 11, it's been bad hardware. And not even bad as in "broken" all the time, e.g. I have a couple of Shuttle SK43Gs that die identically during extended compiles, apparently by design.

The first time I ever learned this lesson was with a 100 MHz Pentium that would only get through a kernel compile if you downclocked it to 90. It was a little mini-tower, and one day after monkeying around inside it my roommate noticed a faint *clink* when I stood it back up. A little while later we found the noise... the thermal compound between the CPU and the heat sink had dried out, and when you stood the board up on its side, the heat sink would fall away from the CPU slightly. A buck's worth of compound later, it worked like a champ at 100.

So script-writer, it's definitely worth checking to make sure all your fans and heat sinks are in good working order, and if you don't find anything, then start looking for a borked part before it really bites you in the ass.



[ Reply to This | # ]
build five times before stopping
Authored by: sigfpe on Jun 25, '04 02:04:37PM
make || make || make || make || make
works fine

[ Reply to This | # ]
A bash script to retry large compilations via 'make'
Authored by: sjk on Jun 25, '04 10:47:18PM

I'm struggling to understand how rerunning make after it's failed will "magically" work unless you've fixed the reason for the original failure(!)

The only thing that sounds close to what you're describing should be accomplished by adding the "-i" and/or "-k" options to your make command line.



[ Reply to This | # ]