Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Leverage Dual CPU's with GNU Make' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Leverage Dual CPU's with GNU Make
Authored by: Arakageeta on May 03, '04 09:55:54PM

This hint is not complete-- timkingman got to it first. The -j flag will help on single processor machines as well.

The j stands for job, obviously. But to say that one job will run on one processor while another job will run on the other is totally incorrect. Both processors will share the load of the jobs. With all the processes going on on a given computer, your compile jobs will be competing for processor time with other apps be it Safari, OS X services such as Quartz or any number of other processes. All the -jN flag does is to tell make to do at most N jobs at one time. Since these jobs are independent, the total computation to be done to complete the entire compile job is very parallelizable -- multi-processor systems are best for such jobs.

But back to my point, this will still help on single processor machines. When one job gets kicked out of the CPU on a memory or disk access, another process will take its place-- multiple make jobs allow other compile jobs to step in.

The long and short of it is this: The rule of thumb that I've heard concerning -j is to specify twice the number of processors you have on your machine. By this rule a single processor machine should use -j2. A dual mahine, -j4. You can go higher too, but you will start to experience diminishing returns. I think if you had -j1000 (assuming you had that many independent source files to take advantage), I don't think that would run as well as, say -j10-- there is overhead involved with each process/thread running on the machine. At work I compile on a dual Xeon machine running RedHat 8.0. For a project that usually takes 50 minutes with -j1, only takes 18 minutes with -j8 (-j4 gives me about 25 minutes).

Careful if you are running on a machine that is shared with other users (like a university server). If the machine does not cap individual user processor usage you will dominate the processor and starve the other users' processes-- this may cause you to anger the other users on the system. Be nice.



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: ceesaxp on May 04, '04 01:33:07PM

Very comprehensive, but I'd still add that it may all also depend on (a) version of make in use; (b) OS in use; (c) what is being compiled.

On my FreeBSD machine, buildworld/buildkernel runs with -j10 (it is a dual Celery 466). Can't say it is x10 viz. no -j flag, but it is pretty fast. However, I never compile any ports with -j flag, as sources are really tailored well for parallel make. Pitty, since I would not mind recompile of KDE to take even 50% less than current c.a. 10 hours...



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: krishna on May 04, '04 10:33:16PM

Well, you can *be* nice by *using* nice:

nice -n 20 make -j 2

Will lower the priority (I always forget if it should be +20 or -20, check the man page). Then your build process will have the lowest claim on processor cycles and won't starve out other users or programs for processor time.

In particular, I run

nice -n 20 screen -dmS compile

and then reconnect to that screen session when I want to do a build. That way anything I build in that screen session, even if I do smaller subcompiles by hand, is at a lower priority, and doesn't steal cycles from important things like iTunes :-)



[ Reply to This | # ]