Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Leverage Dual CPU's with GNU Make UNIX
Users who build open source software from source (aka the configure; make; make install method) on dual CPU Macs can take advantage of the the second CPU by adding -j 2 as an argument to the make command. The -j 2 argument tells make to compile two source files at the same time for the duration of the build, with each source file being compiled by one of the processors.

[robg adds: Although this option is listed in the man page, it's not at all clear that it will use the second processor for the additional job...]
    •    
  • Currently 2.40 / 5
  You rated: 2 / 5 (5 votes cast)
 
[12,040 views]  

Leverage Dual CPU's with GNU Make | 15 comments | Create New Account
Click here to return to the 'Leverage Dual CPU's with GNU Make' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Leverage Dual CPU's with GNU Make
Authored by: timkingman on May 03, '04 10:41:41AM

The reason the man page doesn't mention multiple CPUs is because make has no way of determining this. -j 2 simply tells make to start two compiling processes. It's up to the OS to figure out how to effectively run both simultaneous processes, and the usual method is to put them on different CPUs.

Some compiling instructions (especially as you get into distcc and spreading your processes over multiple machines) have you use a higher -j number than the total available CPUs, I assume to make sure every available cycle gets used. YMMV.



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: nathanst on May 03, '04 10:51:56AM

I use this every day -- works fantastically for me. If you have a DP machine you may want to change your .cshrc file by adding the line:

alias make make -j 2

The activity monitor or top will confirm that you are using both processors. (As well as the performance increase!)



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: GaelicWizard on May 03, '04 12:14:31PM

Unfortunately, if you added that alias to your shell then it will break many instances of "make install". It did for me at least. Most makefile writers don't take the time to write the install target correctly, so -j2 will try to install files before the directories have been created. :-/ YMMV

---
Pell



[ Reply to This | # ]
Better yet...
Authored by: gospodin_david on May 03, '04 12:22:28PM

A good general way of handling this sort of thing is to take the number of processors times two and put that in the -j option.

This is because when there are two compilations per processor, one can be compiling while the other can be waiting for something from disk. Thus, with a dual-processor machine, a roughly optimal setting is "-j 4."

With larger numbers of processors, the formula doesn't work as well, but as we Mac users are limited to 2 max at the moment (please Apple, make a 2U 4-proc XServe!), it works just fine.



[ Reply to This | # ]
Better yet...
Authored by: risc_abacus on May 04, '04 10:37:24AM

Some place... don't remember where... but I read the ideal setting would be -j(n*2) where n is the number of processors...

So...
1 CPU -> -j2
2 CPU's -> -j4
3 CPU's -> -j6 (using 'distcc' you might have this set-up, 1-Dual one Single System)
4 CPU's -> -j8
n CPU's -> -j(n*2)



[ Reply to This | # ]
-pipe
Authored by: porkchop_d_clown on May 03, '04 12:53:35PM

You can also use the -pipe flag with gcc. That tells gcc to use pipes between compilation stages instead of temporary files.

This has the effect of allowing the different stages of compilation to run on different processors in overlapping fashion.

---
Everyone loves a clown, but no one will lend him money!



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: Arakageeta on May 03, '04 09:55:54PM

This hint is not complete-- timkingman got to it first. The -j flag will help on single processor machines as well.

The j stands for job, obviously. But to say that one job will run on one processor while another job will run on the other is totally incorrect. Both processors will share the load of the jobs. With all the processes going on on a given computer, your compile jobs will be competing for processor time with other apps be it Safari, OS X services such as Quartz or any number of other processes. All the -jN flag does is to tell make to do at most N jobs at one time. Since these jobs are independent, the total computation to be done to complete the entire compile job is very parallelizable -- multi-processor systems are best for such jobs.

But back to my point, this will still help on single processor machines. When one job gets kicked out of the CPU on a memory or disk access, another process will take its place-- multiple make jobs allow other compile jobs to step in.

The long and short of it is this: The rule of thumb that I've heard concerning -j is to specify twice the number of processors you have on your machine. By this rule a single processor machine should use -j2. A dual mahine, -j4. You can go higher too, but you will start to experience diminishing returns. I think if you had -j1000 (assuming you had that many independent source files to take advantage), I don't think that would run as well as, say -j10-- there is overhead involved with each process/thread running on the machine. At work I compile on a dual Xeon machine running RedHat 8.0. For a project that usually takes 50 minutes with -j1, only takes 18 minutes with -j8 (-j4 gives me about 25 minutes).

Careful if you are running on a machine that is shared with other users (like a university server). If the machine does not cap individual user processor usage you will dominate the processor and starve the other users' processes-- this may cause you to anger the other users on the system. Be nice.



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: ceesaxp on May 04, '04 01:33:07PM

Very comprehensive, but I'd still add that it may all also depend on (a) version of make in use; (b) OS in use; (c) what is being compiled.

On my FreeBSD machine, buildworld/buildkernel runs with -j10 (it is a dual Celery 466). Can't say it is x10 viz. no -j flag, but it is pretty fast. However, I never compile any ports with -j flag, as sources are really tailored well for parallel make. Pitty, since I would not mind recompile of KDE to take even 50% less than current c.a. 10 hours...



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: krishna on May 04, '04 10:33:16PM

Well, you can *be* nice by *using* nice:

nice -n 20 make -j 2

Will lower the priority (I always forget if it should be +20 or -20, check the man page). Then your build process will have the lowest claim on processor cycles and won't starve out other users or programs for processor time.

In particular, I run

nice -n 20 screen -dmS compile

and then reconnect to that screen session when I want to do a build. That way anything I build in that screen session, even if I do smaller subcompiles by hand, is at a lower priority, and doesn't steal cycles from important things like iTunes :-)



[ Reply to This | # ]
Marketing Slang
Authored by: forman on May 04, '04 02:24:18AM
The verb "to leverage" has only one meaning, "to use credit to enhance speculative capacity."

The usage in your title is strangely out-of-place marketing slang (GNU and marketing slang are an odd combination). I suggest "use", "make use of", or "take advantage of" as an alternative.

Main Entry: leverage
Function: transitive verb
Inflected Form(s): -aged; -agĀ·ing
: to provide (as a corporation) or supplement (as money) with leverage; also
: to enhance as if by supplying with financial leverage


[ Reply to This | # ]
Marketing Slang
Authored by: aaronfaby on Aug 18, '04 06:37:14PM

Dictionary.com also defines (def 2) leverage as:

2. Positional advantage; power to act effectively

It seems rather fitting to me. Remember, words are allowed to have more then one meaning.

---
Great Mac Hosting - http://www.serverlogistics.com



[ Reply to This | # ]
Beware - this may break things!
Authored by: merlyn on May 04, '04 05:33:09AM
Beware, however, that not all makefiles can be used with -j, although the likelihood of encountering such a Makefile today would be rare.

And unfortunately, such breakage will sometimes be undetectable, and random. The worst kind of breakage.

In a past life, I was a test engineer at the first company to use commodity hardware for a symmetric multiprocessing system (Sequent), and when we started "parallelizing" Makefiles, many of them broke big time because of fixed-name temp files for some of the steps.

[ Reply to This | # ]

Beware - this may break things!
Authored by: krishna on May 04, '04 10:37:35PM
In a past life, I was a test engineer at the first company to use commodity hardware for a symmetric multiprocessing system (Sequent),

Don't listen to him! He's an old dinosaur! What does he know about MacOSX? This is the new world of UNIX!!!

I wonder if you feel so very, very old as I do :-)

[ Reply to This | # ]

distcc
Authored by: calroth on May 07, '04 07:36:06AM

Why settle for using two processors when you can use all of them?

http://distcc.samba.org/

This is also what Xcode uses behind the scenes to do its distributed compiling magic.



[ Reply to This | # ]
Leverage Dual CPU's with GNU Make
Authored by: Anonymous on Aug 19, '10 09:30:11AM

Or use "-j -l 10" to allow the number of threads to run away into the hundreds, but be capped by system load, in this instance, 10.

The more cores you have, the higher system load can go for a make. I find about 4 whole units of system load per core is a good match between performance and speed.

Lowering that core multiplier to 2 or even just 1 will mean fewer threads created, so less system load.

Similarly, increasing it will get to the point of reduced system responsiveness.

You could couple this with "nice" too. I don't have a need for that.

Making this a user-level alias is, as others have explained above, generally a bad idea. While "make" with compilation targets is often just fine, the "make install" stage will often break.



[ Reply to This | # ]