Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Compile *nix programs faster multi-core Macs' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Compile *nix programs faster multi-core Macs
Authored by: brettmjohnson on May 25, '07 02:55:52PM
robg adds: At some point, I'd like to test this with a large compile, but I haven't had the time as of yet...

Actually, this works poorly for large compiles. There are 2 problems:
  1. If you specify the number of parallel jobs, make -j n, but have nested projects (make spawns other makes in sub-processes), the top level make will execute parallel targets, but the -j n is not passed on to the child make processes. In most large projects I've worked on, most of the compilation actually occurs in the sub projects, and the targets in the root project have a necessary build order that often prevents them from being built concurrently. If you call make -j without specifying a maximum number of parallel jobs, then the -j option does get passed down to the child make processes. But specifying make -j without a maximum job count has a serious drawback...

  2. If you run make -j without a maximum job count, and have a large number of files that need to be compiled, make will spawn an unlimited number of jobs. By default, Mac OS X limits the maximum number of process per user to 100. If make tries to spawn more than 100 jobs (actually 100 - number of already running processes), then the build will fail with a large number of the following error: "vfork: Resource temporarily unavailable." You can avoid this by increasing the maximum number of processes per user using launchctl maxprocs ... However, even bumping the maxprocs to 512 or 1024 has another undesirable side effect: spawning 500 or 1000 active (non-idle) processes quickly overwhelms the kernel CPU scheduler, even on my 4-core PowerMac. The result is that make spends 90% of its time context switching, not compiling.
So in my experience with large compile jobs, make -j creates too many concurrent processes, and make -j n creates too few concurrent processes. I believe make -j should be enhanced to spawn no more than 4 or 6 jobs per CPU core.

[ Reply to This | # ]
Compile *nix programs faster multi-core Macs
Authored by: woodgie on May 25, '07 03:49:21PM

That is a beautifully written, concise description of what is going on and I appreciate you taking the time to post.

It's posts like this that are utterly invaluable to learning as it gives me a nice grounding for going out to find more.

Thank you very, very much.



[ Reply to This | # ]
Compile *nix programs faster multi-core Macs
Authored by: jochen Küpper on May 26, '07 09:35:23AM

Set the MAKE environment variable. At least with GNU make that works nicely on all platforms I have used it (including Mac OS X;)

See the make info manual (info) as well.



[ Reply to This | # ]
Compile *nix programs faster multi-core Macs
Authored by: bugmenot on May 26, '07 10:10:11AM

like so:
export MAKEFLAGS="-j3"



[ Reply to This | # ]
Compile *nix programs faster multi-core Macs
Authored by: jochen Küpper on May 26, '07 09:37:34AM

Well, sorry about my "random" first comment...

What's about he "-l" flag? Does that not work for you?



[ Reply to This | # ]