Some large bittorrent files will be composed of hundreds of smaller .rar files. If the total number of files in a torrent is over 256, OS X bittorrent applications will fail with the "too many open files" error. This is because by default, OS X applications have a maximum limit of 256 open files. To get around this, in a terminal type:
ulimit -n 2000
Use 2000 or some other suitably large number. Then launch your bittorrent application from the same terminal window by cd'ing into the package, etc. Note that this will not work unless your application is launched from the same terminal.
Mac OS X Hints
http://hints.macworld.com/article.php?story=20041003195620508