[robg says: The following hint is presented as it was submitted. As noted in the comments by the hint's author, the introduction is misleading -- your Mac can do real surround sound, assuming the source has an AC3 soundtrack. However, what the author then goes on to describe explains how to convert certain AAC-surround-encoded files to AC3 mode for true surround playback.
I've chosen to leave the hint online, as the hint (and moreso, the comments) contain a wealth of useful information. Just take the intro to the hint with a grain of salt, as it's not the whole truth. I have also modified the title of the hint to more accurately reflect what it's about.]
Your Mac can not do real surround sound from its built-in optical audio port; in fact, not even your Apple TV can. "But Wait!" you say, "Yes it can, Apple even advertise surround sound as a feature of the Apple TV!" or "I can play a DVD and I'm hearing surround sound."
Well, the truth lies somewhere in between these two extremes. First, the Apple TV. Apple supply media to the Apple TV with one of two different options for the soundtrack. It sometimes uses a stereo soundtrack that uses Dolby ProLogic to do surround sound. This isn't "true" surround sound, it's surround information matrix-encoded into a regular two-channel audio stream, and done extremely cleverly.
Other times, and more often on the HD content, it is actually a real 5.1 surround soundtrack, but it's in AAC format. Your surround receiver probably can't decode AAC, and at any rate, the Apple TV won't send it as AAC, it decodes it, mixes it back up as a stereo soundtrack (using Dolby ProLogic) and outputs that. Either way, you're hearing Dolby ProLogic, not Dolby Digital.
Now, for the Mac. Under a certain set of circumstances, your Mac can output a surround stream from the optical output that a surround receiver can decode as proper surround -- this is if the media file you're using already contains an AC3 encoded soundtrack. AC3 is the codec that Dolby Digital uses, so if you've already got a Dolby Digital soundtrack, and your optical port is configured properly (as a digital passthrough), then you may get the AC3 stream output through the optical port, and your surround receiver decodes it. You will have real surround sound from this setup.
If you're watching media that uses, for example, an AAC-encoded multi-channel soundtrack (most of the Apple HD trailers are like this) then it will be like the Apple TV situation above -- your Mac can't send the AAC stream out the audio port, as it's only a two-channel device, so QuickTime player (or VLC or...) mixes it down to stereo and outputs this. If you're lucky, it'll be Dolby ProLogic; if you're not, it will be plain old 2.0 stereo.
Well, after a decent amount of research and tweaking, there is a solution to this problem. I wish you the best of luck getting this to work on an Apple TV, though. It works perfectly on my Mac mini, and the only downside is the manual configuration that needs to be performed.
This is going to be pretty heavy going, and it gets quite technical. If you're looking for a quick fix, you're not going to find it here; at the moment there is no easy solution. What solution there is, we can thank the author of AC3Jack (Jesse Chappell), and the authors of Jack OS X (Stephane Letz, Johnny Petrantoni and Dan Nigrin).
I did this on my Mac mini, which I use as a media center computer. It's an older model 1.83GHz Intel Core Duo with 2GB RAM. It should work on pretty much any Intel Mac, as I believe they've all got an optical output. If not, you're out of luck.
It's critical that the steps below are performed in the order listed. Don't, for instance, go and launch QuickTime before anything else, as it won't work.
Mac OS X Hints
http://hints.macworld.com/article.php?story=20090215033647292