Getting 5.1 surround sound output from iTunes can be difficult. There are some hoops you need to jump through to get it to work; here's how I got it working. First, the things you need:
Open the Audio MIDI Setup application located in ~/Applications » Utilities. You want to change the format to 48000.0Hz and 24-bit, if it's not set to that already different. It's confusing, but you do want two-channel.
The next step is to start playing the movie you want to watch. If encoded properly, the movie will have two audio tracks. Make sure the volume is at 100%, otherwise you will get horrible static. If you forget this step, you'll probably get static -- and depending on how loud your receiver is set, it could be quite shocking.
Press Play, and then hover over the HUD menu in the video. Change the volume track as seen in the screenshot at right. That's it; if you meet the requirements, you should be hearing multichannel sound right in iTunes.
Something you have to keep in mind, you need to turn off SRS iWOW if you want surround sound output to work. In my testing, that plug-in seemed to kill the output. This blog entry on my site includes more detail on all of the above.
Note that I have not played with any other types of video on iTunes, other than that converted for the Apple TV, so your mileage may vary if you try something else. Hopefully this helps if you're trying to get surround sound output on iTunes. I don't have a clue on how to get it to work on Front Row just yet.
[robg adds: I'll admit to never having tried anything related to 5.1 audio and iTunes videos; I don't know if the above is the easiest way to do this or not. Feel free to comment if you know more about this subject.]
Mac OS X Hints
http://hints.macworld.com/article.php?story=20091118074715846