As the MPEG4 limit for iPod encoding is 480x480 (230,400 pixels) and one may alter the ratio as long as the pixel limit is respected (640x360 is also okay), I wondered if the same "pixel limit" held true for H264 (320x240 = 76,800 pixels total). In other words: Would an H264 width larger than 320 pixels be accepted by the iPod as long as the width times height was no more than 76,800 pixels?
I'm really pleased to report that this works fine. I used ffmpegX (9uR2) to convert an AVI over to "iPod H264" last night. As the title was in widescreen format, I played with my calculator a bit until I came up with 416x176 as matching the original ratio. 416x176 = 73,216 pixels -- under the 76,800 pixel limit.
In order to maximize the quality (and still keep the file size reasonable), I used 450KBps, Qmin:3, Qmax:30, Keyframe:20, Audio:96kb, ME function: Multi-hexagon. I use "Decode with QuickTime" all the time simply because it works all the time. Other checkboxes are left as is.
The 1:24:00 file ended up at 332MB and, at H264's quality, is a substantial improvement over an MPEG4 encoding at the same pixel size, or even a considerably larger pixel size. It's also considerably better than if I had used 320x144 (the closest acceptable width to the movie's original ratio). Do the math: 320x144 = 46,080, or 63% of the 73,216 pixels I had in my final file.
By the way, the method I use to drop my video files into my iPod is just that: Drop the file directly into the iPod within the iTunes interface (not the mounted iPod in the Finder). The file never exists in my iTunes Library. Some folks are reporting issues copying from the iTunes Library to the iPod with the 6.0.2 iTunes and the latest firmware update. Using this method, I have no problems.
[robg adds: This previous hint seemed to indicate that H.264 would not work. Since I don't have a Video iPod to test with, I thought I'd run this one as a new comment so that the info is out there, in case it does work...]
Mac OS X Hints
http://hints.macworld.com/article.php?story=2006011307200996