I was adding a Smart Playlist to iTunes (10.7) and could not find the option to add nested rules... until I held down the Option key and the plus icon became an ellipsis which then provided the ability to add a nested rule.
[kirkmc adds: This isn't new in iTunes 10.7, but rather iTunes 10. However, I see that it's not on the site, so it's worth mentioning. Apple has a technical note about smart playlists, and they call the ellipsis button the "Nest" button.]
Mac OS X Hints