First, just drag an item from the iTunes library onto the iTunes LCD, the display at the top of the window that shows what's playing. This can be a single song, an album, or a playlist. The iTunes LCD will show a blue border when you bring the item over it, and the Up Next icon will flash with art of the item you have added.
The second way is to press the Option key and hover your cursor over an item. The track number next to its name will change to a + icon. Click that icon to add it to Up Next. (Thanks to David Chartier for this one.)

