I also noticed that this dramatically slows the application, as was mentioned here on macosxhints before. The author of a previous hint suggested that you set the "IMAP path prefix" to "/var/spool/mail/username." While this makes mail.app more responsive, it does not allow you to save mail to any mailbox on the server.
My resolution to this issue was simply to set my IMAP path prefix to "~/mail" which allows me to create mailboxes that are located in a folder called "mail" on the server in my home directory (mail.app will create this folder). This also makes mail.app more responsive, as it does not have to parse all the other folders in my home directory, and you can then create mailboxes and save mail on the server!
Hope this tip helps some.

