Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Fix message counts and 'ghosts' in Mail's smart mailboxes Apps
Out of all the great ideas I initially had for using smart mailboxes in Mail.app, only one I could actually implement due to Apple's decision to bless us with a really limited and sometimes incredibly stupid feature set in the GUI, hiding away most of what would be technically possible: That was having one smart mailbox showing all unread and another one showing all flagged messages.

From my previous experience and what I had read elsewhere about Spotlight and friends, however, it did not surprise me that these folders showed totally incorrect results, rendering the smart mailboxes completely useless for me. The message counts were totally wrong and there were even ghost messages which did not exist anymore in any real (as opposed to smart) mailbox. Mail.app would always say that those messages have not yet been downloaded; mostly those were really old messages. most of the 'ghosts' appeared several times.

So for months, I have been ignoring these smart mailboxes, occasionally doing some research on the Apple discussions and elsewhere about what could be the cause of the problem, but I could never find any appropriate info. Just now, however, I finally managed to fix it. I quit Mail.app, made a backup of my entire ~/Library/Mail/ folder (just in case), deleted the file Envelope Index from within that folder, restarted Mail.app, clicked OK in the message that Mail.app needed to import messages -- and finally had working smart mailboxes with correct results! Yeah, after months of using 10.4, the first possibly useful application of any of those smart things in Tiger is in sight!

I had been tempted before to try the method described, but the warning message that Mail.app would need to import messages had always made me cancel the procedure, as I was not willing to guide Mail.app through a tedious import procedure for the more than 16,000 messages in about 45 mailboxes. It seems that deleting the file Envelope Index really only forces the recreation of the index file (as the name of the file suggests), and has nothing to do with "importing" messages/mailboxes (as in File -> Import Mailboxes...) or losing other settings (as Mail.app's warning message suggested to me).

I had tried various other methods before which seeemed to be worth a try: turning Spotlight off and on, forcing the deleting and rebuilding of Spotlight indices, deleting Mail.app's cache files, recreating the smart mailboxes, rebuilding all mailboxes (Menu Mailbox > Rebuild), temporarily switching the caching of all messages and attachments in my IMAP accounts off and on, etc., but nothing had worked. It seems I was too much influenced by all this Spotlight advertising stuff which made me think that Mail.app was actually using Spotlight technology for this. Instead, it just uses an index file which has always been there, even before 10.4. At least, for pre-10.4 systems, it was used for read/flagged status, from/to/subject/date headers, etc. For searching message contents, Spotlight indices seem to be used, as I found out when I completely turned Spotlight off because it was just not useful, reliable and fast enough to depend on. So that's why the method described here could also be useful for someone using pre-10.4, but I haven't tried.
    •    
  • Currently 3.40 / 5
  You rated: 5 / 5 (5 votes cast)
 
[20,094 views]  

Fix message counts and 'ghosts' in Mail's smart mailboxes | 13 comments | Create New Account
Click here to return to the 'Fix message counts and 'ghosts' in Mail's smart mailboxes' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: smorr on Aug 29, '05 12:58:36PM

Some background info.

Having worked around alot with Mail plugins and indexing information, I have acquired what I think a good understanding of Mail's internal structures. Mail uses a combination of an internal sqlite3 database (envelope index) and spotlighting technology. Spotlight would be prohibitively slow for most searches and mailbox listings so it stores the sender, date, subject etc in the sqlite database (so the table list view is fast and most searches are fast -- compare a search on full text vs a search on subject and you will see what I mean) For smart mailboxes that search both full text and headers, it uses a combination, searching the spotlight index and then searching the results agains the sqlite database -- in fact there is an issue in Mail (up to 10.4.2 at least) where it will always perform this as AND (all criteria must match) ignoring the OR (any criteria may match)

However, it also uses spotlight for full text searches and indexes the other information (sender, date, subject etc) with spotlight as well so that the system wide spotlight search works well. To do the spotlight indexing Mail 2.0 had to go with 1 message=1 file approach (because spotlight indexes files, not records in other databases) but the happy thing about this means that most information is stored redundantly (in the sqlite database), in the emlx file (as a xml plist at the end of the file) and in the spotlight database.

Btw, MailTags extends the information in the emlx file and spotlight, but not the sqlite database to do its magic, so MailTags searches are similar to Spotlight searches.

[ Reply to This | # ]

Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: moritzh on Aug 30, '05 01:46:35PM

Good summary of how things work with Mail.app and Spotlight. Your explanations match well with what I experienced.

For smart mailboxes that search both full text and headers, it uses a combination, searching the spotlight index and then searching the results agains the sqlite database -- in fact there is an issue in Mail (up to 10.4.2 at least) where it will always perform this as AND (all criteria must match) ignoring the OR (any criteria may match)

Thanks especially for explaining that, this is one of the things I referred to in the hint when I said that smart mailboxes/Spotlight in Mail.app in many configurations just seemed to buggy/unreliable from my experience to depend on.



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: Mikey-San on Aug 29, '05 03:43:31PM

Do we really need the negative, Slashdot-esque editorials in the hints?

Submitter should be hit with a Mighty Mouse.



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: mactectomy on Aug 29, '05 10:54:29PM

I will offer my new Mighty Mouse, since Apple never bothered to bless it with the ability to scroll to the right. At least it can scroll up--it's a pity that so many others were not thus blessed.



[ Reply to This | # ]
negative editorials?
Authored by: sjk on Aug 30, '05 01:12:44AM
Are you referring to negative editorials in this hint? I don't see 'em.

[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: martyl on Aug 29, '05 08:51:25PM

My smart mailbox "Flagged" has 7 messages in it, but the message count says 17... tried this hint twice, but no change at all. Also tried trashing the smartmailboxes.plist and its .backup file, deleting and recreating the mailbox, changing the and/or option and other criteria; any other ideas?
(Mail 2.0.2, Tiger 10.4.2)

Marty

---
Marty Lindower



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: moritzh on Aug 30, '05 01:25:54PM

There are some other suggeestions in the hint, did you try them all? I'd definitely give rebuilding mailboxes a try. Maybe, you also have the option set that deleted messges are just hidden (as opposed to "moved to the trash"), so maybe some messages were flagged before they were deleted and thus still show up? Try View > Show deleted messages. If that option is greyed-out, then I guess that is not the cause.



[ Reply to This | # ]
Mismatched Subject lines anyone?
Authored by: jetcowbob on Aug 30, '05 02:03:40AM

I have not really paid much attention to the counts, however, I have had a much worse problem than even those ghosts with my Mail.app.....

I have an IMAP account and i have a bunch of folders outside the inbox. There are several rules to automagically sort mail into these folders.. find and dandy.. all the mail is sorted properly... but any folder besides the inbox, even smart folders, have a problem where opening a message of one subject pulls up the message window with the correct subject, but the body of a different email.

At first i though that this was a some simple offset, but it seems to be completely random. rebuilding doesnt seem to help. one time it will open a message 3 below the one i clicked, other times it will be 10 below or 2 above.. I have no idea how to fix this other than to use everything out of the Inbox... which is defeating the whole purpose of what i want to do, and causes space issues etc. on our servers....

Any suggestions?



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: Cameroon on Aug 30, '05 07:39:39AM

What I find especially interesting is that I have smart folders that do exactly what the hint claims is broken.

That is, I have a folder that shows Unread messages only and so it does, and with proper message counts. I also have it show Flagged messages, and again it does so correctly.

While I certainly don't claim that maybe for some people something is broken or malfunctioning, but "it works for me."

Perhaps it's due to having done a clean installation, rather than an upgrade. Although I did bring over all my old mail, so that seems less likely to be the reason.



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: moritzh on Aug 30, '05 01:32:36PM

Indeed not very many people have the same problem I had and it seems to work just well for most, else I would have found a solution more quickly with google I think...

I, too, did a fresh install of Tiger and Mail, by the way. I ecreated all my accounts and settings from scratch and then manually imported those of my old mailboxes I still needed. I thought that was better and safer than upgrading an existings 10.3 version of ~/Library/Mail... I really have no idea why the problem occured.



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: graemeaustin on Sep 19, '05 10:41:17AM

I have had a similar incorrect message count problem and fixed it by going throuhg every single mailbox and rebuilding them. It would appear as though the index files for some of the mailboxes were not up-to-date. This suggestion appeared in the www.apple.com/support discussion forum.

However, I am now in the process of figuring out a way to get the computer to automatically rebuild all the mailboxes using applescript (and GUI Scripting).



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: CPruitt on May 09, '06 03:03:26PM

I'd recommend being careful when you try this. I tried it & afterwards all of my email only had the following body:

"The message from <(email address)> <(email address)> concerning "(subject)" has not been downloaded from the server. You need to take this account online in order to download it."

I can restore it from my backup but I didn't notice soon enough so all my recent email will be deleted. Lovely... :-(



[ Reply to This | # ]
Fix message counts and 'ghosts' in Mail's smart mailboxes
Authored by: morgs on Nov 17, '07 08:09:32PM

Thanks heaps for this hint.

After upgrading to Leopard I have all of these ghost emails that kept appearing every time I did a search. You have helped me fix a very big frustration of mine.



[ Reply to This | # ]