Regarding Calculator and low precision values
Dec 12, '05 06:22:00AM
Contributed by: Anonymous
The precision setting in Calculator.app is either broken or doesn't work according to Help. To demonstrate, choose View > Precision > 2 from the menu, then try the following:
10000 + 2500 =
My 10.4.3 PowerBook G4 returns 12,000, when the correct result is 12,500 (that's off by 4%). I have confirmed the problem on a Mac mini also running the latest versions from Software Update. Lacking a precision setting, the Calculator widget does not reproduce the issue. So if you're getting odd results from Calculator, check the Precision setting and make sure it's at least three or higher when working with whole numbers!
I wonder why software calculators are so often caught wrong with Math?
[robg adds: According to Calculator's Help, the only thing Precision should affect is the display of decimals: "Choose View > Precision and choose the number of decimal places you want displayed. The internally stored value is not rounded. If the displayed value shows fewer decimal places than you specified, the undisplayed decimal places are zeros."
Based on some experiments I ran, what I think is happening is this ... Calculator uses the Precision menu to control significant digits, not rounding. So 10,000 + 2,500 to two significant digits is indeed 12,000 (only the 1 and the 2 are 'significant,' since they're the first two digits). I tested using a variety of Precision settings, and my results were always correct based on significant digits. So I think this is actually a bug in Help, not a bug in Calculator, since Help refers to decimals, not significant digits.
In any event, you should be aware of how Calculator works with the Precision setting, just so you're not surprised by any results.]
Comments (31)
Mac OS X Hints
http://hints.macworld.com/article.php?story=20051205144655847