Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Regarding Calculator and low precision values' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Regarding Calculator and low precision values
Authored by: ether on Dec 12, '05 09:58:17AM

I won't get into the religious wars about what precision means, or how to round, but I will note that calculator 4.0 under 10.4.2 on my machine doesn't behave anything like what's being described.

With precision 2:

10000 + 2500 = 12500
2.575 = 2.58
2.574999999999 = 2.57
5.25 * 3 = 15.75

With precision 1:

5.25 * 3 = 15.8 / 3 = 5.3 - .03 = 5.3 -.03 = 5.3 ...

.55 = .6
.5499999 = .5

In other words, precision n means round to n decimal places, and do so with the stored internal value, not just the displayed value.

.5 is rounded up.

All this is consistent with the help.

Based on the posts, it looks like calculator must have changed between 10.4.2 and 10.4.3 which is truly odd.



[ Reply to This | # ]
I definitely see the problem.
Authored by: porkchop_d_clown on Dec 12, '05 09:53:00PM

I hate to ask a stupid question - but are you sure you're using Apple's calculator app and not an add-on program?

---
Everyone loves a clown, but no one will lend him money!



[ Reply to This | # ]