Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Regarding Calculator and low precision values Apps
The precision setting in Calculator.app is either broken or doesn't work according to Help. To demonstrate, choose View > Precision > 2 from the menu, then try the following:
10000 + 2500 =
My 10.4.3 PowerBook G4 returns 12,000, when the correct result is 12,500 (that's off by 4%). I have confirmed the problem on a Mac mini also running the latest versions from Software Update. Lacking a precision setting, the Calculator widget does not reproduce the issue. So if you're getting odd results from Calculator, check the Precision setting and make sure it's at least three or higher when working with whole numbers!

I wonder why software calculators are so often caught wrong with Math?

[robg adds: According to Calculator's Help, the only thing Precision should affect is the display of decimals: "Choose View > Precision and choose the number of decimal places you want displayed. The internally stored value is not rounded. If the displayed value shows fewer decimal places than you specified, the undisplayed decimal places are zeros."

Based on some experiments I ran, what I think is happening is this ... Calculator uses the Precision menu to control significant digits, not rounding. So 10,000 + 2,500 to two significant digits is indeed 12,000 (only the 1 and the 2 are 'significant,' since they're the first two digits). I tested using a variety of Precision settings, and my results were always correct based on significant digits. So I think this is actually a bug in Help, not a bug in Calculator, since Help refers to decimals, not significant digits.

In any event, you should be aware of how Calculator works with the Precision setting, just so you're not surprised by any results.]
    •    
  • Currently 3.80 / 5
  You rated: 4 / 5 (5 votes cast)
 
[7,170 views]  

Regarding Calculator and low precision values | 31 comments | Create New Account
Click here to return to the 'Regarding Calculator and low precision values' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Regarding Calculator and low precision values
Authored by: rawhead on Dec 12, '05 07:44:40AM

If it's calculating significant digits,

10,000 + 2,500 = 12,500

and since the first two digits (12) are significant, the number should be rounded up (since 12 is followed by a number larger than or equal to 5). Hence, the display should be

13,000

So, if the precision setting is not functioning, its not functioning because it's rounding down when it should round up.

---
All these moments will be lost in time
Like tears in rain.
Time to die.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: rawhead on Dec 12, '05 07:49:45AM

Yeah, that's the problem.

Try

10,000 + 2,501

The display becomes 13,000 (like it should, when calculating significant digits).

So, that's where the bug is. The cutoff line for displaying 13,000 should be 12,500, but its set at 12,501.

Also, try throwing in a decimal.

10,000 + 2500.1

This displays 12500.1 which is way too wacky (whatever comes after 5 shouldn't make a difference. The result should always be 13,000).

So, the help is right. The precision setting is really not functioning at all.

---
All these moments will be lost in time
Like tears in rain.
Time to die.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: dhayton on Dec 12, '05 08:14:16AM

Actually, the rules for rounding say that you round a number ending in 5 to the *even* number. So,

12,500 correctly rounds to 12,000 (if you want two significant figures).
11,500 would round to 12,000 (again, if you wan two significant figures).

Calculator.app appears to apply correctly the rounding rules.

Best,
darin



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: JonathanBoyd on Dec 12, '05 09:29:08AM

Having spent several years studying physics, I can reliably inform you that the proper scientific practice is to round 5s up every time. Always rounding to even numbers produces a bias towards them and brings in an additional inaccuracy.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: fstengel on Dec 12, '05 09:40:24AM

I wouldn't use "proper scientific practice", I would rather use "force of habit" or more bluntly "inertia". None of the various research/lecturing colleages (in Physic, Maths, Biology) whom, ages ago, I asked why one rounds up on a 5 has been able to give any other answer than "because everybody else does so"...



[ Reply to This | # ]
The "round to even" does NOT produce a bias
Authored by: Krioni on Dec 12, '05 10:06:07AM
"Having spent several years studying physics..."

Well, I have a degree in physics (Bachelor, and some graduate school work towards Physics PhD). Also, I know basic statistics.

"Always rounding to even numbers produces a bias towards them and brings in an additional inaccuracy."

The only way you can say that is if you know that your original numbers (to some chosen significance) tend to be odd. Do they? No, they do not. On average, you are just as likely to get even numbers as you are to get odd numbers for a given range of possibilities. Thus, if you then follow the rule to "round to nearest even for numbers EXACTLY between two digits" you will NOT have a bias. If you instead do as suggested here (and in elementary school) and round upwards, you will absolutely introduce a bias to larger values in your data.

For more, read the Wikipedia entry (look under Statistician's Method, which is what scientists who wish to have non-biased data would use): http://en.wikipedia.org/wiki/Rounding

(note the phrase "This method is sometimes known as "round to even" and is used in order to eliminate the bias that would come from always rounding a number ending in five up every time." in particular.)

[ Reply to This | # ]

The "round to even" does NOT produce a bias
Authored by: mikemccallum on Dec 12, '05 02:45:33PM

0 is a digit, as is 1,2,3,4,5,6,7,8,9. Count them. Yup, there is 10 of them. Rounding up on 5,6,7,8,9 gives 5 chances to round up. Truncating on 0,1,2,3,4 gives 5 chances to truncate. It is a lazy analysis which states that "rounding up on 5 introduces a bias".

Zero can be a certain, significant, or uncertain digit just as any of the other nine.

This is, of course, totally to the side of the issue with the calculator function.



[ Reply to This | # ]
The "round to even" does NOT produce a bias
Authored by: peterneillewis on Dec 12, '05 07:29:24PM

To say this is a lazy analysis while providing such a flawed analysis is impressive. If you sum the changes from 0-4 rounding down and 5-9 rounding up, you get 0 + 1 + 2 + 3 + 4 for rounding down (total 10) and 5 + 4 + 3 + 2 + 1 for rounding up (total 15). On the other hand, if you round to even, then the total changes for rounding down is 0 + 1 + 2 + 3 + 4 + (half of 5), and for rounding up is (half of 5) + 4 + 3 + 2 + 1, which you will note is 12.5 for both.

Rounding to even does generate a bias in the distribution towards even numbers, but for most purposes a slight bias towards even numbers is better than a slight bias upwards.



[ Reply to This | # ]
Round to even changes the distribution
Authored by: JonathanBoyd on Dec 12, '05 02:50:02PM

To be honest, I don't trust Wikipedia to be terribly accurate, so I went and tried a few more trustworthy mathematical sites and they seem to say the same thing. I must confess that I'm quite surprised by that. If you take a distribution of random numbers, it should be fairly flat:
f
|
|-------------
|
+------------n
But if you then start rounding them to the nearest even, the distribution changes, so that even numbers are more prevalent:
f
|
|__---__---
|
+------------n
I would have thought that that makes it less accurate and produces a bias by clumping results around even numbers.

I guess that we're looking at it with a different intention behind the numbers. I was thinking about patterns and distributions. If you're looking at adding a long chain of numbers, or calculating a mean, as others have suggested, I concede that round to even is probably going to be better. I probably should have objected to the change in distribution, rather than saying 'bias.'

Incidentally, I don't know if it was intention or not, but saying such phrases as 'Also, I know basic statistics,' 'in elementary school' and 'which is what scientists who wish to have non-biased data would use' comes across as a little condescending, which is rather unnecessary.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: dhirsch226 on Dec 12, '05 12:25:15PM

As I stated in another post for this article: Under what conditions would you conceivably care about a bias-to-even-numbers in a dataset? (I might; I deal with spatial statistics and ordering, but most people? I doubt it.)

On the other hand, there are lots of cases where you would care about the mean of a data set being artificially elevated as a result of rounding up all the time.

-Dave



[ Reply to This | # ]
That's news to me
Authored by: jecwobble on Dec 12, '05 09:42:12AM
I was about to respond just as Jonathan did, but while Googling for corroborative evidence, I found this page, which at the bottom has a rule that supports what Darin is stating.

[ Reply to This | # ]
Forgot to set HTML
Authored by: jecwobble on Dec 12, '05 09:47:13AM
I meant to give a link to this page

[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: jsuen on Dec 12, '05 08:25:54AM

What they teach kids in school is a lie. Rounding 5 up produces a bias, because 5 is exactly in the middle. What you should do is round 5 up half the time, and down the other half. IEEE 754 floating point defines a round to even to do this exactly, so Calculator is correct.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: pub3abn on Dec 12, '05 08:48:09AM

But doesn't that produce a bias towards even numbers?



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: jsuen on Dec 12, '05 09:59:36AM

Whether a number is odd or even is usually not as important as the magnitude of the error. For example, 5.5+3.5+2.5=11. If we round 5 up, we get 6+4+3=13. If we round to even, we get 6+4+2=12. The effect only appears on longer chains of numbers, where the bias of 5 adds up.

Round to even is done on any modern processor. The IRS doesn't care whether your income is odd or even, but they do care if you're $1,000 off.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: dhirsch226 on Dec 12, '05 12:20:19PM

Under what conditions would you conceivably care about a bias towards even numbers? The main issue is usually whether some measure, the mean for example, becomes biased by rounding. That's what round-to-even is for.
You can test this yourself in Excel (or write a simple program):

Cell A1 - Random number. "=RAND()"
Cell B1 - Truncated to 3 decimal places. "=TRUNC(A1,3)"
Cell C1 - Gives first two decimal places as an integer. "=TRUNC(B1*100)"
Cell D1 - Gives third decimal place as an integer. "=10*(B1*100 - C1)"
Cell E1 - Gives the regular rounded value (up on 5). Note that we do this by hand so we don't have to wonder what Excel is doing under the hood. "=IF(D1<5, C1,C1+1)/100"
Cell F1 - Gives a value in which we round down on 5 all the time. "=IF(D1<=5, C1,C1+1)/100"
Cell G1 - Gives a value in which we round to even. "=IF(ISEVEN(C1),F1,E1)"

If you copy this row down on a large number of rows (say, highlight A1 to G10000 and select Edit>Fill>Down), then calculate the averages of each column, you will find that the average for rounding up (Column E) is elevated relative to the average of the truncated values (Column B), while the average for the Round-to-Even values (Column G) is approximately equal to the original values.

Hope this illuminates things. It's always best to test these things yourself.
-Dave



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: JonathanBoyd on Dec 12, '05 03:19:31PM

The bias to even numbers would change the shape of the distribution.

I concede that rounding up does increase the mean. However, it only increases the magnitude of the mean. As numbers are shifted away from zero by rounding up, things should balance out when there are both positive and negative numbers.

The rounding to even method would seem to give a better mean where odd and even numbers are of the same sign. Rounding away from zero gives a better mean where odd and even numbers are of opposing signs. Admittedly, the former is a more likely set of data.

Either way, there are inaccuracies, so I'm sure we can all agree that, the error for any result should be given. Especially if using Apple's calculator.



[ Reply to This | # ]
The distribution only matters if
Authored by: porkchop_d_clown on Dec 12, '05 09:40:49PM
you're doing statistics. If you're doing any other branch of math you want to minimize the total error - the distribution is irrelevant.
Rounding towards even minimizes the total error and is the mechanism used internally by most math chips, IIRC.
Here's a link that refers to both methods as "equally valid".

---
Everyone loves a clown, but no one will lend him money!

[ Reply to This | # ]

Regarding Calculator and low precision values
Authored by: Baggins on Dec 12, '05 08:57:44AM

Sorry to correct you, but you are wrong.

By definition, the LAST digit in the significant figure is the uncertain figure.

So, 13,000 in significant figures simply means that we know the 1 for certain, and know that the 3 is approximate. In other words, the value could be anywhere from 12,000 to 14,000, so you can see that no bias is introduced in rounding.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: jsuen on Dec 12, '05 09:56:53AM

No, thats wrong. If 2 figures are significant, those two figures are exact.

Check out rule 1:
http://dbhs.wvusd.k12.ca.us/webdocs/SigFigs/SigFigRules.html

The site also describes round-to-even.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: Baggins on Dec 12, '05 10:56:55AM

I'm sorry, but the site you linked is incomplete.

Significant figures are used in Physics and Chemistry to specify the accuracy of a measurement. The last digit that is significant is always an approximation or the digit at which error creeps into the measurement.

In strict notation, the significant digits are always followed by an uncertainty value.

For example, 1.234 +/-.002 has four significant figures with an uncertainty of 2 in the last digit. This means the value COULD be anywhere from 1.232 to 1.236. If no error value is listed, then it is assumed the entire digit could be incorrect, meaning the value in our example could be anywhere from 1.230 to 1.239.

A classic example of this at work is when you measure something with a ruler. Assume the ruler is marked to a tenth of an inch. Your significant digits for any measure would be to the hundredth, with the hundredth being an approximation (as judged by your eyeball).

If you were being very precise, you would then add a margin of error and a confidence level to that error. The error simply states you could be off by that amount, and the confidence level is simply the probability that you are right in saying you could be off by your error margin.

When you are performing math where significant digits are involved, the rounding rules are: 0-4 round down, 5-9 round up.

If you still doubt me, go grab a frehsman college text on Chemistry.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: dhayton on Dec 12, '05 08:50:00AM
In trying out a few calculations, Calculator.app does not correctly implement "precision" to mean significant figures either. For example, with precision set at 2:

10,000.00 + 2,500.00 = 12,500
(it should be 12,500.00 if proper significant figure rules are applied; it should be 12,000 if 2 sets the number of significant figures.)

5.25 * 3.00 = 15.75
(it should be 15.8, using correct significant figure and rounding rules; it should be 16 if the precision set the number of significant figures)

Further, if you set precision to 2 and type 15.75, the calculator converts the displayed value to 16. It does, however, return the correct value in the following calculation:
15.75 / 3.00 = 5.25

Sadly, this is just an artefact of the way it understands "precision". For the following operation returns the incorrect answer:
15.75 / 3 = 5.25
(it should be 5 if correct rules for significant figures were applied; it would be 5.3 if 2 set the value of significant figures displayed in the answer.)

Ultimately, however, rules for significant figures don't allow you to specify the number you would like. They are based on the precision of the values used in the calculation. Thus, it would be strange if the Calculator.app were using "Precision" to mean significant figures.

Best,
darin

[ Reply to This | # ]
Rounding Rules
Authored by: WaltFrench on Dec 12, '05 02:52:24PM
"Five rounds up" is not the only rounding rule -- and in my work it is seldom the best.

My concern is that the total (or average) of rounded numbers shouldn't be biased up or down from the originals. (The total of a column of numbers versus the total of the rounded numbers will be exactly equal only by 1-in-10 chance, but you don't want to do anything so that you expect the total to be consistently higher (or lower) than the originals.

To see the bias, think of rounding 1-digit numbers. Obviously, 12.0 "rounds" to 12 with no bias. Then "pair up" 12.1 with 11.9 -- the 0.1 error introduced by rounding one cancels the negative 0.1 of the other. Ditto 12.2 with 11.8, 12.3 with 11.7 and 12.4 with 11.6. But now... 12.5 with 11.5?

Yes! The point five is just as close to either integer, so you pick up sometimes and down sometimes (and therefore, round without making the total or expected result always bigger or smaller). The best-known way is to round to the nearest even number when you have a point 5. 12.5 rounds to 12, as does 11.5. 73.5 and 74.5 both round to 74.

If point 5 always rounds up, here's the bias: nine numbers out of ten have no net up- or down-bias, but one number out of 10 (the ones that end in point 5) rounds to a result that's 0.5 bigger than the source, for an expected increase of 0.05. "Round to nearest/even" introduces no bias.

For jumping more than one decimal at a time, the rule is that only point 5000000... gets the "to even" handling. 12.5002 rounds to 13. 12.4998 rounds down to 12.

I used Excel to create 3000 random numbers between 0 and 100, then rounded them to one digit (with Excel's ROUND function, which rounds fives up). The total of the original and rounded column was close -- just a difference of 1.01 out of 151,829. That's what you expect when you're scrapping about 15 decimal places all at once -- the rounding rule hardly matters. But then I rounded the single-decimal-place numbers to zero places, and the total of those grew by 153.9 -- essentially, the expected 0.05 average bias per number.

Rounding to nearest/even works especially well when multiple rounding is applied. If you repeat "five rounds up," knocking off one digit at a time, the typical bias will be almost 0.1, twice the 0.05 I mentioned above.

BTW, the IEEE, which sets standards for math on CPU's etc., lists 4 rounding modes. Round to nearest/even is the default for X86 and PPC -- probably, for most applications. There are thorny issues about some types of binary calculations but my description is fine for by-hand work and should match what you see from most programs that (stupidly) try to outsmart their CPU's. (This is only one of a long list of Excel's dumb math handling.))

[ Reply to This | # ]
Rounding Rules
Authored by: JonathanBoyd on Dec 12, '05 03:23:06PM

Rounding away from zero, which is what rounding up usually is, should have no net effect on the mean if you have both positive and negative numbers. If you're only using one of those though, then rounding to even certainly seems to be a better way to get the mean.



[ Reply to This | # ]
Rounding Rules
Authored by: porkchop_d_clown on Dec 12, '05 09:49:03PM

I can't imagine many common problems where the median value is zero.

Still, for most purposes the difference between the two algorithms is immaterial. If you need more precision, you should use a larger word size. ;-)

I actually just went through this with my 3rd grade daughter - I learned the even/odd rule growing up, they just taught her the round-away-from-zero rule. This caused some confusion in homework checking.

---
Everyone loves a clown, but no one will lend him money!



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: ether on Dec 12, '05 09:58:17AM

I won't get into the religious wars about what precision means, or how to round, but I will note that calculator 4.0 under 10.4.2 on my machine doesn't behave anything like what's being described.

With precision 2:

10000 + 2500 = 12500
2.575 = 2.58
2.574999999999 = 2.57
5.25 * 3 = 15.75

With precision 1:

5.25 * 3 = 15.8 / 3 = 5.3 - .03 = 5.3 -.03 = 5.3 ...

.55 = .6
.5499999 = .5

In other words, precision n means round to n decimal places, and do so with the stored internal value, not just the displayed value.

.5 is rounded up.

All this is consistent with the help.

Based on the posts, it looks like calculator must have changed between 10.4.2 and 10.4.3 which is truly odd.



[ Reply to This | # ]
I definitely see the problem.
Authored by: porkchop_d_clown on Dec 12, '05 09:53:00PM

I hate to ask a stupid question - but are you sure you're using Apple's calculator app and not an add-on program?

---
Everyone loves a clown, but no one will lend him money!



[ Reply to This | # ]
Forget rounding, give us proper scientific notation!
Authored by: germ on Dec 12, '05 12:40:34PM

I cannot understand why after all these years OS X has been out, the Calculator is still an embarrassment. Have you tried using scientific notation?

I wish Apple would finally give us a decent Calculator by implementing HP-style SCI, ENG, and DEC display styles!!!



[ Reply to This | # ]
Forget rounding, give us proper scientific notation!
Authored by: kirkmc on Dec 13, '05 02:08:10AM

What I don't understand is why the precision is set to _anything_ by default. There should be a None or Off setting in that list. (The default setting is 11.)

---
Read my blog: Kirkville -- http://www.mcelhearn.com
Musings, Opinion and Miscellanea, on Macs, iPods and more



[ Reply to This | # ]
Forget rounding, give us proper scientific notation!
Authored by: phatmatt on Dec 14, '05 05:53:35PM

Have you tried using bc from the command line? It seems to be pretty complete calculator.



[ Reply to This | # ]
Regarding Calculator and low precision values
Authored by: timcrawf on Dec 13, '05 10:01:53AM

also not going to the rounding war
just stating that under 10.3.9 I do NOT see this behavior.
Calculator show 10000 + 2500 = 12500
no matter what the precision is set to.
This is on a PowerMac G5 and a eMac

So, this appears to be a Tiger bug



[ Reply to This | # ]