Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!


Click here to return to the 'Regarding Calculator and low precision values' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Regarding Calculator and low precision values
Authored by: dhayton on Dec 12, '05 08:50:00AM
In trying out a few calculations, Calculator.app does not correctly implement "precision" to mean significant figures either. For example, with precision set at 2:

10,000.00 + 2,500.00 = 12,500
(it should be 12,500.00 if proper significant figure rules are applied; it should be 12,000 if 2 sets the number of significant figures.)

5.25 * 3.00 = 15.75
(it should be 15.8, using correct significant figure and rounding rules; it should be 16 if the precision set the number of significant figures)

Further, if you set precision to 2 and type 15.75, the calculator converts the displayed value to 16. It does, however, return the correct value in the following calculation:
15.75 / 3.00 = 5.25

Sadly, this is just an artefact of the way it understands "precision". For the following operation returns the incorrect answer:
15.75 / 3 = 5.25
(it should be 5 if correct rules for significant figures were applied; it would be 5.3 if 2 set the value of significant figures displayed in the answer.)

Ultimately, however, rules for significant figures don't allow you to specify the number you would like. They are based on the precision of the values used in the calculation. Thus, it would be strange if the Calculator.app were using "Precision" to mean significant figures.

Best,
darin

[ Reply to This | # ]