www.delorie.com/gnu/docs/calc/calc_46.html search
GNU Emacs Calc 2.02 Manual

 [ < ] [ > ] [ << ] [ Up ] [ >> ] [Top] [Contents] [Index] [ ? ]

### 3.7.12 Arithmetic Tutorial Exercise 1

Dividing two integers that are larger than the current precision may give a floating-point result that is inaccurate even when rounded down to an integer. Consider 123456789 / 2 when the current precision is 6 digits. The true answer is 61728394.5, but with a precision of 6 this will be rounded to 12345700. / 2. = 61728500.. The result, when converted to an integer, will be off by 106.

Here are two solutions: Raise the precision enough that the floating-point round-off error is strictly to the right of the decimal point. Or, convert to fraction mode so that 123456789 / 2 produces the exact fraction 123456789:2, which can be rounded down by the F command without ever switching to floating-point format.

 webmaster   donations   bookstore delorie software   privacy Copyright © 2003   by The Free Software Foundation Updated Jun 2003