FB II Compiler







Disk I/O














Made with FB


Round off numbers

I recently added new code to my 100,000 line accounting program. In this new part I subtract one variable from another and display the result in an edit field. The result of subtracting variable values 252.26 from 1000.25 is displayed as 747.889999999 rather than the correct answer of 747.89. As a test, I hard coded (no variables, just numbers) the same equation for display in a test window and had the same result. FB does this same thing for some but not all number combinations. For example the equation 1000.25-252 displayed a correct result. I am not sure if this problem has existing for a long time and I just never noticed it because of program rounding procedures.

Any ideas on what could cause this?

Doug Stemen

What you are seeing is not a bug. Instead, it is an artifact of the way binary computers represent floating point values. Just as there are rational numbers (numbers that can be exactly represented as the ratio of 2 integers) that have infinitely repeating decimal expansions (like 1/3 = 0.333333...) there are binary numbers that possess the same characteristic. This is a normal occurrence in all binary floating point calculations.

There are, generally, 2 methods to adapt to the situation...

1) Round off your results. For example, 747.889999999 can be rounded to 2 decimal places by adding .005 and truncating to 2 decimal digits.

2) If you are always keeping your values accurate to 2 decimal places multiply them by 100 and keep them as integers. Thus 747.89 would be stored as 74789. Doing this means that you are keeping track of pennies instead of dollars but the arithmetic is integral and, therefore, more accurate. You can treat the least significant digits as digits to the right of a decimal point when you display results. Incidently, just about all banks and financial institutions do their calculations this way keeping track of 3 digits to the right of the decimal point (they count tenths of pennies as integers) This is how they don't lose any money due to round off errors.

Charlie Dickman

Some systems represent floating point values internally in BCD format rather than true binary format, precisely as a means of eliminating that rounding error in decimal representations. When stored as BCD values of sufficiently high precision, 1000.25 should be stored as _exactly_ 1000.25, and 252.26 as _exactly_ 252.26.

And that's what confuses me, because I know that FB II does store floating point numbers in BCD format, with a default precision of 6 digits. I can only conclude that FB must temporarily convert its FP numbers from BCD to true binary while doing calculations--which sort of defeats the whole purpose of using BCD in the first place.