# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

1,276
questions

3578votes

31answers

456kviews

### Is floating point math broken?

Consider the following code:
0.1 + 0.2 == 0.3 -> false
0.1 + 0.2 -> 0.30000000000000004
Why do these inaccuracies happen?

414votes

11answers

95kviews

### How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system.
But every time I want to check if for example frame.origin.x is 0 it makes me feel sick:
if (theView.frame.origin.x ...

289votes

6answers

58kviews

### Why are these numbers not equal?

The following code is obviously wrong. What's the problem?
i <- 0.1
i <- i + 0.05
i
## [1] 0.15
if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15")
## i does not equal 0.15

186votes

21answers

177kviews

### What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java.
However, when I type this code:
if(sectionID == currentSectionID)
into my editor and run ...

162votes

4answers

15kviews

### Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?).
But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when
both values ...

131votes

12answers

11kviews

### Is it possible to get 0 by subtracting two unequal floating point numbers?

Is it possible to get division by 0 (or infinity) in the following example?
public double calculation(double a, double b)
{
if (a == b)
{
return 0;
}
else
{
...

118votes

5answers

239kviews

### Dealing with float precision in Javascript [duplicate]

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string.
How do I get around the annoying ...

112votes

31answers

173kviews

### Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this:
5.467 -> 5.46
985.943 -> 985.94
toFixed(2) does just about the right thing but it rounds off the value. I ...

94votes

4answers

6kviews

### What's the closest double to 1.0, that isn't 1.0?

Is there a way to programmatically get the double that is closest to 1.0, but isn't actually 1.0?
One hacky way to do this would be to memcpy the double to a same-sized integer, and then subtract one....

92votes

8answers

53kviews

### PHP - Floating Number Precision [duplicate]

$a = '35';
$b = '-34.99';
echo ($a + $b);
Results in 0.009999999999998
What is up with that? I wondered why my program kept reporting odd results.
Why doesn't PHP return the expected 0.01?

75votes

6answers

39kviews

### Large numbers erroneously rounded in JavaScript

See this code:
var jsonString = '{"id":714341252076979033,"type":"FUZZY"}';
var jsonParsed = JSON.parse(jsonString);
console.log(jsonString, jsonParsed);
When I see my console in Firefox 3.5,...

75votes

3answers

18kviews

### Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true
is my favorite check that a language uses native floating point arithmetic.
C++
#include <cstdio>
int main()
{
printf("%d\n", (0.1 + 0.2 != 0.3))...

69votes

6answers

7kviews

### (.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==.
I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true).
I get that == is reference and ...

58votes

6answers

38kviews

### negative zero in python

I encountered negative zero in output from python; it's created for example as follows:
k = 0.0
print(-k)
The output will be -0.0.
However, when I compare the -k to 0.0 for equality, it yields True....

57votes

4answers

5kviews

### Inaccuracy of decimal in .NET

Yesterday during debugging something strange happened to me and I can't really explain it:
So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but shouldn't ...

57votes

5answers

67kviews

### Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples.
...

56votes

2answers

51kviews

### What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers?
The alternate is loading an external library like
<script type="text/javascript" src="the_bignum_library.js"></script&...

49votes

17answers

77kviews

### How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ).
Code (C++):
int targetlevel = int(log(...

48votes

9answers

48kviews

### How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation).
$ echo "3.1e1*2" | bc -l
(standard_in) 1: parse error
but I need to use it to handle a few records that are ...

44votes

2answers

28kviews

### Why float.Epsilon and not zero?

In the following code, why is there a comparison against float.Epsilon and not 0?
// Coroutine to move elements
protected IEnumerator SmoothMovement (Vector3 end)
{
// Distance computation
...

37votes

10answers

33kviews

### Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue...
let x:Double = 1234.5678
let n1:Double = x % 1.0 ...

37votes

9answers

44kviews

### What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...

37votes

1answer

9kviews

### When to use `std::hypot(x,y)` over `std::sqrt(x*x + y*y)`

The documentation of std::hypot says that:
Computes the square root of the sum of the squares of x and y, without undue overflow or underflow at intermediate stages of the computation.
I struggle ...

36votes

8answers

52kviews

### How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one:
if (fabs(prev.min[i] - cur->min[i]) < 0.000001 &&
fabs(...

34votes

3answers

2kviews

### Is there any accuracy gain when casting to double and back when doing float division?

What is the difference between two following?
float f1 = some_number;
float f2 = some_near_zero_number;
float result;
result = f1 / f2;
and:
float f1 = some_number;
float f2 = ...

33votes

12answers

52kviews

### How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute:
for ( float value = 0.0f; value < 1.0f; value += 0.1f )
System.out.println( ...

32votes

4answers

3kviews

### Is integer division always equal to the floor of regular division?

For large quotients, integer division (//) doesn't seem to be necessarily equal to the floor of regular division (math.floor(a/b)).
According to Python docs (https://docs.python.org/3/reference/...

31votes

2answers

6kviews

### mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...

31votes

4answers

28kviews

### How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like:
String foo = String.format("%.*f\n", precision, ...

29votes

7answers

115kviews

### Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate?
Do you have a favourite example or anecdote which seems to get ...

28votes

9answers

27kviews

### Rounding Errors?

In my course, I am told:
Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...

27votes

11answers

21kviews

### Latitude/Longitude storage and compression in C

Does anyone know the most efficient representation for lat/long coordinates? Accuracy level should be enough for consumer GPS devices.
Most implementations seem to use double for each unit, but I'm ...

23votes

3answers

36kviews

### How to safely floor or ceil a CGFloat to int?

I often need to floor or ceil a CGFloat to an int, for calculation of an array index.
The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with ...

23votes

1answer

19kviews

### Precision lost while using read_csv in pandas

I have files of the below format in a text file which I am trying to read into a pandas dataframe.
895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0....

23votes

1answer

10kviews

### If dealing with money in a float is bad, then why does money_format() do it?

I've been waffling on how to deal with currency display and math in PHP, and for a long time have been storing it in MySQL using the DECIMAL type, and using money_format() to format it for display on ...

22votes

5answers

17kviews

### Fast Exp calculation: possible to improve accuracy without losing too much performance?

I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#:
public static double Exp(double x)
{
var tmp = (long)(...

21votes

6answers

5kviews

### Is there a floating point value of x, for which x-x == 0 is false?

In most cases, I understand that a floating point comparison test should be implemented using over a range of values (abs(x-y) < epsilon), but does self subtraction imply that the result will be ...

21votes

2answers

3kviews

### Numpy dot too clever about symmetric multiplications

Anybody know about documentation for this behaviour?
import numpy as np
A = np.random.uniform(0,1,(10,5))
w = np.ones(5)
Aw = A*w
Sym1 = Aw.dot(Aw.T)
Sym2 = (A*w).dot((A*w).T)
diff = Sym1 - Sym2
...

21votes

6answers

618views

### Efficiently computing (a - K) / (a + K) with improved accuracy

In various contexts, for example for the argument reduction for mathematical functions, one needs to compute (a - K) / (a + K), where a is a positive variable argument and K is a constant. In many ...

19votes

14answers

54kviews

### Truncate float numbers with PHP

When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...

19votes

2answers

12kviews

### Does "epsilon" really guarantees anything in floating-point computations?

To make the problem short let's say I want to compute the expression a / (b - c) on floats.
To make sure the result is meaningful, I can check if b and c are in equal:
float EPS = std::numeric_limits&...

19votes

8answers

3kviews

### printing float, preserving precision

I am writing a program that prints floating point literals to be used inside another program.
How many digits do I need to print in order to preserve the precision of the original float?
Since a ...

19votes

2answers

6kviews

### PHP/MySQL: Best money operations/storing practices?

So, I am planning to make an application (PHP/MySQL) which deals a lot with money, and I am thinking about how to store and operate with the money, referring to PHP float data type and MySQL decimal.
...

18votes

7answers

930views

### Why does (int)(33.46639 * 1000000) return 33466389?

(int)(33.46639 * 1000000) returns 33466389
Why does this happen?

18votes

4answers

4kviews

### Precise sum of floating point numbers

I am aware of a similar question, but I want to ask for people opinion on my algorithm to sum floating point numbers as accurately as possible with practical costs.
Here is my first solution:
put ...

18votes

2answers

3kviews

### Java vs C floating point: "x * x" differs from "pow(x,2)"?

Why is this true? Java appears to produce a result with a small discrepancy when multiplying two floats compared to C and even the Java Math.pow method.
Java:
float a = 0.88276923;
double b = a * a;...

18votes

2answers

748views

### Invertability of IEEE 754 floating-point division

What is the invertability of the IEEE 754 floating-point division? I mean is it guaranteed by the standard that if double y = 1.0 / x then x == 1.0 / y, i.e. x can be restored precisely bit by bit?
...

17votes

5answers

1kviews

### c++ floating point precision loss: 3015/0.00025298219406977296

The problem.
Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu.
Code:
double a = 3015.0;
double b = 0.00025298219406977296;
//*((unsigned __int64*)(&a)) == ...

17votes

4answers

5kviews

### Does casting `std::floor()` and `std::ceil()` to integer type always give the correct result?

I am being paranoid that one of these functions may give an incorrect result like this:
std::floor(2000.0 / 1000.0) --> std::floor(1.999999999999) --> 1
or
std::ceil(18 / 3) --> std::ceil(6....

17votes

5answers

6kviews

### Why do I need 17 significant digits (and not 16) to represent a double?

Can someone give me an example of a floating point number (double precision), that needs more than 16 significant decimal digits to represent it?
I have found in this thread that sometimes you need up ...