Questions tagged [floating-accuracy]
Concerning the accuracy of operations performed on floating point numbers.
277
questions
3578votes
31answers
456kviews
Is floating point math broken?
Consider the following code:
0.1 + 0.2 == 0.3 -> false
0.1 + 0.2 -> 0.30000000000000004
Why do these inaccuracies happen?
- 41k
289votes
6answers
58kviews
Why are these numbers not equal?
The following code is obviously wrong. What's the problem?
i <- 0.1
i <- i + 0.05
i
## [1] 0.15
if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15")
## i does not equal 0.15
- 5,043
29votes
7answers
115kviews
Floating point inaccuracy examples
How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate?
Do you have a favourite example or anecdote which seems to get ...
414votes
11answers
95kviews
How dangerous is it to compare floating point values?
I know UIKit uses CGFloat because of the resolution independent coordinate system.
But every time I want to check if for example frame.origin.x is 0 it makes me feel sick:
if (theView.frame.origin.x ...
- 39.4k
75votes
6answers
39kviews
Large numbers erroneously rounded in JavaScript
See this code:
var jsonString = '{"id":714341252076979033,"type":"FUZZY"}';
var jsonParsed = JSON.parse(jsonString);
console.log(jsonString, jsonParsed);
When I see my console in Firefox 3.5,...
- 17.4k
186votes
21answers
177kviews
What's wrong with using == to compare floats in Java?
According to this java.sun page == is the equality comparison operator for floating point numbers in Java.
However, when I type this code:
if(sectionID == currentSectionID)
into my editor and run ...
- 10.3k
92votes
8answers
53kviews
PHP - Floating Number Precision [duplicate]
$a = '35';
$b = '-34.99';
echo ($a + $b);
Results in 0.009999999999998
What is up with that? I wondered why my program kept reporting odd results.
Why doesn't PHP return the expected 0.01?
- 1,255
57votes
5answers
67kviews
Is floating point arbitrary precision available?
Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples.
...
- 15.6k
56votes
2answers
51kviews
What is the standard solution in JavaScript for handling big numbers (BigNum)?
Is there a bignum built into JavaScript or browsers?
The alternate is loading an external library like
<script type="text/javascript" src="the_bignum_library.js"></script&...
118votes
5answers
239kviews
Dealing with float precision in Javascript [duplicate]
I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string.
How do I get around the annoying ...
- 30.2k
33votes
12answers
52kviews
How to avoid floating point precision errors with floats or doubles in Java?
I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute:
for ( float value = 0.0f; value < 1.0f; value += 0.1f )
System.out.println( ...
- 22.3k
13votes
0answers
2kviews
ray and ellipsoid intersection accuracy improvement
I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid.
This is the core ...
- 44.6k
9votes
8answers
16kviews
Dealing with accuracy problems in floating-point numbers
I was wondering if there is a way of overcoming an accuracy problem that seems to be the result of my machine's internal representation of floating-point numbers:
For the sake of clarity the problem ...
- 3,835
49votes
17answers
77kviews
How to do an integer log2() in C++?
In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ).
Code (C++):
int targetlevel = int(log(...
- 25.8k
11votes
4answers
7kviews
Python float - str - float weirdness
>>> float(str(0.65000000000000002))
0.65000000000000002
>>> float(str(0.47000000000000003))
0.46999999999999997 ???
What is going on here?
How do I convert 0....
- 2,000
37votes
9answers
44kviews
What is a simple example of floating point/rounding error?
I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...
- 41k
10votes
5answers
54kviews
C++ floating point precision [duplicate]
Possible Duplicate:
Floating point inaccuracy examples
double a = 0.3;
std::cout.precision(20);
std::cout << a << std::endl;
result: 0.2999999999999999889
double a, b;
a = 0.3;
b = ...
- 870
112votes
31answers
173kviews
Truncate (not round off) decimal numbers in javascript
I am trying to truncate decimal numbers to decimal places. Something like this:
5.467 -> 5.46
985.943 -> 985.94
toFixed(2) does just about the right thing but it rounds off the value. I ...
- 1,527
58votes
6answers
38kviews
negative zero in python
I encountered negative zero in output from python; it's created for example as follows:
k = 0.0
print(-k)
The output will be -0.0.
However, when I compare the -k to 0.0 for equality, it yields True....
- 44.3k
10votes
9answers
28kviews
Manipulating and comparing floating points in java
In Java the floating point arithmetic is not represented precisely. For example this java code:
float a = 1.2;
float b= 3.0;
float c = a * b;
if(c == 3.6){
System.out.println("c is 3.6");
}
...
- 419
162votes
4answers
15kviews
Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?
I know that most decimals don't have an exact floating point representation (Is floating point math broken?).
But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when
both values ...
- 6,336
6votes
2answers
19kviews
floating point issue in R? [duplicate]
Possible Duplicate:
Why are these numbers not equal?
The below expression, which evaluates to 0.1, is considered larger than 0.1.
> round(1740/600,0) - 1740/600
[1] 0.1
> (round(1740/600,...
- 19.7k
14votes
7answers
40kviews
Floating point equality and tolerances
Comparing two floating point number by something like a_float == b_float is looking for trouble since a_float / 3.0 * 3.0 might not be equal to a_float due to round off error.
What one normally does ...
- 26.2k
36votes
8answers
52kviews
How to correctly and standardly compare floats?
Every time I start a new project and when I need to compare some float or double variables I write the code like this one:
if (fabs(prev.min[i] - cur->min[i]) < 0.000001 &&
fabs(...
- 5,150
37votes
10answers
33kviews
Getting the decimal part of a double in Swift
I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue...
let x:Double = 1234.5678
let n1:Double = x % 1.0 ...
- 459
10votes
4answers
10kviews
python floating number [duplicate]
i am kind of confused why python add some additional decimal number in this case, please help to explain
>>> mylist = ["list item 1", 2, 3.14]
>>> print mylist ['list item 1', 2, 3....
- 111
28votes
9answers
27kviews
Rounding Errors?
In my course, I am told:
Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...
- 7,053
0votes
2answers
399views
Floating point error in representation?
when i make this multiplication
0.94 * 8700
the output is
8177.999999999999
but it should have been
8178
i'm using java , but i don't think this error is related to a particular ...
6votes
3answers
28kviews
Floating Point Limitations [duplicate]
My code:
a = '2.3'
I wanted to display a as a floating point value.
Since a is a string, I tried:
float(a)
The result I got was :
2.2999999999999998
I want a solution for this problem. Please, ...
- 143k
1vote
2answers
1kviews
Rounding oddity - what is special about "100"? [duplicate]
Does anyone have an explanation for this strange rounding in haskell (GHCi, version 7.2.1). Everything seems fine unless I multiply with 100.
*Main> 1.1
1.1
*Main> 1.1 *10
11.0
*Main> 1.1 ...
- 4,971
69votes
6answers
7kviews
(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?
My question is not about floating precision. It is about why Equals() is different from ==.
I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true).
I get that == is reference and ...
- 1,025
3votes
1answer
335views
Floating point less-than-equal comparisons after addition and substraction
Is there a "best practice" for less-than-equal comparisons with floating point number after a series of floating-point arithmetic operations?
I have the following example in R (although the question ...
- 4,359
23votes
1answer
19kviews
Precision lost while using read_csv in pandas
I have files of the below format in a text file which I am trying to read into a pandas dataframe.
895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0....
user904976
3votes
6answers
19kviews
Decimal place issues with floats and decimal.Decimal
I seem to be losing a lot of precision with floats.
For example I need to solve a matrix:
4.0x -2.0y 1.0z =11.0
1.0x +5.0y -3.0z =-6.0
2.0x +2.0y +5.0z =7.0
This is the code I use to import the ...
- 531
1vote
1answer
1kviews
Avg of float inconsistency
The select returns right at 23,000 rows
The except will return between 60 to 200 rows (and not the same rows)
The except should return 0 as it is select a except select a
PK: [docSVenum1].[enumID]...
- 43.4k
48votes
9answers
48kviews
How to get bc to handle numbers in scientific (aka exponential) notation?
bc doesn't like numbers expressed in scientific notation (aka exponential notation).
$ echo "3.1e1*2" | bc -l
(standard_in) 1: parse error
but I need to use it to handle a few records that are ...
- 3,694
19votes
14answers
54kviews
Truncate float numbers with PHP
When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...
5votes
10answers
2kviews
1.265 * 10000 = 126499.99999999999? [duplicate]
When I multiply 1.265 by 10000 , I get 126499.99999999999 when using Javascript.
Why is this so?
- 209
31votes
2answers
6kviews
mean from pandas and numpy differ
I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...
- 313
4votes
9answers
11kviews
Integers and float precision
This is more of a numerical analysis rather than programming question, but I suppose some of you will be able to answer it.
In the sum two floats, is there any precision lost? Why?
In the sum of a ...
- 19k
8votes
1answer
290views
C fundamentals: double variable not equal to double expression?
I am working with an array of doubles called indata (in the heap, allocated with malloc), and a local double called sum.
I wrote two different functions to compare values in indata, and obtained ...
5votes
2answers
5kviews
IEEE-754 floating-point precision: How much error is allowed?
I'm working on porting the sqrt function (for 64-bit doubles) from fdlibm to a model-checker tool I'm using at the moment (cbmc).
As part of my doings, I read a lot about the ieee-754 standard, but I ...
- 29.8k
0votes
4answers
2kviews
Why don't I get zero when I subtract the same floating point number from itself in Perl? [duplicate]
Possible Duplicates:
Why is floating point arithmetic in C# imprecise?
Why does ghci say that 1.1 + 1.1 + 1.1 > 3.3 is True?
#!/usr/bin/perl
$l1 = "0+0.590580+0.583742+0.579787+0.564928+0....
- 47
75votes
3answers
18kviews
Why 0.1 + 0.2 == 0.3 in D?
assert(0.1 + 0.2 != 0.3); // shall be true
is my favorite check that a language uses native floating point arithmetic.
C++
#include <cstdio>
int main()
{
printf("%d\n", (0.1 + 0.2 != 0.3))...
- 11.1k
22votes
5answers
17kviews
Fast Exp calculation: possible to improve accuracy without losing too much performance?
I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#:
public static double Exp(double x)
{
var tmp = (long)(...
- 15.4k
21votes
2answers
3kviews
Numpy dot too clever about symmetric multiplications
Anybody know about documentation for this behaviour?
import numpy as np
A = np.random.uniform(0,1,(10,5))
w = np.ones(5)
Aw = A*w
Sym1 = Aw.dot(Aw.T)
Sym2 = (A*w).dot((A*w).T)
diff = Sym1 - Sym2
...
- 1,629
8votes
10answers
16kviews
For-loop in C++ using double breaking out one step early, boundary value not reached
I have a simple C++ program compiled using gcc 4.2.4 on 32-bit Ubuntu 8.04. It has a for-loop in which a double variable is incremented from zero to one with a certain step size. When the step size is ...
user157558
31votes
4answers
28kviews
How to Java String.format with a variable precision?
I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like:
String foo = String.format("%.*f\n", precision, ...
- 6,238
19votes
2answers
12kviews
Does "epsilon" really guarantees anything in floating-point computations?
To make the problem short let's say I want to compute the expression a / (b - c) on floats.
To make sure the result is meaningful, I can check if b and c are in equal:
float EPS = std::numeric_limits&...
- 2,715
18votes
4answers
4kviews
Precise sum of floating point numbers
I am aware of a similar question, but I want to ask for people opinion on my algorithm to sum floating point numbers as accurately as possible with practical costs.
Here is my first solution:
put ...
- 1,235