# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

277 questions
Filter by
Sorted by
Tagged with
456kviews

### Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
58kviews

### Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
• 5,043
115kviews

### Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate? Do you have a favourite example or anecdote which seems to get ...
95kviews

### How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x ...
• 39.4k
39kviews

### Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5,...
• 17.4k
177kviews

### What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run ...
• 10.3k
53kviews

### PHP - Floating Number Precision [duplicate]

\$a = '35'; \$b = '-34.99'; echo (\$a + \$b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
• 1,255
67kviews

### Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples. ...
• 15.6k
51kviews

### What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers? The alternate is loading an external library like <script type="text/javascript" src="the_bignum_library.js"></script&...
239kviews

### Dealing with float precision in Javascript [duplicate]

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying ...
• 30.2k
52kviews

### How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( ...
• 22.3k
2kviews

### ray and ellipsoid intersection accuracy improvement

I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid. This is the core ...
• 44.6k
16kviews

### Dealing with accuracy problems in floating-point numbers

I was wondering if there is a way of overcoming an accuracy problem that seems to be the result of my machine's internal representation of floating-point numbers: For the sake of clarity the problem ...
• 3,835
77kviews

### How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log(...
• 25.8k
7kviews

### Python float - str - float weirdness

>>> float(str(0.65000000000000002)) 0.65000000000000002 >>> float(str(0.47000000000000003)) 0.46999999999999997 ??? What is going on here? How do I convert 0....
• 2,000
44kviews

### What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...
• 41k
54kviews

### C++ floating point precision [duplicate]

Possible Duplicate: Floating point inaccuracy examples double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = ...
• 870
173kviews

### Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I ...
• 1,527
38kviews

### negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True....
• 44.3k
28kviews

### Manipulating and comparing floating points in java

In Java the floating point arithmetic is not represented precisely. For example this java code: float a = 1.2; float b= 3.0; float c = a * b; if(c == 3.6){ System.out.println("c is 3.6"); } ...
• 419
15kviews

### Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values ...
• 6,336
19kviews

### floating point issue in R? [duplicate]

Possible Duplicate: Why are these numbers not equal? The below expression, which evaluates to 0.1, is considered larger than 0.1. > round(1740/600,0) - 1740/600 [1] 0.1 > (round(1740/600,...
• 19.7k
40kviews

### Floating point equality and tolerances

Comparing two floating point number by something like a_float == b_float is looking for trouble since a_float / 3.0 * 3.0 might not be equal to a_float due to round off error. What one normally does ...
• 26.2k
52kviews

### How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one: if (fabs(prev.min[i] - cur->min[i]) < 0.000001 && fabs(...
• 5,150
33kviews

### Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1.0 ...
• 459
10kviews

### python floating number [duplicate]

i am kind of confused why python add some additional decimal number in this case, please help to explain >>> mylist = ["list item 1", 2, 3.14] >>> print mylist ['list item 1', 2, 3....
• 111
27kviews

### Rounding Errors?

In my course, I am told: Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...
• 7,053
399views

### Floating point error in representation?

when i make this multiplication 0.94 * 8700 the output is 8177.999999999999 but it should have been 8178 i'm using java , but i don't think this error is related to a particular ...
28kviews

### Floating Point Limitations [duplicate]

My code: a = '2.3' I wanted to display a as a floating point value. Since a is a string, I tried: float(a) The result I got was : 2.2999999999999998 I want a solution for this problem. Please, ...
• 143k
1vote
1kviews

### Rounding oddity - what is special about "100"? [duplicate]

Does anyone have an explanation for this strange rounding in haskell (GHCi, version 7.2.1). Everything seems fine unless I multiply with 100. *Main> 1.1 1.1 *Main> 1.1 *10 11.0 *Main> 1.1 ...
• 4,971
7kviews

### (.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and ...
• 1,025
335views

### Floating point less-than-equal comparisons after addition and substraction

Is there a "best practice" for less-than-equal comparisons with floating point number after a series of floating-point arithmetic operations? I have the following example in R (although the question ...
19kviews

### Precision lost while using read_csv in pandas

I have files of the below format in a text file which I am trying to read into a pandas dataframe. 895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0....
19kviews

### Decimal place issues with floats and decimal.Decimal

I seem to be losing a lot of precision with floats. For example I need to solve a matrix: 4.0x -2.0y 1.0z =11.0 1.0x +5.0y -3.0z =-6.0 2.0x +2.0y +5.0z =7.0 This is the code I use to import the ...
• 531
1vote
1kviews

### Avg of float inconsistency

The select returns right at 23,000 rows The except will return between 60 to 200 rows (and not the same rows) The except should return 0 as it is select a except select a PK: [docSVenum1].[enumID]...
• 43.4k
48kviews

### How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation). \$ echo "3.1e1*2" | bc -l (standard_in) 1: parse error but I need to use it to handle a few records that are ...
• 3,694
54kviews

### Truncate float numbers with PHP

When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...
2kviews

### 1.265 * 10000 = 126499.99999999999? [duplicate]

When I multiply 1.265 by 10000 , I get 126499.99999999999 when using Javascript. Why is this so?
• 209
6kviews

### mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...
• 313
11kviews

### Integers and float precision

This is more of a numerical analysis rather than programming question, but I suppose some of you will be able to answer it. In the sum two floats, is there any precision lost? Why? In the sum of a ...
• 19k
290views

### C fundamentals: double variable not equal to double expression?

I am working with an array of doubles called indata (in the heap, allocated with malloc), and a local double called sum. I wrote two different functions to compare values in indata, and obtained ...
5kviews

### IEEE-754 floating-point precision: How much error is allowed?

I'm working on porting the sqrt function (for 64-bit doubles) from fdlibm to a model-checker tool I'm using at the moment (cbmc). As part of my doings, I read a lot about the ieee-754 standard, but I ...
• 29.8k
2kviews

### Why don't I get zero when I subtract the same floating point number from itself in Perl? [duplicate]

Possible Duplicates: Why is floating point arithmetic in C# imprecise? Why does ghci say that 1.1 + 1.1 + 1.1 > 3.3 is True? #!/usr/bin/perl \$l1 = "0+0.590580+0.583742+0.579787+0.564928+0....
• 47
18kviews

### Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include <cstdio> int main() { printf("%d\n", (0.1 + 0.2 != 0.3))...
• 11.1k
17kviews

### Fast Exp calculation: possible to improve accuracy without losing too much performance?

I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#: public static double Exp(double x) { var tmp = (long)(...
• 15.4k
3kviews

### Numpy dot too clever about symmetric multiplications

Anybody know about documentation for this behaviour? import numpy as np A = np.random.uniform(0,1,(10,5)) w = np.ones(5) Aw = A*w Sym1 = Aw.dot(Aw.T) Sym2 = (A*w).dot((A*w).T) diff = Sym1 - Sym2 ...
• 1,629
16kviews

### For-loop in C++ using double breaking out one step early, boundary value not reached

I have a simple C++ program compiled using gcc 4.2.4 on 32-bit Ubuntu 8.04. It has a for-loop in which a double variable is incremented from zero to one with a certain step size. When the step size is ...
28kviews

### How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like: String foo = String.format("%.*f\n", precision, ...
• 6,238
12kviews

### Does "epsilon" really guarantees anything in floating-point computations?

To make the problem short let's say I want to compute the expression a / (b - c) on floats. To make sure the result is meaningful, I can check if b and c are in equal: float EPS = std::numeric_limits&...
• 2,715