Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Filter by
Sorted by
Tagged with
3578votes
31answers
456kviews

Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
user avatar
289votes
6answers
58kviews

Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
user avatar
  • 5,043
29votes
7answers
115kviews

Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate? Do you have a favourite example or anecdote which seems to get ...
414votes
11answers
95kviews

How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x ...
user avatar
  • 39.4k
75votes
6answers
39kviews

Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5,...
user avatar
  • 17.4k
186votes
21answers
177kviews

What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run ...
user avatar
  • 10.3k
92votes
8answers
53kviews

PHP - Floating Number Precision [duplicate]

$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
user avatar
  • 1,255
57votes
5answers
67kviews

Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples. ...
user avatar
56votes
2answers
51kviews

What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers? The alternate is loading an external library like <script type="text/javascript" src="the_bignum_library.js"></script&...
118votes
5answers
239kviews

Dealing with float precision in Javascript [duplicate]

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying ...
user avatar
  • 30.2k
33votes
12answers
52kviews

How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( ...
user avatar
  • 22.3k
13votes
0answers
2kviews

ray and ellipsoid intersection accuracy improvement

I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid. This is the core ...
user avatar
  • 44.6k
9votes
8answers
16kviews

Dealing with accuracy problems in floating-point numbers

I was wondering if there is a way of overcoming an accuracy problem that seems to be the result of my machine's internal representation of floating-point numbers: For the sake of clarity the problem ...
user avatar
  • 3,835
49votes
17answers
77kviews

How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log(...
user avatar
  • 25.8k
11votes
4answers
7kviews

Python float - str - float weirdness

>>> float(str(0.65000000000000002)) 0.65000000000000002 >>> float(str(0.47000000000000003)) 0.46999999999999997 ??? What is going on here? How do I convert 0....
user avatar
  • 2,000
37votes
9answers
44kviews

What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...
user avatar
10votes
5answers
54kviews

C++ floating point precision [duplicate]

Possible Duplicate: Floating point inaccuracy examples double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = ...
user avatar
  • 870
112votes
31answers
173kviews

Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I ...
user avatar
  • 1,527
58votes
6answers
38kviews

negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True....
user avatar
  • 44.3k
10votes
9answers
28kviews

Manipulating and comparing floating points in java

In Java the floating point arithmetic is not represented precisely. For example this java code: float a = 1.2; float b= 3.0; float c = a * b; if(c == 3.6){ System.out.println("c is 3.6"); } ...
user avatar
  • 419
162votes
4answers
15kviews

Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values ...
user avatar
  • 6,336
6votes
2answers
19kviews

floating point issue in R? [duplicate]

Possible Duplicate: Why are these numbers not equal? The below expression, which evaluates to 0.1, is considered larger than 0.1. > round(1740/600,0) - 1740/600 [1] 0.1 > (round(1740/600,...
user avatar
  • 19.7k
14votes
7answers
40kviews

Floating point equality and tolerances

Comparing two floating point number by something like a_float == b_float is looking for trouble since a_float / 3.0 * 3.0 might not be equal to a_float due to round off error. What one normally does ...
user avatar
  • 26.2k
36votes
8answers
52kviews

How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one: if (fabs(prev.min[i] - cur->min[i]) < 0.000001 && fabs(...
user avatar
  • 5,150
37votes
10answers
33kviews

Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1.0 ...
user avatar
10votes
4answers
10kviews

python floating number [duplicate]

i am kind of confused why python add some additional decimal number in this case, please help to explain >>> mylist = ["list item 1", 2, 3.14] >>> print mylist ['list item 1', 2, 3....
user avatar
28votes
9answers
27kviews

Rounding Errors?

In my course, I am told: Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...
user avatar
0votes
2answers
399views

Floating point error in representation?

when i make this multiplication 0.94 * 8700 the output is 8177.999999999999 but it should have been 8178 i'm using java , but i don't think this error is related to a particular ...
user avatar
6votes
3answers
28kviews

Floating Point Limitations [duplicate]

My code: a = '2.3' I wanted to display a as a floating point value. Since a is a string, I tried: float(a) The result I got was : 2.2999999999999998 I want a solution for this problem. Please, ...
user avatar
  • 143k
1vote
2answers
1kviews

Rounding oddity - what is special about "100"? [duplicate]

Does anyone have an explanation for this strange rounding in haskell (GHCi, version 7.2.1). Everything seems fine unless I multiply with 100. *Main> 1.1 1.1 *Main> 1.1 *10 11.0 *Main> 1.1 ...
user avatar
69votes
6answers
7kviews

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and ...
user avatar
  • 1,025
3votes
1answer
335views

Floating point less-than-equal comparisons after addition and substraction

Is there a "best practice" for less-than-equal comparisons with floating point number after a series of floating-point arithmetic operations? I have the following example in R (although the question ...
user avatar
23votes
1answer
19kviews

Precision lost while using read_csv in pandas

I have files of the below format in a text file which I am trying to read into a pandas dataframe. 895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0....
user avatar
3votes
6answers
19kviews

Decimal place issues with floats and decimal.Decimal

I seem to be losing a lot of precision with floats. For example I need to solve a matrix: 4.0x -2.0y 1.0z =11.0 1.0x +5.0y -3.0z =-6.0 2.0x +2.0y +5.0z =7.0 This is the code I use to import the ...
user avatar
  • 531
1vote
1answer
1kviews

Avg of float inconsistency

The select returns right at 23,000 rows The except will return between 60 to 200 rows (and not the same rows) The except should return 0 as it is select a except select a PK: [docSVenum1].[enumID]...
user avatar
  • 43.4k
48votes
9answers
48kviews

How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation). $ echo "3.1e1*2" | bc -l (standard_in) 1: parse error but I need to use it to handle a few records that are ...
user avatar
19votes
14answers
54kviews

Truncate float numbers with PHP

When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...
user avatar
5votes
10answers
2kviews

1.265 * 10000 = 126499.99999999999? [duplicate]

When I multiply 1.265 by 10000 , I get 126499.99999999999 when using Javascript. Why is this so?
user avatar
  • 209
31votes
2answers
6kviews

mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...
user avatar
  • 313
4votes
9answers
11kviews

Integers and float precision

This is more of a numerical analysis rather than programming question, but I suppose some of you will be able to answer it. In the sum two floats, is there any precision lost? Why? In the sum of a ...
user avatar
  • 19k
8votes
1answer
290views

C fundamentals: double variable not equal to double expression?

I am working with an array of doubles called indata (in the heap, allocated with malloc), and a local double called sum. I wrote two different functions to compare values in indata, and obtained ...
user avatar
5votes
2answers
5kviews

IEEE-754 floating-point precision: How much error is allowed?

I'm working on porting the sqrt function (for 64-bit doubles) from fdlibm to a model-checker tool I'm using at the moment (cbmc). As part of my doings, I read a lot about the ieee-754 standard, but I ...
user avatar
  • 29.8k
0votes
4answers
2kviews

Why don't I get zero when I subtract the same floating point number from itself in Perl? [duplicate]

Possible Duplicates: Why is floating point arithmetic in C# imprecise? Why does ghci say that 1.1 + 1.1 + 1.1 > 3.3 is True? #!/usr/bin/perl $l1 = "0+0.590580+0.583742+0.579787+0.564928+0....
user avatar
75votes
3answers
18kviews

Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include <cstdio> int main() { printf("%d\n", (0.1 + 0.2 != 0.3))...
user avatar
  • 11.1k
22votes
5answers
17kviews

Fast Exp calculation: possible to improve accuracy without losing too much performance?

I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#: public static double Exp(double x) { var tmp = (long)(...
user avatar
21votes
2answers
3kviews

Numpy dot too clever about symmetric multiplications

Anybody know about documentation for this behaviour? import numpy as np A = np.random.uniform(0,1,(10,5)) w = np.ones(5) Aw = A*w Sym1 = Aw.dot(Aw.T) Sym2 = (A*w).dot((A*w).T) diff = Sym1 - Sym2 ...
user avatar
  • 1,629
8votes
10answers
16kviews

For-loop in C++ using double breaking out one step early, boundary value not reached

I have a simple C++ program compiled using gcc 4.2.4 on 32-bit Ubuntu 8.04. It has a for-loop in which a double variable is incremented from zero to one with a certain step size. When the step size is ...
user avatar
31votes
4answers
28kviews

How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like: String foo = String.format("%.*f\n", precision, ...
user avatar
19votes
2answers
12kviews

Does "epsilon" really guarantees anything in floating-point computations?

To make the problem short let's say I want to compute the expression a / (b - c) on floats. To make sure the result is meaningful, I can check if b and c are in equal: float EPS = std::numeric_limits&...
user avatar
18votes
4answers
4kviews

Precise sum of floating point numbers

I am aware of a similar question, but I want to ask for people opinion on my algorithm to sum floating point numbers as accurately as possible with practical costs. Here is my first solution: put ...
user avatar

15 30 50 per page
1
2 3 4 5 6