Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Filter by
Sorted by
Tagged with
3578votes
31answers
456kviews

Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
user avatar
414votes
11answers
95kviews

How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x ...
user avatar
  • 39.4k
289votes
6answers
58kviews

Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ## [1] 0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
user avatar
  • 5,043
186votes
21answers
177kviews

What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run ...
user avatar
  • 10.3k
162votes
4answers
15kviews

Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values ...
user avatar
  • 6,336
131votes
12answers
11kviews

Is it possible to get 0 by subtracting two unequal floating point numbers?

Is it possible to get division by 0 (or infinity) in the following example? public double calculation(double a, double b) { if (a == b) { return 0; } else { ...
user avatar
  • 19.4k
118votes
5answers
239kviews

Dealing with float precision in Javascript [duplicate]

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying ...
user avatar
  • 30.2k
112votes
31answers
173kviews

Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I ...
user avatar
  • 1,527
94votes
4answers
6kviews

What's the closest double to 1.0, that isn't 1.0?

Is there a way to programmatically get the double that is closest to 1.0, but isn't actually 1.0? One hacky way to do this would be to memcpy the double to a same-sized integer, and then subtract one....
user avatar
  • 1,834
92votes
8answers
53kviews

PHP - Floating Number Precision [duplicate]

$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
user avatar
  • 1,255
75votes
6answers
39kviews

Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5,...
user avatar
  • 17.4k
75votes
3answers
18kviews

Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include <cstdio> int main() { printf("%d\n", (0.1 + 0.2 != 0.3))...
user avatar
  • 11.1k
69votes
6answers
7kviews

(.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and ...
user avatar
  • 1,025
58votes
6answers
38kviews

negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True....
user avatar
  • 44.3k
57votes
4answers
5kviews

Inaccuracy of decimal in .NET

Yesterday during debugging something strange happened to me and I can't really explain it: So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but shouldn't ...
user avatar
57votes
5answers
67kviews

Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples. ...
user avatar
56votes
2answers
51kviews

What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers? The alternate is loading an external library like <script type="text/javascript" src="the_bignum_library.js"></script&...
49votes
17answers
77kviews

How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log(...
user avatar
  • 25.8k
48votes
9answers
48kviews

How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation). $ echo "3.1e1*2" | bc -l (standard_in) 1: parse error but I need to use it to handle a few records that are ...
user avatar
44votes
2answers
28kviews

Why float.Epsilon and not zero?

In the following code, why is there a comparison against float.Epsilon and not 0? // Coroutine to move elements protected IEnumerator SmoothMovement (Vector3 end) { // Distance computation ...
user avatar
  • 14.6k
37votes
10answers
33kviews

Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1.0 ...
user avatar
37votes
9answers
44kviews

What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...
user avatar
37votes
1answer
9kviews

When to use `std::hypot(x,y)` over `std::sqrt(x*x + y*y)`

The documentation of std::hypot says that: Computes the square root of the sum of the squares of x and y, without undue overflow or underflow at intermediate stages of the computation. I struggle ...
user avatar
  • 5,314
36votes
8answers
52kviews

How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one: if (fabs(prev.min[i] - cur->min[i]) < 0.000001 && fabs(...
user avatar
  • 5,150
34votes
3answers
2kviews

Is there any accuracy gain when casting to double and back when doing float division?

What is the difference between two following? float f1 = some_number; float f2 = some_near_zero_number; float result; result = f1 / f2; and: float f1 = some_number; float f2 = ...
user avatar
33votes
12answers
52kviews

How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( ...
user avatar
  • 22.3k
32votes
4answers
3kviews

Is integer division always equal to the floor of regular division?

For large quotients, integer division (//) doesn't seem to be necessarily equal to the floor of regular division (math.floor(a/b)). According to Python docs (https://docs.python.org/3/reference/...
user avatar
31votes
2answers
6kviews

mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...
user avatar
  • 313
31votes
4answers
28kviews

How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like: String foo = String.format("%.*f\n", precision, ...
user avatar
29votes
7answers
115kviews

Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate? Do you have a favourite example or anecdote which seems to get ...
28votes
9answers
27kviews

Rounding Errors?

In my course, I am told: Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...
user avatar
27votes
11answers
21kviews

Latitude/Longitude storage and compression in C

Does anyone know the most efficient representation for lat/long coordinates? Accuracy level should be enough for consumer GPS devices. Most implementations seem to use double for each unit, but I'm ...
user avatar
  • 14k
23votes
3answers
36kviews

How to safely floor or ceil a CGFloat to int?

I often need to floor or ceil a CGFloat to an int, for calculation of an array index. The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with ...
user avatar
  • 39.4k
23votes
1answer
19kviews

Precision lost while using read_csv in pandas

I have files of the below format in a text file which I am trying to read into a pandas dataframe. 895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0....
user avatar
23votes
1answer
10kviews

If dealing with money in a float is bad, then why does money_format() do it?

I've been waffling on how to deal with currency display and math in PHP, and for a long time have been storing it in MySQL using the DECIMAL type, and using money_format() to format it for display on ...
user avatar
  • 4,995
22votes
5answers
17kviews

Fast Exp calculation: possible to improve accuracy without losing too much performance?

I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#: public static double Exp(double x) { var tmp = (long)(...
user avatar
21votes
6answers
5kviews

Is there a floating point value of x, for which x-x == 0 is false?

In most cases, I understand that a floating point comparison test should be implemented using over a range of values (abs(x-y) < epsilon), but does self subtraction imply that the result will be ...
user avatar
21votes
2answers
3kviews

Numpy dot too clever about symmetric multiplications

Anybody know about documentation for this behaviour? import numpy as np A = np.random.uniform(0,1,(10,5)) w = np.ones(5) Aw = A*w Sym1 = Aw.dot(Aw.T) Sym2 = (A*w).dot((A*w).T) diff = Sym1 - Sym2 ...
user avatar
  • 1,629
21votes
6answers
618views

Efficiently computing (a - K) / (a + K) with improved accuracy

In various contexts, for example for the argument reduction for mathematical functions, one needs to compute (a - K) / (a + K), where a is a positive variable argument and K is a constant. In many ...
user avatar
  • 20.7k
19votes
14answers
54kviews

Truncate float numbers with PHP

When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...
user avatar
19votes
2answers
12kviews

Does "epsilon" really guarantees anything in floating-point computations?

To make the problem short let's say I want to compute the expression a / (b - c) on floats. To make sure the result is meaningful, I can check if b and c are in equal: float EPS = std::numeric_limits&...
user avatar
19votes
8answers
3kviews

printing float, preserving precision

I am writing a program that prints floating point literals to be used inside another program. How many digits do I need to print in order to preserve the precision of the original float? Since a ...
user avatar
19votes
2answers
6kviews

PHP/MySQL: Best money operations/storing practices?

So, I am planning to make an application (PHP/MySQL) which deals a lot with money, and I am thinking about how to store and operate with the money, referring to PHP float data type and MySQL decimal. ...
user avatar
  • 2,558
18votes
7answers
930views

Why does (int)(33.46639 * 1000000) return 33466389?

(int)(33.46639 * 1000000) returns 33466389 Why does this happen?
user avatar
  • 24.5k
18votes
4answers
4kviews

Precise sum of floating point numbers

I am aware of a similar question, but I want to ask for people opinion on my algorithm to sum floating point numbers as accurately as possible with practical costs. Here is my first solution: put ...
user avatar
18votes
2answers
3kviews

Java vs C floating point: "x * x" differs from "pow(x,2)"?

Why is this true? Java appears to produce a result with a small discrepancy when multiplying two floats compared to C and even the Java Math.pow method. Java: float a = 0.88276923; double b = a * a;...
user avatar
  • 1,348
18votes
2answers
748views

Invertability of IEEE 754 floating-point division

What is the invertability of the IEEE 754 floating-point division? I mean is it guaranteed by the standard that if double y = 1.0 / x then x == 1.0 / y, i.e. x can be restored precisely bit by bit? ...
user avatar
  • 7,698
17votes
5answers
1kviews

c++ floating point precision loss: 3015/0.00025298219406977296

The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == ...
user avatar
  • 25.4k
17votes
4answers
5kviews

Does casting `std::floor()` and `std::ceil()` to integer type always give the correct result?

I am being paranoid that one of these functions may give an incorrect result like this: std::floor(2000.0 / 1000.0) --> std::floor(1.999999999999) --> 1 or std::ceil(18 / 3) --> std::ceil(6....
user avatar
  • 10.1k
17votes
5answers
6kviews

Why do I need 17 significant digits (and not 16) to represent a double?

Can someone give me an example of a floating point number (double precision), that needs more than 16 significant decimal digits to represent it? I have found in this thread that sometimes you need up ...
user avatar

15 30 50 per page
1
2 3 4 5
26