# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

1,276 questions
Filter by
Sorted by
Tagged with
456kviews

### Is floating point math broken?

Consider the following code: 0.1 + 0.2 == 0.3 -> false 0.1 + 0.2 -> 0.30000000000000004 Why do these inaccuracies happen?
95kviews

### How dangerous is it to compare floating point values?

I know UIKit uses CGFloat because of the resolution independent coordinate system. But every time I want to check if for example frame.origin.x is 0 it makes me feel sick: if (theView.frame.origin.x ...
58kviews

### Why are these numbers not equal?

The following code is obviously wrong. What's the problem? i <- 0.1 i <- i + 0.05 i ##  0.15 if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15") ## i does not equal 0.15
177kviews

### What's wrong with using == to compare floats in Java?

According to this java.sun page == is the equality comparison operator for floating point numbers in Java. However, when I type this code: if(sectionID == currentSectionID) into my editor and run ...
15kviews

### Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values ...
11kviews

### Is it possible to get 0 by subtracting two unequal floating point numbers?

Is it possible to get division by 0 (or infinity) in the following example? public double calculation(double a, double b) { if (a == b) { return 0; } else { ...
239kviews

### Dealing with float precision in Javascript [duplicate]

I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string. How do I get around the annoying ...
173kviews

### Truncate (not round off) decimal numbers in javascript

I am trying to truncate decimal numbers to decimal places. Something like this: 5.467 -> 5.46 985.943 -> 985.94 toFixed(2) does just about the right thing but it rounds off the value. I ...
6kviews

### What's the closest double to 1.0, that isn't 1.0?

Is there a way to programmatically get the double that is closest to 1.0, but isn't actually 1.0? One hacky way to do this would be to memcpy the double to a same-sized integer, and then subtract one....
53kviews

### PHP - Floating Number Precision [duplicate]

\$a = '35'; \$b = '-34.99'; echo (\$a + \$b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
39kviews

### Large numbers erroneously rounded in JavaScript

See this code: var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); When I see my console in Firefox 3.5,...
18kviews

### Why 0.1 + 0.2 == 0.3 in D?

assert(0.1 + 0.2 != 0.3); // shall be true is my favorite check that a language uses native floating point arithmetic. C++ #include <cstdio> int main() { printf("%d\n", (0.1 + 0.2 != 0.3))...
7kviews

### (.1f+.2f==.3f) != (.1f+.2f).Equals(.3f) Why?

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference and ...
38kviews

### negative zero in python

I encountered negative zero in output from python; it's created for example as follows: k = 0.0 print(-k) The output will be -0.0. However, when I compare the -k to 0.0 for equality, it yields True....
5kviews

### Inaccuracy of decimal in .NET

Yesterday during debugging something strange happened to me and I can't really explain it: So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but shouldn't ...
67kviews

### Is floating point arbitrary precision available?

Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples. ...
51kviews

### What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers? The alternate is loading an external library like <script type="text/javascript" src="the_bignum_library.js"></script&...
77kviews

### How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log(...
48kviews

### How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation). \$ echo "3.1e1*2" | bc -l (standard_in) 1: parse error but I need to use it to handle a few records that are ...
28kviews

### Why float.Epsilon and not zero?

In the following code, why is there a comparison against float.Epsilon and not 0? // Coroutine to move elements protected IEnumerator SmoothMovement (Vector3 end) { // Distance computation ...
33kviews

### Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1.0 ...
44kviews

### What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the ...
9kviews

### When to use `std::hypot(x,y)` over `std::sqrt(x*x + y*y)`

The documentation of std::hypot says that: Computes the square root of the sum of the squares of x and y, without undue overflow or underflow at intermediate stages of the computation. I struggle ...
52kviews

### How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one: if (fabs(prev.min[i] - cur->min[i]) < 0.000001 && fabs(...
2kviews

### Is there any accuracy gain when casting to double and back when doing float division?

What is the difference between two following? float f1 = some_number; float f2 = some_near_zero_number; float result; result = f1 / f2; and: float f1 = some_number; float f2 = ...
52kviews

### How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( ...
3kviews

### Is integer division always equal to the floor of regular division?

For large quotients, integer division (//) doesn't seem to be necessarily equal to the floor of regular division (math.floor(a/b)). According to Python docs (https://docs.python.org/3/reference/...
6kviews

### mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given ...
28kviews

### How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right now I'm trying something like: String foo = String.format("%.*f\n", precision, ...
115kviews

### Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate? Do you have a favourite example or anecdote which seems to get ...
27kviews

### Rounding Errors?

In my course, I am told: Continuous values are represented approximately in memory, and therefore computing with floats involves rounding errors. These are tiny discrepancies in bit patterns; thus ...
21kviews

### Latitude/Longitude storage and compression in C

Does anyone know the most efficient representation for lat/long coordinates? Accuracy level should be enough for consumer GPS devices. Most implementations seem to use double for each unit, but I'm ...
36kviews

### How to safely floor or ceil a CGFloat to int?

I often need to floor or ceil a CGFloat to an int, for calculation of an array index. The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with ...
19kviews

### Precision lost while using read_csv in pandas

I have files of the below format in a text file which I am trying to read into a pandas dataframe. 895|2015-4-23|19|10000|LA|0.4677978806|0.4773469340|0.4089938425|0.8224291972|0.8652525793|0.... 10kviews

### If dealing with money in a float is bad, then why does money_format() do it?

I've been waffling on how to deal with currency display and math in PHP, and for a long time have been storing it in MySQL using the DECIMAL type, and using money_format() to format it for display on ...
17kviews

### Fast Exp calculation: possible to improve accuracy without losing too much performance?

I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#: public static double Exp(double x) { var tmp = (long)(...
5kviews

### Is there a floating point value of x, for which x-x == 0 is false?

In most cases, I understand that a floating point comparison test should be implemented using over a range of values (abs(x-y) < epsilon), but does self subtraction imply that the result will be ...
3kviews

### Numpy dot too clever about symmetric multiplications

Anybody know about documentation for this behaviour? import numpy as np A = np.random.uniform(0,1,(10,5)) w = np.ones(5) Aw = A*w Sym1 = Aw.dot(Aw.T) Sym2 = (A*w).dot((A*w).T) diff = Sym1 - Sym2 ...
618views

### Efficiently computing (a - K) / (a + K) with improved accuracy

In various contexts, for example for the argument reduction for mathematical functions, one needs to compute (a - K) / (a + K), where a is a positive variable argument and K is a constant. In many ...
54kviews

### Truncate float numbers with PHP

When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit after ...
12kviews

### Does "epsilon" really guarantees anything in floating-point computations?

To make the problem short let's say I want to compute the expression a / (b - c) on floats. To make sure the result is meaningful, I can check if b and c are in equal: float EPS = std::numeric_limits&...
3kviews

### printing float, preserving precision

I am writing a program that prints floating point literals to be used inside another program. How many digits do I need to print in order to preserve the precision of the original float? Since a ...
6kviews

### PHP/MySQL: Best money operations/storing practices?

So, I am planning to make an application (PHP/MySQL) which deals a lot with money, and I am thinking about how to store and operate with the money, referring to PHP float data type and MySQL decimal. ...
930views

### Why does (int)(33.46639 * 1000000) return 33466389?

(int)(33.46639 * 1000000) returns 33466389 Why does this happen?
4kviews

### Precise sum of floating point numbers

I am aware of a similar question, but I want to ask for people opinion on my algorithm to sum floating point numbers as accurately as possible with practical costs. Here is my first solution: put ...
3kviews

### Java vs C floating point: "x * x" differs from "pow(x,2)"?

Why is this true? Java appears to produce a result with a small discrepancy when multiplying two floats compared to C and even the Java Math.pow method. Java: float a = 0.88276923; double b = a * a;...
748views

### Invertability of IEEE 754 floating-point division

What is the invertability of the IEEE 754 floating-point division? I mean is it guaranteed by the standard that if double y = 1.0 / x then x == 1.0 / y, i.e. x can be restored precisely bit by bit? ...
1kviews

### c++ floating point precision loss: 3015/0.00025298219406977296

The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == ...
5kviews

### Does casting `std::floor()` and `std::ceil()` to integer type always give the correct result?

I am being paranoid that one of these functions may give an incorrect result like this: std::floor(2000.0 / 1000.0) --> std::floor(1.999999999999) --> 1 or std::ceil(18 / 3) --> std::ceil(6....