# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

152
questions with no upvoted or accepted answers

13votes

0answers

2kviews

### ray and ellipsoid intersection accuracy improvement

I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid.
This is the core ...

4votes

0answers

112views

### Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...

4votes

0answers

110views

### add3 instruction for a+b+c with one single rounding

Background
It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplication ...

4votes

1answer

1kviews

### Accounting system - MySQL and PHP precision

I'm building a kind of accounting system with PHP and MySQL. My database has DECIMAL (11,2) columns for currency, and also DECIMALfor all other values used on operations with currency (like ...

3votes

0answers

81views

### wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one.
I'm getting 1 instead of 1+epsilon when I perform the addition and print the result.
I've implemented a getEpsilon function. I ...

3votes

2answers

544views

### Find smallest integer that satisfies floating point inequality equation

I am looking for a fast algorithm that finds the smallest integer N that will satisfy the following inequality where s, q, u, and p are float numbers (using the IEEE-754 binary32 format):
s > q + ...

3votes

0answers

155views

### Floating point [in]accuracy of C program, when running on the same machine, changed over last two weeks

The following C code was compiled today on two systems with Microsoft's compiler (installed with Visual Studio 2017 Community), both of which had modern 64-bit Intel processors and were running ...

3votes

0answers

237views

### How is additional rounding error introduced when using f64::mul_add?

According to the docs, f64::mul_add can be used to reduce the number of opportunities for rounding errors:
pub fn mul_add(self, a: f64, b: f64) -> f64
Fused multiply-add. Computes (self * a) ...

3votes

0answers

98views

### Why does the result of a floating point operation depend on splitting one statement into two?

I stumbled upon a problem, where the result of the floating point operation depends on whether I subtract a value now (f) or later (f2). Why is that?
int main(int argc, char * argv[])
{
const ...

3votes

1answer

124views

### MATLAB precision when substracting

I need high precision for a project I'm working on.
The problem I have is illustrated here when substracting:
>> 1-0.9999999999999999
ans = 1.1102e-16
>> 1-0.99999999999999999
ans = 0
...

3votes

3answers

2kviews

### Stop rounding when importing Google finance currency

I am importing currency from Google finance to Google spreadsheets (latest version). The number that is imported is rounded both in terms of what is displayed and the underlying number. Is there a way ...

3votes

0answers

394views

### R: alternatives to all.equal?

While all.equal is a very powerful tool, with a cornucopia of informative response strings, I wanted an equivalent function which simply returned TRUE/FALSE for near-equality. (I'm aware of the ...

3votes

1answer

232views

### What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float.
In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read:
...

3votes

1answer

84views

### std::fmod(4.2, 0.12) is equal to epsilon * 1.5

auto a{ 4.2 };
auto b{ 0.12 };
auto result = std::fmod(a, b);
if(result <= std::numeric_limits<double>::epsilon())
result = 0; // <-- This line isn't triggered
In this example, 4.2 is ...

2votes

0answers

118views

### R vs Python differences in floating point resolution : Is it possible to set Python act like R?

I am working on a personal project involving the manipulation of linear algebra concepts.
This project involves, in the first instance, replicating (translating) an R (64bit version) code into a ...

2votes

0answers

57views

### Numerical stability of argument of complex number / branch cuts

I am implementing a numerical evaluation of some analytical expressions which involve factors like exp(1i*arg(z) / 2), where z is in principle a complex number, which sometimes happens to be almost ...

2votes

2answers

230views

### How to set the width of td as float number

Can I set width of table cell as float type?
td{ width:70.556123px;}
Or is there any other way if I can set bandWidth of scale in d3.js as integer, not float?
var xScale = d3.scaleBand();
xScale....

2votes

1answer

1kviews

### How to solve difference equations numerically in python

Consider this difference equation:
The solution is
I am trying to solve it numerically in python, to explicate issues that arise with floating ...

2votes

0answers

95views

### Avoid accuracy problems while computing the permanent using the Ryser formula

Task
I want to calculate the permanent P of a NxN matrix for N up to 100. I can make use of the fact that the matrix features only M=4 (or slightly more) different rows and cols. The matrix might ...

2votes

0answers

59views

### Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double
T x[n];
These numbers are strictly positive and sorted, i.e.
0 < x[0] < x[1] < x[2] < ... &...

2votes

0answers

120views

### Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2):
(1/2)B^-p <= (1/2)ulp <= (B/2)B^-p
...

2votes

0answers

28views

### Unintentional shortening (truncating) values in GraphicPath.AddLine

I'm loading coordinate values with GraphichsPath.AddLine command, then drawing them and I've noticed my map is not quite accurate.
The code in question is as follows. I'm casting start and end ...

2votes

1answer

141views

### How to fix floating point artifacts in StdDev calculation?

I trying to calculate standard deviation with a next method:
private static double? StdDev(IReadOnlyCollection<double> items) {
if(items == null) {
throw new ArgumentNullException("items")...

2votes

1answer

668views

### numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...

2votes

1answer

217views

### Why is the mean smaller than the minimum and why does this change with 64bit floats?

I have an input array, which is a masked array.
When I check the mean, I get a nonsensical number: less than the reported minimum value!
So, raw array: numpy.mean(A) < numpy.min(A). Note A.dtype ...

2votes

1answer

1kviews

### JavaScript: Rounding number without impacting result accuracy

The problem: If you do log1000 the result you will get is log1000 = 2.9999999999999996 rather than 3.
I tried to remove this rounding error in the JavaScript eval() function without impacting result ...

2votes

2answers

425views

### Number Of Digits in Fractional Portion of Float/Double

I am trying to implement http://www.exploringbinary.com/correct-decimal-to-floating-point-using-big-integers/
I have read through it quite a few times and feel very comfortable with it. The first ...

2votes

0answers

233views

### Is it safe to use floats in YAML hashes with Ruby?

If I have a YAML file like
---
2.1: my product version
without any quotation marks, Ruby will treat the 2.1 as a Float (at least under Syck - I'm not sure about Psych).
Are there any risks in ...

1vote

0answers

62views

### When have enough bits of my series with non-negative terms been calculated?

I have a power series with all terms non-negative which I want to evaluate to some arbitrarily set precision p (the length in binary digits of a MPFR floating-point mantissa). The result should be ...

1vote

0answers

160views

### Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC:
#include <iostream>
namespace {
template<typename Out>
constexpr auto caster{[](auto x) constexpr ...