Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

152 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
13votes
0answers
2kviews

ray and ellipsoid intersection accuracy improvement

I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid. This is the core ...
user avatar
  • 44.6k
4votes
0answers
112views

Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
user avatar
  • 571
4votes
0answers
110views

add3 instruction for a+b+c with one single rounding

Background It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplication ...
user avatar
4votes
1answer
1kviews

Accounting system - MySQL and PHP precision

I'm building a kind of accounting system with PHP and MySQL. My database has DECIMAL (11,2) columns for currency, and also DECIMALfor all other values used on operations with currency (like ...
user avatar
  • 70.3k
3votes
0answers
81views

wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
user avatar
  • 441
3votes
2answers
544views

Find smallest integer that satisfies floating point inequality equation

I am looking for a fast algorithm that finds the smallest integer N that will satisfy the following inequality where s, q, u, and p are float numbers (using the IEEE-754 binary32 format): s > q + ...
user avatar
3votes
0answers
155views

Floating point [in]accuracy of C program, when running on the same machine, changed over last two weeks

The following C code was compiled today on two systems with Microsoft's compiler (installed with Visual Studio 2017 Community), both of which had modern 64-bit Intel processors and were running ...
user avatar
  • 1,606
3votes
0answers
237views

How is additional rounding error introduced when using f64::mul_add?

According to the docs, f64::mul_add can be used to reduce the number of opportunities for rounding errors: pub fn mul_add(self, a: f64, b: f64) -> f64 Fused multiply-add. Computes (self * a) ...
user avatar
3votes
0answers
98views

Why does the result of a floating point operation depend on splitting one statement into two?

I stumbled upon a problem, where the result of the floating point operation depends on whether I subtract a value now (f) or later (f2). Why is that? int main(int argc, char * argv[]) { const ...
user avatar
  • 3,617
3votes
1answer
124views

MATLAB precision when substracting

I need high precision for a project I'm working on. The problem I have is illustrated here when substracting: >> 1-0.9999999999999999 ans = 1.1102e-16 >> 1-0.99999999999999999 ans = 0 ...
user avatar
3votes
3answers
2kviews

Stop rounding when importing Google finance currency

I am importing currency from Google finance to Google spreadsheets (latest version). The number that is imported is rounded both in terms of what is displayed and the underlying number. Is there a way ...
user avatar
  • 31
3votes
0answers
394views

R: alternatives to all.equal?

While all.equal is a very powerful tool, with a cornucopia of informative response strings, I wanted an equivalent function which simply returned TRUE/FALSE for near-equality. (I'm aware of the ...
user avatar
3votes
1answer
232views

What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
user avatar
3votes
1answer
84views

std::fmod(4.2, 0.12) is equal to epsilon * 1.5

auto a{ 4.2 }; auto b{ 0.12 }; auto result = std::fmod(a, b); if(result <= std::numeric_limits<double>::epsilon()) result = 0; // <-- This line isn't triggered In this example, 4.2 is ...
user avatar
  • 31
2votes
0answers
118views

R vs Python differences in floating point resolution : Is it possible to set Python act like R?

I am working on a personal project involving the manipulation of linear algebra concepts. This project involves, in the first instance, replicating (translating) an R (64bit version) code into a ...
user avatar
  • 711

15 30 50 per page
1
2 3 4 5
11