Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

152 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
13votes
0answers
2kviews

ray and ellipsoid intersection accuracy improvement

I need to enhance precision for function in one of mine Atmospheric scattering GLSL fragment shader which computes the intersection between single ray and axis aligned ellipsoid. This is the core ...
user avatar
  • 44.6k
4votes
0answers
112views

Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
user avatar
  • 571
4votes
0answers
110views

add3 instruction for a+b+c with one single rounding

Background It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplication ...
user avatar
4votes
1answer
1kviews

Accounting system - MySQL and PHP precision

I'm building a kind of accounting system with PHP and MySQL. My database has DECIMAL (11,2) columns for currency, and also DECIMALfor all other values used on operations with currency (like ...
user avatar
  • 70.3k
3votes
0answers
81views

wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
user avatar
  • 441
3votes
2answers
544views

Find smallest integer that satisfies floating point inequality equation

I am looking for a fast algorithm that finds the smallest integer N that will satisfy the following inequality where s, q, u, and p are float numbers (using the IEEE-754 binary32 format): s > q + ...
user avatar
3votes
0answers
155views

Floating point [in]accuracy of C program, when running on the same machine, changed over last two weeks

The following C code was compiled today on two systems with Microsoft's compiler (installed with Visual Studio 2017 Community), both of which had modern 64-bit Intel processors and were running ...
user avatar
  • 1,606
3votes
0answers
237views

How is additional rounding error introduced when using f64::mul_add?

According to the docs, f64::mul_add can be used to reduce the number of opportunities for rounding errors: pub fn mul_add(self, a: f64, b: f64) -> f64 Fused multiply-add. Computes (self * a) ...
user avatar
3votes
0answers
98views

Why does the result of a floating point operation depend on splitting one statement into two?

I stumbled upon a problem, where the result of the floating point operation depends on whether I subtract a value now (f) or later (f2). Why is that? int main(int argc, char * argv[]) { const ...
user avatar
  • 3,617
3votes
1answer
124views

MATLAB precision when substracting

I need high precision for a project I'm working on. The problem I have is illustrated here when substracting: >> 1-0.9999999999999999 ans = 1.1102e-16 >> 1-0.99999999999999999 ans = 0 ...
user avatar
3votes
3answers
2kviews

Stop rounding when importing Google finance currency

I am importing currency from Google finance to Google spreadsheets (latest version). The number that is imported is rounded both in terms of what is displayed and the underlying number. Is there a way ...
user avatar
  • 31
3votes
0answers
394views

R: alternatives to all.equal?

While all.equal is a very powerful tool, with a cornucopia of informative response strings, I wanted an equivalent function which simply returned TRUE/FALSE for near-equality. (I'm aware of the ...
user avatar
3votes
1answer
232views

What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
user avatar
3votes
1answer
84views

std::fmod(4.2, 0.12) is equal to epsilon * 1.5

auto a{ 4.2 }; auto b{ 0.12 }; auto result = std::fmod(a, b); if(result <= std::numeric_limits<double>::epsilon()) result = 0; // <-- This line isn't triggered In this example, 4.2 is ...
user avatar
  • 31
2votes
0answers
118views

R vs Python differences in floating point resolution : Is it possible to set Python act like R?

I am working on a personal project involving the manipulation of linear algebra concepts. This project involves, in the first instance, replicating (translating) an R (64bit version) code into a ...
user avatar
  • 711
2votes
0answers
57views

Numerical stability of argument of complex number / branch cuts

I am implementing a numerical evaluation of some analytical expressions which involve factors like exp(1i*arg(z) / 2), where z is in principle a complex number, which sometimes happens to be almost ...
user avatar
  • 1,206
2votes
2answers
230views

How to set the width of td as float number

Can I set width of table cell as float type? td{ width:70.556123px;} Or is there any other way if I can set bandWidth of scale in d3.js as integer, not float? var xScale = d3.scaleBand(); xScale....
user avatar
  • 884
2votes
1answer
1kviews

How to solve difference equations numerically in python

Consider this difference equation:      The solution is      I am trying to solve it numerically in python, to explicate issues that arise with floating ...
user avatar
  • 143
2votes
0answers
95views

Avoid accuracy problems while computing the permanent using the Ryser formula

Task I want to calculate the permanent P of a NxN matrix for N up to 100. I can make use of the fact that the matrix features only M=4 (or slightly more) different rows and cols. The matrix might ...
user avatar
  • 43
2votes
0answers
59views

Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x[0] < x[1] < x[2] < ... &...
user avatar
  • 1,945
2votes
0answers
120views

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
user avatar
2votes
0answers
28views

Unintentional shortening (truncating) values in GraphicPath.AddLine

I'm loading coordinate values with GraphichsPath.AddLine command, then drawing them and I've noticed my map is not quite accurate. The code in question is as follows. I'm casting start and end ...
user avatar
  • 377
2votes
1answer
141views

How to fix floating point artifacts in StdDev calculation?

I trying to calculate standard deviation with a next method: private static double? StdDev(IReadOnlyCollection<double> items) { if(items == null) { throw new ArgumentNullException("items")...
user avatar
2votes
1answer
668views

numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
user avatar
  • 6,788
2votes
1answer
217views

Why is the mean smaller than the minimum and why does this change with 64bit floats?

I have an input array, which is a masked array. When I check the mean, I get a nonsensical number: less than the reported minimum value! So, raw array: numpy.mean(A) < numpy.min(A). Note A.dtype ...
user avatar
2votes
1answer
1kviews

JavaScript: Rounding number without impacting result accuracy

The problem: If you do log1000 the result you will get is log1000 = 2.9999999999999996 rather than 3. I tried to remove this rounding error in the JavaScript eval() function without impacting result ...
user avatar
2votes
2answers
425views

Number Of Digits in Fractional Portion of Float/Double

I am trying to implement http://www.exploringbinary.com/correct-decimal-to-floating-point-using-big-integers/ I have read through it quite a few times and feel very comfortable with it. The first ...
user avatar
  • 2,861
2votes
0answers
233views

Is it safe to use floats in YAML hashes with Ruby?

If I have a YAML file like --- 2.1: my product version without any quotation marks, Ruby will treat the 2.1 as a Float (at least under Syck - I'm not sure about Psych). Are there any risks in ...
user avatar
  • 73.6k
1vote
0answers
62views

When have enough bits of my series with non-negative terms been calculated?

I have a power series with all terms non-negative which I want to evaluate to some arbitrarily set precision p (the length in binary digits of a MPFR floating-point mantissa). The result should be ...
user avatar
1vote
0answers
160views

Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC: #include <iostream> namespace { template<typename Out> constexpr auto caster{[](auto x) constexpr ...
user avatar
1vote
0answers
141views

Calculating gradient by numerical approximation

I'm trying to calculate the gradient of a given function by the following definition: I'm trying to do that with the following code: def numerical_diff_gradient(func, vector: np.matrix, epsilon): ...
user avatar
1vote
1answer
304views

Error bound on floating point multiplication

I'm looking for a formula which bounds the error propagated by a floating point multiplication. I've seen the following stated (but not proved). If xa is a floating point approximation to x with ...
user avatar
  • 71.9k
1vote
0answers
133views

Rounded float is returned with all decimals

I have a rest endpoint that returns some simple statistics: total votes, average vote,... function getRatings($data) { global $wpdb; $struttura_id = intval( $data['struttura_id'] ); $sql = " ...
user avatar
1vote
0answers
58views

Multiply list of floats to retain maximum precision on a modern CPU

I have a list of many single-precision IEEE floating point numbers. How does one go through the list and multiply all numbers to ensure maximum precision, and no over/underflows? We assume there ...
user avatar
  • 337
1vote
0answers
129views

Error preprocessing the input data when using Tensorflow Dataset API

I have images of [64,512,5] stored in *.npy files which I convert into *.tfrecords files. I have verified that the reading of said records corresponds correctly with what is present in the *.npy ...
user avatar
1vote
2answers
49views

Only show decimal to 2 places in input box?

I have a calculator that I'm building where the user can convert united back and forth between imperial and metric. My code for this is as follows: oldVal = jQuery(this).val(); oldVal = parseFloat(...
user avatar
1vote
0answers
84views

Failures in OpenCV 3 Test on NVIDIA Jetson TX2 (related to gcc-7 vs gcc < 7)

I built OpenCV 3.3.1 with OpenCV_Extra on Jetson TX2 like the following with CUDA disabled: cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_PNG=OFF -DBUILD_TIFF=OFF -DBUILD_TBB=...
user avatar
  • 11
1vote
0answers
251views

Portable constant time integer log2 in C

I'm looking for an equivalent for (int) log2(X) that does not suffer from floating point accuracy problems. I have found several discussions about this problem (both in stackoverflow and outside), ...
user avatar
1vote
0answers
63views

Different answer when dividing by large to small and from small to large

This is for a homework assignment about floats and doubles, we are to add 1/1 + 1/2 + 1/3 ... up to 1/10,000,000 and to do it again from 1/10,000,000 back down to 1/1 and to print both answers as ...
user avatar
1vote
0answers
437views

Can I use Python's Decimal type with NumPy?

I'm doing some calculations with Python3 and NumPy using float64 and sometimes, when I change my formulas a little (but mathematically equivalent), then I get slightly different results. I know that ...
user avatar
  • 1,317
1vote
1answer
121views

Accuracy of Model not change, remain at zero

I have build a neural network with two hidden layer. For the first two hidden I used ReLU activation and for the last layer I used a sigmoid function. When I launch the model, loss function decreases (...
user avatar
  • 279
1vote
0answers
600views

Java - natural log with correct rounding

The documentation of the SE Math library is, thankfully, very transparent about rounding errors: If a method always has an error less than 0.5 ulps, the method always returns the floating-point ...
user avatar
1vote
0answers
1kviews

Floating point: choose epsilon to prevent zero division?

I have simple formula in my C++ program: const double z = x / tan(y); x is number from 1000 to 10 * 1000, y any angle. What is the right way to protect the code from zero division? I mean, I know ...
user avatar
  • 6,641
1vote
1answer
142views

Reliable Margin of Error for Float -> String -> Float Conversion?

I have a float value that I need to store as a string in PHP and then compare later after casting back into a float. Due to the conversion I know that relying on equality would be a mistake, as there'...
user avatar
  • 2,877
1vote
0answers
169views

How to check input location with accuracy inside in polygon on map or not?

I have plotted polygon on a map using MkMapView successfully. Now I want to check user location inside this polygon with accuracy. suppose I have latitude = 50.840473 and longitude = -0.146755 and ...
user avatar
1vote
0answers
225views

Can VGG 19 fine-tuned model outperform Inception -V3 fine-tuned model?

I am using pre-trained and fine-tuned models for cell classification. The images were not seen before by any of the transfer learning models. In the process, I found that a fine-tuned VGG19 model ...
user avatar
  • 967
1vote
1answer
321views

Is double arithmetic the same in cpp and javascript?

Context: I'm trying to port into c++ the implementation of a hash function that uses double arithmetic from its original implementation in JavaScript. In this function, the doubles we are operating on ...
user avatar
1vote
0answers
234views

Floating accuracy in pandas

when I apply pd.rolling_corr on some data set, I find the correlation matrix generated has values like -1.0000000000074845. I know it is about floating accuracy, but I do not know the usual way to ...
user avatar
  • 859
1vote
3answers
2kviews

Saving float and double precision in C++ during cin/cout and scanf/printf

I want to read floats and doubles from standart input and save its precision (exact the same digits after floating point) and be able to output (cout/printf) as it is. What the most convinient (and ...
user avatar
  • 71
1vote
0answers
641views

How can I improve the accuracy of my fft processing?

I completed an on-line Digital Signal Processing Course without doing the optional programming assignments which I am attempting now after the course is over. Several test inputs are provided as well ...
user avatar

15 30 50 per page