# All Questions

64 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
112views

### Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
110views

### add3 instruction for a+b+c with one single rounding

Background It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplication ...
81views

### wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
544views

### Find smallest integer that satisfies floating point inequality equation

I am looking for a fast algorithm that finds the smallest integer N that will satisfy the following inequality where s, q, u, and p are float numbers (using the IEEE-754 binary32 format): s > q + ...
155views

### Floating point [in]accuracy of C program, when running on the same machine, changed over last two weeks

The following C code was compiled today on two systems with Microsoft's compiler (installed with Visual Studio 2017 Community), both of which had modern 64-bit Intel processors and were running ...
232views

### What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
84views

### std::fmod(4.2, 0.12) is equal to epsilon * 1.5

auto a{ 4.2 }; auto b{ 0.12 }; auto result = std::fmod(a, b); if(result <= std::numeric_limits<double>::epsilon()) result = 0; // <-- This line isn't triggered In this example, 4.2 is ...
118views

### R vs Python differences in floating point resolution : Is it possible to set Python act like R?

I am working on a personal project involving the manipulation of linear algebra concepts. This project involves, in the first instance, replicating (translating) an R (64bit version) code into a ...
1kviews

### How to solve difference equations numerically in python

Consider this difference equation:      The solution is      I am trying to solve it numerically in python, to explicate issues that arise with floating ...
59views

### Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x < x < x < ... &...
120views

### Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
141views

### How to fix floating point artifacts in StdDev calculation?

I trying to calculate standard deviation with a next method: private static double? StdDev(IReadOnlyCollection<double> items) { if(items == null) { throw new ArgumentNullException("items")...
668views

### numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
425views

### Number Of Digits in Fractional Portion of Float/Double

I am trying to implement http://www.exploringbinary.com/correct-decimal-to-floating-point-using-big-integers/ I have read through it quite a few times and feel very comfortable with it. The first ...
1vote
160views

### Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC: #include <iostream> namespace { template<typename Out> constexpr auto caster{[](auto x) constexpr ...
1vote
304views

### Error bound on floating point multiplication

I'm looking for a formula which bounds the error propagated by a floating point multiplication. I've seen the following stated (but not proved). If xa is a floating point approximation to x with ...
1vote
437views

### Can I use Python's Decimal type with NumPy?

I'm doing some calculations with Python3 and NumPy using float64 and sometimes, when I change my formulas a little (but mathematically equivalent), then I get slightly different results. I know that ...
1vote
1kviews

### Floating point: choose epsilon to prevent zero division?

I have simple formula in my C++ program: const double z = x / tan(y); x is number from 1000 to 10 * 1000, y any angle. What is the right way to protect the code from zero division? I mean, I know ...
1vote
142views

### Reliable Margin of Error for Float -> String -> Float Conversion?

I have a float value that I need to store as a string in PHP and then compare later after casting back into a float. Due to the conversion I know that relying on equality would be a mistake, as there'...
1vote
321views

### Is double arithmetic the same in cpp and javascript?

Context: I'm trying to port into c++ the implementation of a hash function that uses double arithmetic from its original implementation in JavaScript. In this function, the doubles we are operating on ...
1vote
235views

### Using a Scaled Tolerance to Compare Floats

It occurred to me that a good way to compare floats is to scale the tolerance with the floats. Is it a good idea? This routine needs to be expanded for special quantities, such as NaN, +/-Inf, but it ...
1vote
3kviews

### Floating point truncation vs rounding by hand

I am trying to convert a decimal to a floating point integer on using 32 bit registers. I have to do this by hand (pencil and paper) so far my number is 1.11010110111100110100010011(base 2) x 2^26 ...
74views

### Lua float errors

I'm fairly new to lua, but I read that it does not have integers, so all numbers are floats. I noticed in my log that some numbers get a slight inaccuracy added to it. For instance 0.14 is written as ...
60views

### golang How to get the correct result? floating point arithmetic issues

testA, testB, testC := big.NewFloat(0), big.NewFloat(0), big.NewFloat(0) testA.SetPrec(500) testB.SetPrec(500) testA.SetString("0.081531021188798896") testB.SetString("0.9975") ...
56views

### as.character() with numeric loses precision when a string is transformed into numeric

In R I want to run following code: str <- "2811.1019192874087" val <- as.numeric(str) str2 <- as.character(val) print(str2) print(val) It prints: "2811.10191928741" 2811....
91views

### Floating point in google colab for verification

I'm implementing an algorithm in Python (GoogleColab) and it outputs two lists; now, I would like to verify if this lists are equal, actualy I know that they should be equal. The problem is that ...
37views

### Why do excel MATCH and "=" return different results?

In a previous question here it was determined that the MATCH error was being caused by floating point imprecision. There were still seemingly inconsistent results from excel that exceeded the scope ...
64views

### Implementation of max ulp deviation of single precision approximation

I'm facing the same situation for several times now: I have an approximation of a given function (e.g. cosine) in single precision, and I want evaluate the maximum error of my approximation measured ...
42views

### test-js: scientific software: significant figures

I am attempting to test scientific software written in TypeScript using Jest. I want to test all my floating point results to 8 significant figures, but it seems the only test method available in ts-...
56views

### Google calculator giving weird result?

So consider the following calculation 5777100 - 5777090.001370444 When I type it into Google, the result I get is 9.99862955604 and when I try using WolframAlpha I get 9.998629556 which is the ...
103views

### Floating-point inaccuracies in Python - how to fix

I'm writing a program that calculates the expected value of the sum of two discrete random variables. Here is the code: EX = 0 EY = 0 X_number=int(input("Enter the number of possible values for X: ")...
57views

### Fixing floating point error in R - error with MDMR package displaying approximate inequality rather than precise answer

I'm using the latest version of the MDMR package in R to analyze whether there is a significant relationship between a set of independent and dependent variables. However, when I execute something ...
30views

### What is the best way to handle IEEE 754 floating point imprecision issues in R?

So the issue is .1 + .2 =/= .3 The language is R. I have a bunch of fractions that in the end should sum to 1. And I have an assert statement that fails because this invariant does not always hold ...
249views

### Calculating sum of float numbers using math.fsum() showing incorrect values

I am performing a simple sum operation on float numbers using math.fsum() function in Python but the result is not matching with what am expecting. Please find the example below: math.fsum([-5.2,-...
45views

### String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
326views

### How to compute relative error for the rounding of floating point numbers when the rounded number is 0?

The relative rounding error for a floating point number x is defined as e_r = |(round(x) - x) / x| = |round(x)/x - 1| (1) Assuming that the rounding to nearest mode is used for round(x), the ...
46views

### Player facing at the wrong direction when stopping

I am using tic-80, and when the player stops, the xvelocity becomes 1.942...-16. I'm new to this, so I can't really see what the problem is. Here's my code.
177views

### How to deal with floating point number approximation?

I have a TDBEdit connected to a TFloatField which has DisplayFormat = '0.################'. I've set the field's value to 12.123456789 and I've noticed that my edit control displays it as '12,...
93views

### float precision in python

I want a floating point precision like Fortran in python program. In Fortran using pi gives a value of 3.141592653589793238462643 while using same numpy.pi gives a value of 3.141592653589793. Is it ...
90views

### Sum of fractions of 1 not exactly equally 1

I'm creating a 3D matrix(lat,long,landuse)and trying to calculate the fraction of the landuse for each matrix cell: One example cell could produce a vector specified by: fractions=landuse/...
58views

### Converting floats to ints reliably

I have an application that uses floats to handle distances, for example, 2.5f is 2.5 meters (250 centimeters). Now I want to interface with that application without using floats via network. I can't ...
57views

### Testing for alignment with floating point

This is for a physics engine: When an object collides over a slope, its speed will adjust accordingly to be parallel to said slope. To check if an object is moving towards or away from the slope, I ...
207views

### controlling floating point optimizations OS X C++ compiler

Is the system C++ compiler with Mac OS X (El Capitan) using strict IEEE floating point behavior by default? I am trying to debug some differences between Mac OS X and Linux, and I think it might be ...
238views

### My structure can not scan the double value in C

I can't figure it out what is happening. It can reads two integer numbers, but it can not read double. Its just become 0 and never change after scanning. The output: ---=== EMPLOYEE DATA ===--- 1. ...
58views

### php acting funny with large numbers

During one of my college competition the problem stated to take input from a file as numbers and then simply add half of it to itself. The numbers were 18 digit long integers. Most of my php output ...
76views

### logic error? diving 2 int that result in a float

I am writing a combat simulator. My attacker class initiates an attack and my defender class is supposed to block it. manager is my main class that computes and prints the results. My problem is with ...
91views

### Adding two numbers without float variable

i need to write function that add two numbers but we have a processor that does not support a Variable type Float !! So to represent fractions , is using a long integer is represented with actually ...
40views

### Calculate max digits of percision on a floating point number

According to wikipedia a single-percision floating point number maintains percision as long as there's less than 7 significant digits. Is there a formula for finding this maximum number of significant ...
223views

### Comparing doubles in a function

this is a part of my program: int calculation ( double f_a, double f_b, double s_a, double s_b, double seam) { int x, y, r1, r2; double waste_x, waste_y, c; x = (int)floor( s_a/f_a); ...