All Questions

64 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4votes
0answers
112views

Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
user avatar
  • 571
4votes
0answers
110views

add3 instruction for a+b+c with one single rounding

Background It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplication ...
user avatar
3votes
0answers
81views

wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
user avatar
  • 441
3votes
2answers
544views

Find smallest integer that satisfies floating point inequality equation

I am looking for a fast algorithm that finds the smallest integer N that will satisfy the following inequality where s, q, u, and p are float numbers (using the IEEE-754 binary32 format): s > q + ...
user avatar
3votes
0answers
155views

Floating point [in]accuracy of C program, when running on the same machine, changed over last two weeks

The following C code was compiled today on two systems with Microsoft's compiler (installed with Visual Studio 2017 Community), both of which had modern 64-bit Intel processors and were running ...
user avatar
  • 1,606
3votes
1answer
232views

What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
user avatar
3votes
1answer
84views

std::fmod(4.2, 0.12) is equal to epsilon * 1.5

auto a{ 4.2 }; auto b{ 0.12 }; auto result = std::fmod(a, b); if(result <= std::numeric_limits<double>::epsilon()) result = 0; // <-- This line isn't triggered In this example, 4.2 is ...
user avatar
  • 31
2votes
0answers
118views

R vs Python differences in floating point resolution : Is it possible to set Python act like R?

I am working on a personal project involving the manipulation of linear algebra concepts. This project involves, in the first instance, replicating (translating) an R (64bit version) code into a ...
user avatar
  • 711
2votes
1answer
1kviews

How to solve difference equations numerically in python

Consider this difference equation:      The solution is      I am trying to solve it numerically in python, to explicate issues that arise with floating ...
user avatar
  • 143
2votes
0answers
59views

Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x[0] < x[1] < x[2] < ... &...
user avatar
  • 1,945
2votes
0answers
120views

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
user avatar
2votes
1answer
141views

How to fix floating point artifacts in StdDev calculation?

I trying to calculate standard deviation with a next method: private static double? StdDev(IReadOnlyCollection<double> items) { if(items == null) { throw new ArgumentNullException("items")...
user avatar
2votes
1answer
668views

numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
user avatar
  • 6,788
2votes
2answers
425views

Number Of Digits in Fractional Portion of Float/Double

I am trying to implement http://www.exploringbinary.com/correct-decimal-to-floating-point-using-big-integers/ I have read through it quite a few times and feel very comfortable with it. The first ...
user avatar
  • 2,861
1vote
0answers
160views

Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC: #include <iostream> namespace { template<typename Out> constexpr auto caster{[](auto x) constexpr ...
user avatar
1vote
1answer
304views

Error bound on floating point multiplication

I'm looking for a formula which bounds the error propagated by a floating point multiplication. I've seen the following stated (but not proved). If xa is a floating point approximation to x with ...
user avatar
  • 71.9k
1vote
0answers
437views

Can I use Python's Decimal type with NumPy?

I'm doing some calculations with Python3 and NumPy using float64 and sometimes, when I change my formulas a little (but mathematically equivalent), then I get slightly different results. I know that ...
user avatar
  • 1,317
1vote
0answers
1kviews

Floating point: choose epsilon to prevent zero division?

I have simple formula in my C++ program: const double z = x / tan(y); x is number from 1000 to 10 * 1000, y any angle. What is the right way to protect the code from zero division? I mean, I know ...
user avatar
  • 6,641
1vote
1answer
142views

Reliable Margin of Error for Float -> String -> Float Conversion?

I have a float value that I need to store as a string in PHP and then compare later after casting back into a float. Due to the conversion I know that relying on equality would be a mistake, as there'...
user avatar
  • 2,877
1vote
1answer
321views

Is double arithmetic the same in cpp and javascript?

Context: I'm trying to port into c++ the implementation of a hash function that uses double arithmetic from its original implementation in JavaScript. In this function, the doubles we are operating on ...
user avatar
1vote
1answer
235views

Using a Scaled Tolerance to Compare Floats

It occurred to me that a good way to compare floats is to scale the tolerance with the floats. Is it a good idea? This routine needs to be expanded for special quantities, such as NaN, +/-Inf, but it ...
user avatar
  • 2,962
1vote
2answers
3kviews

Floating point truncation vs rounding by hand

I am trying to convert a decimal to a floating point integer on using 32 bit registers. I have to do this by hand (pencil and paper) so far my number is 1.11010110111100110100010011(base 2) x 2^26 ...
user avatar
0votes
1answer
74views

Lua float errors

I'm fairly new to lua, but I read that it does not have integers, so all numbers are floats. I noticed in my log that some numbers get a slight inaccuracy added to it. For instance 0.14 is written as ...
user avatar
  • 97
0votes
1answer
60views

golang How to get the correct result? floating point arithmetic issues

testA, testB, testC := big.NewFloat(0), big.NewFloat(0), big.NewFloat(0) testA.SetPrec(500) testB.SetPrec(500) testA.SetString("0.081531021188798896") testB.SetString("0.9975") ...
user avatar
  • 1,152
0votes
0answers
56views

as.character() with numeric loses precision when a string is transformed into numeric

In R I want to run following code: str <- "2811.1019192874087" val <- as.numeric(str) str2 <- as.character(val) print(str2) print(val) It prints: "2811.10191928741" 2811....
user avatar
  • 1,526
0votes
0answers
91views

Floating point in google colab for verification

I'm implementing an algorithm in Python (GoogleColab) and it outputs two lists; now, I would like to verify if this lists are equal, actualy I know that they should be equal. The problem is that ...
user avatar
  • 129
0votes
0answers
37views

Why do excel MATCH and "=" return different results?

In a previous question here it was determined that the MATCH error was being caused by floating point imprecision. There were still seemingly inconsistent results from excel that exceeded the scope ...
user avatar
0votes
0answers
64views

Implementation of max ulp deviation of single precision approximation

I'm facing the same situation for several times now: I have an approximation of a given function (e.g. cosine) in single precision, and I want evaluate the maximum error of my approximation measured ...
user avatar
  • 111
0votes
1answer
42views

test-js: scientific software: significant figures

I am attempting to test scientific software written in TypeScript using Jest. I want to test all my floating point results to 8 significant figures, but it seems the only test method available in ts-...
user avatar
  • 51
0votes
0answers
56views

Google calculator giving weird result?

So consider the following calculation 5777100 - 5777090.001370444 When I type it into Google, the result I get is 9.99862955604 and when I try using WolframAlpha I get 9.998629556 which is the ...
user avatar
  • 375
0votes
0answers
103views

Floating-point inaccuracies in Python - how to fix

I'm writing a program that calculates the expected value of the sum of two discrete random variables. Here is the code: EX = 0 EY = 0 X_number=int(input("Enter the number of possible values for X: ")...
user avatar
0votes
0answers
57views

Fixing floating point error in R - error with MDMR package displaying approximate inequality rather than precise answer

I'm using the latest version of the MDMR package in R to analyze whether there is a significant relationship between a set of independent and dependent variables. However, when I execute something ...
user avatar
0votes
0answers
30views

What is the best way to handle IEEE 754 floating point imprecision issues in R?

So the issue is .1 + .2 =/= .3 The language is R. I have a bunch of fractions that in the end should sum to 1. And I have an assert statement that fails because this invariant does not always hold ...
user avatar
0votes
0answers
249views

Calculating sum of float numbers using math.fsum() showing incorrect values

I am performing a simple sum operation on float numbers using math.fsum() function in Python but the result is not matching with what am expecting. Please find the example below: math.fsum([-5.2,-...
user avatar
  • 1
0votes
0answers
45views

String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
user avatar
  • 2,453
0votes
1answer
326views

How to compute relative error for the rounding of floating point numbers when the rounded number is 0?

The relative rounding error for a floating point number x is defined as e_r = |(round(x) - x) / x| = |round(x)/x - 1| (1) Assuming that the rounding to nearest mode is used for round(x), the ...
user avatar
  • 5,025
0votes
0answers
46views

Player facing at the wrong direction when stopping

I am using tic-80, and when the player stops, the xvelocity becomes 1.942...-16. I'm new to this, so I can't really see what the problem is. Here's my code.
user avatar
  • 11
0votes
0answers
177views

How to deal with floating point number approximation?

I have a TDBEdit connected to a TFloatField which has DisplayFormat = '0.################'. I've set the field's value to 12.123456789 and I've noticed that my edit control displays it as '12,...
user avatar
  • 7,128
0votes
1answer
93views

float precision in python

I want a floating point precision like Fortran in python program. In Fortran using pi gives a value of 3.141592653589793238462643 while using same numpy.pi gives a value of 3.141592653589793. Is it ...
user avatar
  • 23
0votes
0answers
90views

Sum of fractions of 1 not exactly equally 1

I'm creating a 3D matrix(lat,long,landuse)and trying to calculate the fraction of the landuse for each matrix cell: One example cell could produce a vector specified by: fractions=landuse/...
user avatar
  • 322
0votes
0answers
58views

Converting floats to ints reliably

I have an application that uses floats to handle distances, for example, 2.5f is 2.5 meters (250 centimeters). Now I want to interface with that application without using floats via network. I can't ...
user avatar
  • 872
0votes
1answer
57views

Testing for alignment with floating point

This is for a physics engine: When an object collides over a slope, its speed will adjust accordingly to be parallel to said slope. To check if an object is moving towards or away from the slope, I ...
user avatar
  • 317
0votes
1answer
207views

controlling floating point optimizations OS X C++ compiler

Is the system C++ compiler with Mac OS X (El Capitan) using strict IEEE floating point behavior by default? I am trying to debug some differences between Mac OS X and Linux, and I think it might be ...
user avatar
  • 3,456
0votes
1answer
238views

My structure can not scan the double value in C

I can't figure it out what is happening. It can reads two integer numbers, but it can not read double. Its just become 0 and never change after scanning. The output: ---=== EMPLOYEE DATA ===--- 1. ...
user avatar
  • 17
0votes
0answers
58views

php acting funny with large numbers

During one of my college competition the problem stated to take input from a file as numbers and then simply add half of it to itself. The numbers were 18 digit long integers. Most of my php output ...
user avatar
0votes
1answer
76views

logic error? diving 2 int that result in a float

I am writing a combat simulator. My attacker class initiates an attack and my defender class is supposed to block it. manager is my main class that computes and prints the results. My problem is with ...
user avatar
0votes
1answer
91views

Adding two numbers without float variable

i need to write function that add two numbers but we have a processor that does not support a Variable type Float !! So to represent fractions , is using a long integer is represented with actually ...
user avatar
0votes
1answer
40views

Calculate max digits of percision on a floating point number

According to wikipedia a single-percision floating point number maintains percision as long as there's less than 7 significant digits. Is there a formula for finding this maximum number of significant ...
user avatar
0votes
0answers
223views

Comparing doubles in a function

this is a part of my program: int calculation ( double f_a, double f_b, double s_a, double s_b, double seam) { int x, y, r1, r2; double waste_x, waste_y, c; x = (int)floor( s_a/f_a); ...
user avatar
0votes
0answers
85views

How computer shows us the result of 0.7 when we add 0.1 and 0.6 in calculator?

1) As I know, the computer converts the decimal numbers to binary digits and transacts as it. For example, when we add the decimal numbers like "12" and "37" in computer's calculator. Is it correct? ...
user avatar
  • 119

15 30 50 per page