All Questions
Tagged with floating-point math
58
questions with no upvoted or accepted answers
3votes
1answer
62views
Dealing with floating point point inaccuracy in very small numbers efficiently
The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data.
I am drawing lines, that sometimes exceed these bounds ...
- 43
3votes
1answer
124views
Javascript wrongfully changes the result of a simple multiplication. How can I fix it?
function roundUp(num, precision) {
return Math.ceil(num * precision) / precision;
}
var num = 0.07;
var precision = 100;
console.log(roundUp(num, precision));
When the arguments to the ...
- 1,154
2votes
0answers
120views
Floating point relative error bounds, clarification needed
I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2):
(1/2)B^-p <= (1/2)ulp <= (B/2)B^-p
...
- 19.5k
2votes
0answers
175views
How can I find out if my JVM is using hardware square root?
If you read the top of the incredibly-hard-to-find native sqrt method for Java, which is located at jdk1.6\src\jdk\src\share\native\java\lang\fdlibm\src\e_sqrt.c you will find this:
/* __ieee754_sqrt(...
- 10.6k
1vote
1answer
51views
Is Java floating arithmetic division always within 1 ULP of the true result?
Is Java's floating arithmetic division always within 1 ULP of the true result? I read that CPUs sometimes do floating-point division a/b by doing a * 1/b. However, 1/b may be off by 1 ULP, and ...
- 502
1vote
0answers
60views
How to deterministically divide floats when there is a known deviation from IEEE 754?
DirectX 11 allows GPU manufacturers to deviate from the rounding behavior specified in the IEEE 754 standard. I cannot enable IEEE strictness, because I don't control the shader compilation process.
...
- 2,253
1vote
1answer
423views
'round()' vs 'std::round()' and 'fabs()' vs 'std::fabs()' in C++ / GCC 4.8
By accident I was calling round() and fabs() instead of std::round() and std::fabs() and for the largest integer a long double can hold without loosing precision there was a difference.
Consider this ...
- 3,684
1vote
0answers
55views
How to implement GNUMP in this Bernoulli numbers algortihm I wrote in C?
I wrote this C algortihm to calculate Bernoulli numbers when the input is Nth, but it only works until 136th because of the floating-point limitations of C. I'm not a programmer, more a mathematician, ...
1vote
0answers
39views
How to handle precision Decimals in Python and overcome binary representation limitations in computers?
How to obtain precision without rounding?
For instance:
import math
def truncate(f, n):
return math.floor(f * 10 ** n) / 10 ** n
r = lambda f,p: f - f % p
size = 3245.249999999999853146309903
y ...
- 1,231
1vote
0answers
46views
Is it possible to manually increase the precision of a double as a bit string through calculations
In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa.
Double precision uses 11 exponent bits, and 52 mantissa bits.
Let's say we ...
1vote
0answers
90views
Does cpp_dec_float give consistent results across platforms?
Can a custom number type such as cpp_dec_float (from Boost) guarantee a consistent floating-point arithmetic behavior (on trigonometric functions such as cos, sin, tan, atan) across different system ...
- 1,749
1vote
0answers
23views
Interpreting parameter values from scipy stat and popt
Really dumb question, but I am new to python and my math is super rusty.
I ran a function using the curve_fit function in python and got a list of parameter my parameter for my A and B variable came ...
- 129
1vote
0answers
69views
How to know the effective input domain of special function in SciPy or GSL?
I need to know the effective input domain of special functions in SciPy (Link to special functions of SciPy) or GSL(Link to special functions of GSL). For example, for the function gamma, the ...
- 59
1vote
0answers
88views
Map float to a partition possible in constant time?
Suppose I have a partition (where each partition is a continuous range) over a range of floats:
R = [a_0, a_n)
P(R) = [a_0, a_1), [a_1, a_2), ... , [a_n-1, a_n)
For any given float f in R, is there ...
user6582112
1vote
0answers
91views
Error-free sum in F#
I'm continuing with a port of GeographicLib into F#, and I'm wondering about the use of the error-free sum. In the C++ codebase, it is defined as
/**
* The error-free sum of two numbers.
*
* @...
- 11.3k
1vote
1answer
90views
Why does matlab use the following interpolation function?
If you try performing linear interpolation in matlab, it uses a "gridded" interpolant apparently.
Normally, I would expect the linear interpolation function to look something like this:
METHOD A:
...
- 3,410
1vote
2answers
231views
How to scale down blocks without causing pixel overlapping
I have a bunch of blocks that needs to be drawn into a grid. Now displaying them unscaled everything is fine but when I try to scale them down to fit withing a window I get "scale-artifacts" because I ...
- 51
1vote
3answers
609views
comparing floats precisely after a cross product operation
This is about performing the following comparison precisely :
CrossProduct(P - P0, P1 - P0) >, < or == 0
Problem definition
Given a set of grid points described by the following equation
P(x,...
- 71
0votes
0answers
37views
Fast inverse square root in python on float32
I have done some checks for the fast inverse square root method in python (jupyterlab using python version 3.8.8) and for some reason then I've come to the conclusion that I must either be doing ...
- 1
0votes
0answers
52views
Inconsistent results with floating point arithmetic (color conversion formulae)
I am trying to parse a bunch of color values from various files, perform lightness changes, and write them back into the files. I have to put this in a PKGBUILD for the Arch User Repository and would ...
- 1,757
0votes
0answers
46views
Increasing precision for decimal part from 8 decimals to 18 decimals using Uint 128 bit
I am trying to write the translation of unsigned integers to floating points with precision of up to 18 decimals.
The language (Clarity) that I am using doesn't have floating points in it so I have to ...
- 275
0votes
1answer
39views
Floating points in big numbers from a select list (js)
I have 2 select lists in html, place to enter a number and second place to show the result. It works like a conversion, so when you choose milimeters from the first list and meters from the second it ...
- 71
0votes
1answer
35views
Average and conversion order of operation
I have a data that I need to analyse.
The data consists of a series of numbers (floating point) represent duration in milliseconds.
From the duration I need to calculate frequency of those events (...
- 591
0votes
0answers
59views
C++ Why is it not possible to use all the range of floating point numbers?
I am trying to understand up to which point I can add or substract floating point numbers that are very far apart (e.g. 3.34e-20 + 1.22) with a correct answer. I ran this code:
#include <iostream&...
- 143
0votes
1answer
83views
Convert very small number to larger positive value in Python
I have numbers including very small numbers like [1e-30, 1e-22, 1e-18, 1, 10].
I want to convert them larger value than 0 and current numbers like [1.1, 1.3, 2.5, 10.0, 11.0]. (the value can be ...
- 91
0votes
0answers
119views
Floating point addition / subtraction
I get confused because of the hidden bit in the mantissa.
From what i know:
Subtract the two exponent, find the smaller number and shift the mantissa with the hidden bit (?) by the result of the ...
- 37
0votes
0answers
63views
How to find the order of floating point operations(additions), given a list of floats and the sum of the list
Introduction
Floating point operations are not associative.
This means if you have a list of floats, the sum you will get, depends on the order of the operations.
I am wondering whether someone ...
- 739
0votes
0answers
53views
What's the easiest way to switch between floating point types?
I'm dealing with some calculations and would like to see which results will be obtained with float or long double instead of the default double precision. It should be costless way, without side ...
- 65
0votes
1answer
56views
How to find magic multipliers for divisions by constant on a GPU?
I was looking at implementing the following computation, where divisor is nonzero and not a power of two
unsigned multiplier(unsigned divisor)
{
unsigned shift = 31 - clz(divisor);
uint64_t t =...
- 674
0votes
1answer
18views
Scaling down block drawing without one block fighting to be two things
The code I lost in a hard drive failure however over a year later it still bugs me about it and I really want an answer so I can sleep at night without this popping into my head.
So I was writing a ...
- 111
0votes
1answer
70views
Strange result using long double in C++
Using long double arithmetic in C++, the number 50,000,056,019,485.52605438232421875 squared yields 2,500,005,601,951,690,788,240,883,712. Meanwhile, the number 50,000,056,019,485.526050567626953125 (...
- 1
0votes
1answer
16views
I cannot perform math operation on data in qtablewidget cells
My OS - Fedora
QtCreator Qt4.8
My Qtablewidget has floats in 50 rows in column 2
I am trying to sum the values in those 50 cells, my return local is 'nan' I'm puzzled.
the data looks like this which ...
- 17
0votes
2answers
57views
I am not able to compare two float number in MATLAB it is giving zero because of very small change how to avoid this
x=20;
LHS1=(sind(x)+cosd(x))^2
RHS1=1+2*sind(x)*cosd(x)
LHS1==RHS1
LHS2=(1-2*cosd(x)-3*(cosd(x))^2)/(sind(x))^2
RHS2=(1-3*cosd(x))/(1-cosd(x))
LHS2==RHS2
I am getting this answer as
LHS1 =
1....
0votes
0answers
126views
Python performance: square roots and hot-loop operation strategy
In comparing math.sqrt(…) to the built-in binary operator, … ** 0.5 in a REPL using timeit.timeit(…), it looks as though the binary op has a slight edge over the math module function:
… although I ...
- 4,149
0votes
0answers
126views
Will someone ever be faced with wrong results of simple arithmetic operations on floating point numbers if using rounding?
Note: numbers which I'm talking about are currency, thus don't have more than two digits in fractional part.
I've tested 4 libraries (Decimal.js, Numeral.js, Big.js and Math.js) and a simple plain ...
- 8,108
0votes
0answers
233views
Keras elementwise variable multiplication error?
I am experimenting with Keras variable to build a custom loss function and I stumbled in a strange behavior.
Let's take this elementwise operation in np.arrays
np_yt = np.arange(10)/10
np_yw = np....
- 197
0votes
1answer
92views
Multiplication by %s when language dictates division rounds down by default
In erlang:
cost(I, Miners) ->
BasePrice = lists:nth(I, prices()),
Owned = lists:nth(I, Miners),
Rate = increaseRate(I) / 100,
Multiplier = math:pow((1 + Rate), Owned),
floor(...
- 9,278
0votes
0answers
82views
Simplest approach to make a calculator?
I've read other questions on this and they're not quite as general as I want this discussion to be.
My question is as stated: What's the simplest approach to making a basic calculator, say with ...
- 310
0votes
2answers
65views
Given two data sets with 3 columns, extract the rows for which the value of the third column is almost equal in both data sets
Given these two data sets:
data_set_1.txt: https://pastebin.com/VQrsv8kU
data_set_2.txt: https://pastebin.com/jFBa6kZq
(It is possible to download these without registering)
I am trying to obtain ...
- 629
0votes
1answer
53views
PHP Find the first scientificly significant 'non zero' digits of an floating number
In this question, I am using the word significant in a scientific context
I want to retrieve the first n digits from the left of a floating number. The function should take in:
the $number which is ...
- 431
0votes
1answer
51views
How should i figure out the exact meaning of such sentence?
There is a confusing description of built-in function divmod(),i post below:
If x is very close to an exact integer multiple of y, it’s possible for x//y to be one larger than (x-x%y)//y due to ...
- 745
0votes
0answers
258views
Float Precision in the Visual Studio Immediate Window
Undertaking a code review I have come across some strange behaviour I don't understand in the C# immediate window.
all the variables below are floats and calculate the gradient of a line. ...
- 2,950
0votes
1answer
43views
Dividing float numbers with 2 decimal places vs same float numbers with 6 decimal places
We have 2 float numbers with 6 decimal places. We are dividing them and showing the result on our web page. We are also showing the 2 numbers we used for division on the web page. As those 2 numbers ...
- 33
0votes
1answer
83views
Add 2 ints and compare to a float
I have 2 numbers:
int LIMIT1 = 20;
int LIMIT2 = 10;
int a, b;
float c;
The threshold is defined as final float threshold= Math.round(a * b / c)
The following code works:
if(threshold ...
- 17.8k
0votes
0answers
1kviews
MIPS Input and output help Floating point
Could someone help me get the odds of my lottery program to output the answer. my program is supposed to take two inputs: one being the amount of balls in the basket (i.e. 59 balls) and two being the ...
- 83
0votes
1answer
238views
Calculating maximum float value
I want to calculate maximum float value (lets say f64) iteratively. I was thinking of multiplying some number like 1.0 times 2 until it reaches infinity, but it won't give me an exact number. ...
- 1
0votes
1answer
810views
Non-Restoring Division for Floating Point
I have found details about the non-restoring division algorithm, but from what I found it assumes that the dividend is greater than the divisor. Does this have to be true?
I am asking because I want ...
- 3,410
0votes
0answers
329views
Newton-Raphson Division and Square Root Different Expressions for Different Precision
I am trying to compute the floating-point square root of x using assembly code using the newton-raphson method for first finding the inverse square root (1/sqrt(x)) and then multiplying by x to find ...
- 3,410
0votes
3answers
774views
Errors multiplying large doubles
I've made a BOMDAS calculator in C++ that uses doubles. Whenever I input an expression like
1000000000000000000000*1000000000000000000000
I get a result like ...
- 39
0votes
2answers
146views
Understanding machine precision based on bit representation
I am trying to understand how this is calculated:
A computer program that represents numbers as follows: 1 bit for the overall
sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...
