# All Questions

Tagged with
58 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
62views

### Dealing with floating point point inaccuracy in very small numbers efficiently

The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data. I am drawing lines, that sometimes exceed these bounds ...
124views

### Javascript wrongfully changes the result of a simple multiplication. How can I fix it?

function roundUp(num, precision) { return Math.ceil(num * precision) / precision; } var num = 0.07; var precision = 100; console.log(roundUp(num, precision)); When the arguments to the ...
120views

### Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
175views

### How can I find out if my JVM is using hardware square root?

If you read the top of the incredibly-hard-to-find native sqrt method for Java, which is located at jdk1.6\src\jdk\src\share\native\java\lang\fdlibm\src\e_sqrt.c you will find this: /* __ieee754_sqrt(...
1vote
51views

### Is Java floating arithmetic division always within 1 ULP of the true result?

Is Java's floating arithmetic division always within 1 ULP of the true result? I read that CPUs sometimes do floating-point division a/b by doing a * 1/b. However, 1/b may be off by 1 ULP, and ...
1vote
60views

### How to deterministically divide floats when there is a known deviation from IEEE 754?

DirectX 11 allows GPU manufacturers to deviate from the rounding behavior specified in the IEEE 754 standard. I cannot enable IEEE strictness, because I don't control the shader compilation process. ...
1vote
423views

### 'round()' vs 'std::round()' and 'fabs()' vs 'std::fabs()' in C++ / GCC 4.8

By accident I was calling round() and fabs() instead of std::round() and std::fabs() and for the largest integer a long double can hold without loosing precision there was a difference. Consider this ...
1vote
55views

### How to implement GNUMP in this Bernoulli numbers algortihm I wrote in C?

I wrote this C algortihm to calculate Bernoulli numbers when the input is Nth, but it only works until 136th because of the floating-point limitations of C. I'm not a programmer, more a mathematician, ...
1vote
39views

### How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding? For instance: import math def truncate(f, n): return math.floor(f * 10 ** n) / 10 ** n r = lambda f,p: f - f % p size = 3245.249999999999853146309903 y ...
1vote
46views

### Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
1vote
90views

### Does cpp_dec_float give consistent results across platforms?

Can a custom number type such as cpp_dec_float (from Boost) guarantee a consistent floating-point arithmetic behavior (on trigonometric functions such as cos, sin, tan, atan) across different system ...
1vote
23views

### Interpreting parameter values from scipy stat and popt

Really dumb question, but I am new to python and my math is super rusty. I ran a function using the curve_fit function in python and got a list of parameter my parameter for my A and B variable came ...
1vote
69views

### How to know the effective input domain of special function in SciPy or GSL?

I need to know the effective input domain of special functions in SciPy (Link to special functions of SciPy) or GSL(Link to special functions of GSL). For example, for the function gamma, the ...
1vote
88views

### Map float to a partition possible in constant time?

Suppose I have a partition (where each partition is a continuous range) over a range of floats: R = [a_0, a_n) P(R) = [a_0, a_1), [a_1, a_2), ... , [a_n-1, a_n) For any given float f in R, is there ... 1vote
91views

### Error-free sum in F#

I'm continuing with a port of GeographicLib into F#, and I'm wondering about the use of the error-free sum. In the C++ codebase, it is defined as /** * The error-free sum of two numbers. * * @...
1vote
90views

### Why does matlab use the following interpolation function?

If you try performing linear interpolation in matlab, it uses a "gridded" interpolant apparently. Normally, I would expect the linear interpolation function to look something like this: METHOD A: ...
1vote
231views

### How to scale down blocks without causing pixel overlapping

I have a bunch of blocks that needs to be drawn into a grid. Now displaying them unscaled everything is fine but when I try to scale them down to fit withing a window I get "scale-artifacts" because I ...
1vote
609views

### comparing floats precisely after a cross product operation

This is about performing the following comparison precisely : CrossProduct(P - P0, P1 - P0) >, < or == 0 Problem definition Given a set of grid points described by the following equation P(x,...
37views

### Fast inverse square root in python on float32

I have done some checks for the fast inverse square root method in python (jupyterlab using python version 3.8.8) and for some reason then I've come to the conclusion that I must either be doing ...
52views

### Inconsistent results with floating point arithmetic (color conversion formulae)

I am trying to parse a bunch of color values from various files, perform lightness changes, and write them back into the files. I have to put this in a PKGBUILD for the Arch User Repository and would ...
46views

### Increasing precision for decimal part from 8 decimals to 18 decimals using Uint 128 bit

I am trying to write the translation of unsigned integers to floating points with precision of up to 18 decimals. The language (Clarity) that I am using doesn't have floating points in it so I have to ...
39views

### Floating points in big numbers from a select list (js)

I have 2 select lists in html, place to enter a number and second place to show the result. It works like a conversion, so when you choose milimeters from the first list and meters from the second it ...
35views

### Average and conversion order of operation

I have a data that I need to analyse. The data consists of a series of numbers (floating point) represent duration in milliseconds. From the duration I need to calculate frequency of those events (...
59views

### C++ Why is it not possible to use all the range of floating point numbers?

I am trying to understand up to which point I can add or substract floating point numbers that are very far apart (e.g. 3.34e-20 + 1.22) with a correct answer. I ran this code: #include <iostream&...
83views

### Convert very small number to larger positive value in Python

I have numbers including very small numbers like [1e-30, 1e-22, 1e-18, 1, 10]. I want to convert them larger value than 0 and current numbers like [1.1, 1.3, 2.5, 10.0, 11.0]. (the value can be ...
119views

### Floating point addition / subtraction

I get confused because of the hidden bit in the mantissa. From what i know: Subtract the two exponent, find the smaller number and shift the mantissa with the hidden bit (?) by the result of the ...
63views

### How to find the order of floating point operations(additions), given a list of floats and the sum of the list

Introduction Floating point operations are not associative. This means if you have a list of floats, the sum you will get, depends on the order of the operations. I am wondering whether someone ...
53views

### What's the easiest way to switch between floating point types?

I'm dealing with some calculations and would like to see which results will be obtained with float or long double instead of the default double precision. It should be costless way, without side ...
56views

### How to find magic multipliers for divisions by constant on a GPU?

I was looking at implementing the following computation, where divisor is nonzero and not a power of two unsigned multiplier(unsigned divisor) { unsigned shift = 31 - clz(divisor); uint64_t t =...
18views

### Scaling down block drawing without one block fighting to be two things

The code I lost in a hard drive failure however over a year later it still bugs me about it and I really want an answer so I can sleep at night without this popping into my head. So I was writing a ...
70views

### Strange result using long double in C++

Using long double arithmetic in C++, the number 50,000,056,019,485.52605438232421875 squared yields 2,500,005,601,951,690,788,240,883,712. Meanwhile, the number 50,000,056,019,485.526050567626953125 (...
16views

### I cannot perform math operation on data in qtablewidget cells

My OS - Fedora QtCreator Qt4.8 My Qtablewidget has floats in 50 rows in column 2 I am trying to sum the values in those 50 cells, my return local is 'nan' I'm puzzled. the data looks like this which ...
57views

### I am not able to compare two float number in MATLAB it is giving zero because of very small change how to avoid this

x=20; LHS1=(sind(x)+cosd(x))^2 RHS1=1+2*sind(x)*cosd(x) LHS1==RHS1 LHS2=(1-2*cosd(x)-3*(cosd(x))^2)/(sind(x))^2 RHS2=(1-3*cosd(x))/(1-cosd(x)) LHS2==RHS2 I am getting this answer as LHS1 = 1....
126views

### Python performance: square roots and hot-loop operation strategy

In comparing math.sqrt(…) to the built-in binary operator, … ** 0.5 in a REPL using timeit.timeit(…), it looks as though the binary op has a slight edge over the math module function: … although I ...
126views

### Will someone ever be faced with wrong results of simple arithmetic operations on floating point numbers if using rounding?

Note: numbers which I'm talking about are currency, thus don't have more than two digits in fractional part. I've tested 4 libraries (Decimal.js, Numeral.js, Big.js and Math.js) and a simple plain ...
233views

### Keras elementwise variable multiplication error?

I am experimenting with Keras variable to build a custom loss function and I stumbled in a strange behavior. Let's take this elementwise operation in np.arrays np_yt = np.arange(10)/10 np_yw = np....
92views

### Multiplication by %s when language dictates division rounds down by default

In erlang: cost(I, Miners) -> BasePrice = lists:nth(I, prices()), Owned = lists:nth(I, Miners), Rate = increaseRate(I) / 100, Multiplier = math:pow((1 + Rate), Owned), floor(...
82views

### Simplest approach to make a calculator?

I've read other questions on this and they're not quite as general as I want this discussion to be. My question is as stated: What's the simplest approach to making a basic calculator, say with ...
65views

### Given two data sets with 3 columns, extract the rows for which the value of the third column is almost equal in both data sets

Given these two data sets: data_set_1.txt: https://pastebin.com/VQrsv8kU data_set_2.txt: https://pastebin.com/jFBa6kZq (It is possible to download these without registering) I am trying to obtain ...
53views

### PHP Find the first scientificly significant 'non zero' digits of an floating number

In this question, I am using the word significant in a scientific context I want to retrieve the first n digits from the left of a floating number. The function should take in: the \$number which is ...
51views

### How should i figure out the exact meaning of such sentence?

There is a confusing description of built-in function divmod(),i post below: If x is very close to an exact integer multiple of y, it’s possible for x//y to be one larger than (x-x%y)//y due to ...
258views

### Float Precision in the Visual Studio Immediate Window

Undertaking a code review I have come across some strange behaviour I don't understand in the C# immediate window. all the variables below are floats and calculate the gradient of a line. ...
43views

### Dividing float numbers with 2 decimal places vs same float numbers with 6 decimal places

We have 2 float numbers with 6 decimal places. We are dividing them and showing the result on our web page. We are also showing the 2 numbers we used for division on the web page. As those 2 numbers ...
83views

### Add 2 ints and compare to a float

I have 2 numbers: int LIMIT1 = 20; int LIMIT2 = 10; int a, b; float c; The threshold is defined as final float threshold= Math.round(a * b / c) The following code works: if(threshold ...
1kviews

### MIPS Input and output help Floating point

Could someone help me get the odds of my lottery program to output the answer. my program is supposed to take two inputs: one being the amount of balls in the basket (i.e. 59 balls) and two being the ...
238views

### Calculating maximum float value

I want to calculate maximum float value (lets say f64) iteratively. I was thinking of multiplying some number like 1.0 times 2 until it reaches infinity, but it won't give me an exact number. ...
810views

### Non-Restoring Division for Floating Point

I have found details about the non-restoring division algorithm, but from what I found it assumes that the dividend is greater than the divisor. Does this have to be true? I am asking because I want ...
329views

### Newton-Raphson Division and Square Root Different Expressions for Different Precision

I am trying to compute the floating-point square root of x using assembly code using the newton-raphson method for first finding the inverse square root (1/sqrt(x)) and then multiplying by x to find ...
774views

### Errors multiplying large doubles

I've made a BOMDAS calculator in C++ that uses doubles. Whenever I input an expression like 1000000000000000000000*1000000000000000000000 I get a result like ...