All Questions

Tagged with
58 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3votes
1answer
62views

Dealing with floating point point inaccuracy in very small numbers efficiently

The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data. I am drawing lines, that sometimes exceed these bounds ...
user avatar
3votes
1answer
124views

Javascript wrongfully changes the result of a simple multiplication. How can I fix it?

function roundUp(num, precision) { return Math.ceil(num * precision) / precision; } var num = 0.07; var precision = 100; console.log(roundUp(num, precision)); When the arguments to the ...
user avatar
  • 1,154
2votes
0answers
120views

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
user avatar
2votes
0answers
175views

How can I find out if my JVM is using hardware square root?

If you read the top of the incredibly-hard-to-find native sqrt method for Java, which is located at jdk1.6\src\jdk\src\share\native\java\lang\fdlibm\src\e_sqrt.c you will find this: /* __ieee754_sqrt(...
user avatar
  • 10.6k
1vote
1answer
51views

Is Java floating arithmetic division always within 1 ULP of the true result?

Is Java's floating arithmetic division always within 1 ULP of the true result? I read that CPUs sometimes do floating-point division a/b by doing a * 1/b. However, 1/b may be off by 1 ULP, and ...
user avatar
1vote
0answers
60views

How to deterministically divide floats when there is a known deviation from IEEE 754?

DirectX 11 allows GPU manufacturers to deviate from the rounding behavior specified in the IEEE 754 standard. I cannot enable IEEE strictness, because I don't control the shader compilation process. ...
user avatar
  • 2,253
1vote
1answer
423views

'round()' vs 'std::round()' and 'fabs()' vs 'std::fabs()' in C++ / GCC 4.8

By accident I was calling round() and fabs() instead of std::round() and std::fabs() and for the largest integer a long double can hold without loosing precision there was a difference. Consider this ...
user avatar
  • 3,684
1vote
0answers
55views

How to implement GNUMP in this Bernoulli numbers algortihm I wrote in C?

I wrote this C algortihm to calculate Bernoulli numbers when the input is Nth, but it only works until 136th because of the floating-point limitations of C. I'm not a programmer, more a mathematician, ...
user avatar
1vote
0answers
39views

How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding? For instance: import math def truncate(f, n): return math.floor(f * 10 ** n) / 10 ** n r = lambda f,p: f - f % p size = 3245.249999999999853146309903 y ...
user avatar
1vote
0answers
46views

Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
user avatar
1vote
0answers
90views

Does cpp_dec_float give consistent results across platforms?

Can a custom number type such as cpp_dec_float (from Boost) guarantee a consistent floating-point arithmetic behavior (on trigonometric functions such as cos, sin, tan, atan) across different system ...
user avatar
  • 1,749
1vote
0answers
23views

Interpreting parameter values from scipy stat and popt

Really dumb question, but I am new to python and my math is super rusty. I ran a function using the curve_fit function in python and got a list of parameter my parameter for my A and B variable came ...
user avatar
1vote
0answers
69views

How to know the effective input domain of special function in SciPy or GSL?

I need to know the effective input domain of special functions in SciPy (Link to special functions of SciPy) or GSL(Link to special functions of GSL). For example, for the function gamma, the ...
user avatar
  • 59
1vote
0answers
88views

Map float to a partition possible in constant time?

Suppose I have a partition (where each partition is a continuous range) over a range of floats: R = [a_0, a_n) P(R) = [a_0, a_1), [a_1, a_2), ... , [a_n-1, a_n) For any given float f in R, is there ...
user avatar
1vote
0answers
91views

Error-free sum in F#

I'm continuing with a port of GeographicLib into F#, and I'm wondering about the use of the error-free sum. In the C++ codebase, it is defined as /** * The error-free sum of two numbers. * * @...
user avatar
  • 11.3k
1vote
1answer
90views

Why does matlab use the following interpolation function?

If you try performing linear interpolation in matlab, it uses a "gridded" interpolant apparently. Normally, I would expect the linear interpolation function to look something like this: METHOD A: ...
user avatar
  • 3,410
1vote
2answers
231views

How to scale down blocks without causing pixel overlapping

I have a bunch of blocks that needs to be drawn into a grid. Now displaying them unscaled everything is fine but when I try to scale them down to fit withing a window I get "scale-artifacts" because I ...
user avatar
1vote
3answers
609views

comparing floats precisely after a cross product operation

This is about performing the following comparison precisely : CrossProduct(P - P0, P1 - P0) >, < or == 0 Problem definition Given a set of grid points described by the following equation P(x,...
user avatar
0votes
0answers
37views

Fast inverse square root in python on float32

I have done some checks for the fast inverse square root method in python (jupyterlab using python version 3.8.8) and for some reason then I've come to the conclusion that I must either be doing ...
user avatar
  • 1
0votes
0answers
52views

Inconsistent results with floating point arithmetic (color conversion formulae)

I am trying to parse a bunch of color values from various files, perform lightness changes, and write them back into the files. I have to put this in a PKGBUILD for the Arch User Repository and would ...
user avatar
0votes
0answers
46views

Increasing precision for decimal part from 8 decimals to 18 decimals using Uint 128 bit

I am trying to write the translation of unsigned integers to floating points with precision of up to 18 decimals. The language (Clarity) that I am using doesn't have floating points in it so I have to ...
user avatar
0votes
1answer
39views

Floating points in big numbers from a select list (js)

I have 2 select lists in html, place to enter a number and second place to show the result. It works like a conversion, so when you choose milimeters from the first list and meters from the second it ...
user avatar
  • 71
0votes
1answer
35views

Average and conversion order of operation

I have a data that I need to analyse. The data consists of a series of numbers (floating point) represent duration in milliseconds. From the duration I need to calculate frequency of those events (...
user avatar
  • 591
0votes
0answers
59views

C++ Why is it not possible to use all the range of floating point numbers?

I am trying to understand up to which point I can add or substract floating point numbers that are very far apart (e.g. 3.34e-20 + 1.22) with a correct answer. I ran this code: #include <iostream&...
user avatar
  • 143
0votes
1answer
83views

Convert very small number to larger positive value in Python

I have numbers including very small numbers like [1e-30, 1e-22, 1e-18, 1, 10]. I want to convert them larger value than 0 and current numbers like [1.1, 1.3, 2.5, 10.0, 11.0]. (the value can be ...
user avatar
  • 91
0votes
0answers
119views

Floating point addition / subtraction

I get confused because of the hidden bit in the mantissa. From what i know: Subtract the two exponent, find the smaller number and shift the mantissa with the hidden bit (?) by the result of the ...
user avatar
  • 37
0votes
0answers
63views

How to find the order of floating point operations(additions), given a list of floats and the sum of the list

Introduction Floating point operations are not associative. This means if you have a list of floats, the sum you will get, depends on the order of the operations. I am wondering whether someone ...
user avatar
  • 739
0votes
0answers
53views

What's the easiest way to switch between floating point types?

I'm dealing with some calculations and would like to see which results will be obtained with float or long double instead of the default double precision. It should be costless way, without side ...
user avatar
0votes
1answer
56views

How to find magic multipliers for divisions by constant on a GPU?

I was looking at implementing the following computation, where divisor is nonzero and not a power of two unsigned multiplier(unsigned divisor) { unsigned shift = 31 - clz(divisor); uint64_t t =...
user avatar
  • 674
0votes
1answer
18views

Scaling down block drawing without one block fighting to be two things

The code I lost in a hard drive failure however over a year later it still bugs me about it and I really want an answer so I can sleep at night without this popping into my head. So I was writing a ...
user avatar
  • 111
0votes
1answer
70views

Strange result using long double in C++

Using long double arithmetic in C++, the number 50,000,056,019,485.52605438232421875 squared yields 2,500,005,601,951,690,788,240,883,712. Meanwhile, the number 50,000,056,019,485.526050567626953125 (...
user avatar
  • 1
0votes
1answer
16views

I cannot perform math operation on data in qtablewidget cells

My OS - Fedora QtCreator Qt4.8 My Qtablewidget has floats in 50 rows in column 2 I am trying to sum the values in those 50 cells, my return local is 'nan' I'm puzzled. the data looks like this which ...
user avatar
0votes
2answers
57views

I am not able to compare two float number in MATLAB it is giving zero because of very small change how to avoid this

x=20; LHS1=(sind(x)+cosd(x))^2 RHS1=1+2*sind(x)*cosd(x) LHS1==RHS1 LHS2=(1-2*cosd(x)-3*(cosd(x))^2)/(sind(x))^2 RHS2=(1-3*cosd(x))/(1-cosd(x)) LHS2==RHS2 I am getting this answer as LHS1 = 1....
user avatar
0votes
0answers
126views

Python performance: square roots and hot-loop operation strategy

In comparing math.sqrt(…) to the built-in binary operator, … ** 0.5 in a REPL using timeit.timeit(…), it looks as though the binary op has a slight edge over the math module function: … although I ...
user avatar
  • 4,149
0votes
0answers
126views

Will someone ever be faced with wrong results of simple arithmetic operations on floating point numbers if using rounding?

Note: numbers which I'm talking about are currency, thus don't have more than two digits in fractional part. I've tested 4 libraries (Decimal.js, Numeral.js, Big.js and Math.js) and a simple plain ...
user avatar
  • 8,108
0votes
0answers
233views

Keras elementwise variable multiplication error?

I am experimenting with Keras variable to build a custom loss function and I stumbled in a strange behavior. Let's take this elementwise operation in np.arrays np_yt = np.arange(10)/10 np_yw = np....
user avatar
  • 197
0votes
1answer
92views

Multiplication by %s when language dictates division rounds down by default

In erlang: cost(I, Miners) -> BasePrice = lists:nth(I, prices()), Owned = lists:nth(I, Miners), Rate = increaseRate(I) / 100, Multiplier = math:pow((1 + Rate), Owned), floor(...
user avatar
0votes
0answers
82views

Simplest approach to make a calculator?

I've read other questions on this and they're not quite as general as I want this discussion to be. My question is as stated: What's the simplest approach to making a basic calculator, say with ...
user avatar
0votes
2answers
65views

Given two data sets with 3 columns, extract the rows for which the value of the third column is almost equal in both data sets

Given these two data sets: data_set_1.txt: https://pastebin.com/VQrsv8kU data_set_2.txt: https://pastebin.com/jFBa6kZq (It is possible to download these without registering) I am trying to obtain ...
user avatar
  • 629
0votes
1answer
53views

PHP Find the first scientificly significant 'non zero' digits of an floating number

In this question, I am using the word significant in a scientific context I want to retrieve the first n digits from the left of a floating number. The function should take in: the $number which is ...
user avatar
0votes
1answer
51views

How should i figure out the exact meaning of such sentence?

There is a confusing description of built-in function divmod(),i post below: If x is very close to an exact integer multiple of y, it’s possible for x//y to be one larger than (x-x%y)//y due to ...
user avatar
  • 745
0votes
0answers
258views

Float Precision in the Visual Studio Immediate Window

Undertaking a code review I have come across some strange behaviour I don't understand in the C# immediate window. all the variables below are floats and calculate the gradient of a line. ...
user avatar
  • 2,950
0votes
1answer
43views

Dividing float numbers with 2 decimal places vs same float numbers with 6 decimal places

We have 2 float numbers with 6 decimal places. We are dividing them and showing the result on our web page. We are also showing the 2 numbers we used for division on the web page. As those 2 numbers ...
user avatar
  • 33
0votes
1answer
83views

Add 2 ints and compare to a float

I have 2 numbers: int LIMIT1 = 20; int LIMIT2 = 10; int a, b; float c; The threshold is defined as final float threshold= Math.round(a * b / c) The following code works: if(threshold ...
user avatar
  • 17.8k
0votes
0answers
1kviews

MIPS Input and output help Floating point

Could someone help me get the odds of my lottery program to output the answer. my program is supposed to take two inputs: one being the amount of balls in the basket (i.e. 59 balls) and two being the ...
user avatar
  • 83
0votes
1answer
238views

Calculating maximum float value

I want to calculate maximum float value (lets say f64) iteratively. I was thinking of multiplying some number like 1.0 times 2 until it reaches infinity, but it won't give me an exact number. ...
user avatar
0votes
1answer
810views

Non-Restoring Division for Floating Point

I have found details about the non-restoring division algorithm, but from what I found it assumes that the dividend is greater than the divisor. Does this have to be true? I am asking because I want ...
user avatar
  • 3,410
0votes
0answers
329views

Newton-Raphson Division and Square Root Different Expressions for Different Precision

I am trying to compute the floating-point square root of x using assembly code using the newton-raphson method for first finding the inverse square root (1/sqrt(x)) and then multiplying by x to find ...
user avatar
  • 3,410
0votes
3answers
774views

Errors multiplying large doubles

I've made a BOMDAS calculator in C++ that uses doubles. Whenever I input an expression like 1000000000000000000000*1000000000000000000000 I get a result like ...
user avatar
0votes
2answers
146views

Understanding machine precision based on bit representation

I am trying to understand how this is calculated: A computer program that represents numbers as follows: 1 bit for the overall sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...
user avatar

15 30 50 per page