All Questions

194 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
610views

problem with vulkan floating point behavior

I try to implement the paper Extended-Precision Floating-Point Numbers for GPU Computation by Andrew Thall, Alma College in a GLSL Vulkan compute shader. I need this because some of my devices don't ...
• 111
62views

Dealing with floating point point inaccuracy in very small numbers efficiently

The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data. I am drawing lines, that sometimes exceed these bounds ...
85views

R not working properly with big numbers because of default options?

Seems like R coerces big numbers and cannot compare them effectively: x = 123412415124231251233213 x == 123412415124231251233214 [1] TRUE x == 123412415124231251233217 [1] TRUE Any idea why (maybe a ...
• 408
339views

First (smallest) even number not representable with IEEE 754 floating point?

I'm not sure how to go about this problem. I know that the smallest integer not representable by IEEE 754 would be 2^(mantissa+1) + 1 but how would I take that info and change it to an even number? ...
• 2,429
232views

What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
• 18.1k
105views

What is a significance of 9.536743e-7?

I realized that outputs of a continuous function (takes a vector, returns a scalar) that I wrote in python are discretized at the resolution of 9.536743e-7. I googled this number and learned that some ...
• 121
86views

python search float return wrong because of precision problem?

I want to find a float in a arry like this: arr = np.asarray([1351.1 , 1351.11, 1351.14, 1351.16, 1351.17]) index = np.searchsorted(arr, 1351.14, side="right") - 1 # return 2 But I find ...
• 7,635
99views

Is precision of pow in C dependent on scale of parameters?

Legacy code I am to maintain has a function with a variable x of type double. This variable contains a position measured in meters. It is expected to hold a value in the mm range, let's say 0.0001 <...
• 385
1kviews

Understanding sys.float_info for maximum floats in python

I'm trying to understand the information given by sys.float_info to understand what the maximum floats in Python are. On my computer, this gives me the following: >>> import sys >>> ...
1kviews

How to get the maximum number of decimal places in each column of a pandas dataframe?

I need to set the global float precision to the minimum value possible Also, I need to get the precision for each column, in part to get the global precision and on the other hand, I would like to ...
• 8,950
52views

How much precision is needed to represent n/2^k, given a boundary of -2 to +2?

Apologies for the poorly phrased title. I tried to do better, but perhaps someone can suggest a better title. As a test case for another project, I'm working with the Mandelbrot set, using only value ...
• 527
100views

Is there a way for Python to read __float128 from a binary file?

I am trying to read a binary file that stores some __float128 variables from other C project using Python. I've been looking in the numpy.float128 but it seems that it is implemented in a different ...
95views

Is there options to set on Visual Studio to keep deterministic floating-point results between different multi-thread setup?

I am working with OpenMp with VS2017 in order to compute our statistics with multi-thread process. But, for few values, we found out that results can be differents with the number of threads set. For ...
568views

Casting from (long)double to size_t

I'm trying to implement Sieve of Eratosthenes as efficient as possible. I'd like to set length of my prime array to upper bond of pi(n) < 1.25506n / ln n but I'm not sure how to proceed with ...
• 177
205views

Floating-point precision problems

Using Microsoft Excel 2010. Why do these two formulae produce different results? = (0.5 - 0.4 - 0.1) produces -2.775E-17 while = 0.5 - 0.4 - 0.1 produces exactly 0. The only difference is in the ...
• 5,570
657views

What MySQL "DECIMAL" precision and scale is required to store a C# decimal value?

I want to store some decimal values in my MySQL database. The obvious MySQL database type for that would be DECIMAL I guess. What precision "a" and scale "b" do I have to set for DECIMAL(a,b ) ...
• 5,504
253views

How to evaluate an alternating series when addends contain rounding errors?

I want to numerically evaluate the transition probability of a linear Birth and Death process where is the binomial coefficient and I am able to evaluate it with an acceptable numerical error (...
118views

Finding smallest float x such that x+d=y

How to find the smallest float/double number x which satisfies x + d = y given d and y? (iiuc this is theoretically solved via setting fesetround (FE_DOWNWARD) and just doing y - d but in clang/Xcode ...
• 22.2k
39views

precison loss in floating point multiply

The following mentions http://floating-point-gui.de/errors/propagation/ that floating point multiply and divide are safe. Empirically testing across various chains of multiplication also suggest that ...
565views

How to print scientific notation with all significant digits moved after decimal point in python

I am trying to print scientific numbers with all the significant digits after the decimal point. ` Python's scientific notation prints with one digit before the dot like 1.2345678E+03, '%E' % ...
• 494
106views

How can I specify tolerance for floating point operation in CGAL library?

Is there a way to specify tolerance for CGAL geometry operations? I use exact_predicates_inexact_constructions kernel for computational speed. However sometimes operations such as determining a point ...
• 791
59views

Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x[0] < x[1] < x[2] < ... &...
• 1,945
120views

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
• 19.5k
99views

Floating Point Number issue JavaScript

I'm struggling to resolve a Floating Point Number issue where var change returns as 0.0999 recurring, and i need to return 0.01 (one penny). The code works fine, except the very last penny, because of ...
422views

16-bit floating point on fpga

I try to use Altera's floating point IP to generate half precision instead of single (32-bit) blocks for addition , multiplication etc. However when configuring the IP it seems that half precision fp ...
• 157
234views

Can computing tan(x)=sin(x)/cos(x) cause a loss of precision?

After a call to sincos(x,&s,&c) from the unix m math library it would be natural to get the tangent as s/c. Is this safe or there may be (ill) cases in which the (supposedly) more expensive ...
• 5,175
136views

Using Matlab to read binary data from file

I'm currently trying to read data from a .surf file using Matlab. (I realise there will probably be quite a lot of other questions similar to this, but each is specific to its own problem, and I wasn'...
• 358
146views

Why are these two calculations which are exactly same giving different results in Fortran using gfortran?

real, dimension(3), parameter :: boxlen = [4.0, 5.0, 7.0] real, parameter :: mindist = 0.1 integer ::i write(*,"(A)") "Operation on array" print*, floor(boxlen/mindist) write(*,"(/A)") "...
• 384
668views

numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
• 6,788
94views

For this code segment double count = 0.0; while( count != 1.0) {count += 1.0/3;} I was wondering what parts of (IEEE 754) would cause count 3: to be 1.0 instead of .9999999999999999. I realize that ...
1vote
198views

Exact same WebGL code renders differently on different devices

I'm fairly new to shaders, just started playing with them in VisionsOfChaos for Android around a year ago. I implemented my 4D fractal viewer in it (as far as I know, a completely new approach, which ...
• 1,652
1vote
60views

Python is automatically rounding large numbers with decimals

I need to perform the following calculation in Python: (10^16) + (9 * 54945054708534.5) The output it prints is: 5494505492376810.0 but this is incorrect as the solution should end in 0.5. I have ...
1vote
67views

JavaScript: parseFloat() strips trailing zeros

I have been trying to find a solution to the following problem. I have a string that is a floating-point number like, var value = '12.30'; When I try to cast it to be a floating number parseFloat('...
• 1,824
1vote
58views

floating point arithmetics: interpolating two half-precision floating point values with one full-precision interpolation

let's say I have two IEEE 32bit floating point values a and b plus one IEEE 32bit fp interpolation value x (something between 0 and 1) so I get the interpolated result: float a = 123.456f; float b = ...
• 4,250
1vote
79views

Why do I get (slightly) different results everytime I run the same query on Google Bigquery?

I have a query on BQ that that produces slightly different results everytime I run it. Not big differences, e.g. 4421.10399999999 vs 4421.104 which looks like it is simply inconsistent rounding, but I ...
• 69
1vote
215views

Pandas Float Precision - Apparently Identical Numbers Showing as Not Equal

Hopefully a very simple solution to this. I have tried the solutions for two similar questions on SO, but these haven't worked for me. Essentially I have a process which evaluates whether numbers in ...
1vote
46views

Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
1vote
68views

Determine precision of number, including trailing zeros

I suspect that this isn't possible, but giving it a try: Is it possible to determine the number of decimal points of a number, whether or not those decimal points are trailing zeros? getPrecision(3) ...
• 1,649
1vote
1kviews

How to create a linspace like pytorch tensor which ensures that all values are between start and stop?

I want to create a 1d tensor of values similar to torch.linspace or torch.arange in pytorch, which ensures that all values of the created tensor are between start and stop. Example values which break ...
• 601
1vote
84views

Why there's no precision error when printing 0.1 as a double with many digits after point?

The function (written in Kotlin) below fun main() { println ("%.20f".format(0.1f)) print ("%.100f".format(0.1)) } prints 0.10000000149011612000 0....
• 1,681
1vote
140views

Float equality comparison using epsilon and hash code in Swift?

I want to create Set of floats, compared using epsilon. However if some values can produce different hash codes, but be equal is you compare them this way: func isEqual(to other: Float, usingEpsilon ...
• 3,500
1vote
141views

How to find accuracy of matrix multiplication with floating-point numbers?

I am trying to analyze how floating-point computation becomes more inaccurate when the data size decreases. In order to do that, I wanted to perform simple matrix operations on different variations of ...
1vote
411views

Node Buffer.readFloatBE() extends data to 64-bit

When converting data to/from a Buffer using 32-bit floating point numbers, it seems that nodejs extends the float to double using some criteria that makes the transformation not reversible: > f=3....
• 673
1vote
123views

How to print glm vector with manually specified precion?

I know about glm::to_string but I cannot find away to set precision of the string conversion. A few issues come up because of this. If I want to serialize glm structures, I won't be able to store ...
• 3,881
1vote
162views

How to preserve the accuracy of a floating point literal in Rust when using generics?

What's a reasonable strategy for preserving the accuracy of a floating point literal in Rust when using generics? As an example, consider the following code: // External functions use std::fmt::...
• 5,670
1vote
248views

Conversion Precision Error when converting IEE Half Precision Floating Point to Decimal

I have some precision error during the conversion from 16 bit half precision floating point format to decimal. It is able to accurately convert certain numbers while at the same time not accurate for ...
• 31
1vote
59views

How can we work on fixed amount of precision that is less than double but more than float?

I'm just trying to do something. There is a some computation to be done using double-type variables. We have two parameters "x" and "y" where we need to store log2(x) with y-bit precision. First ...
• 458
1vote
187views

Key-space analysis of an algorithm

I have an algorithm that requires a set of keys to encrypt a text (image of size N x N). It is an image encryption algorithm for a project. The key can be thought of as a tuple x_0, lambda, d, i, j, ...
• 192
1vote
885views

truncate float values to 8 decimal places with pandas.to_excel

I have a data frame with columns containing float values. One of the cells contain a value of 0.021893287 I have set the pandas option to: pd.set_option('precision', 8) When I try to export the ...
1vote