# All Questions

194 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
610views

### problem with vulkan floating point behavior

I try to implement the paper Extended-Precision Floating-Point Numbers for GPU Computation by Andrew Thall, Alma College in a GLSL Vulkan compute shader. I need this because some of my devices don't ...
62views

### Dealing with floating point point inaccuracy in very small numbers efficiently

The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data. I am drawing lines, that sometimes exceed these bounds ...
85views

### R not working properly with big numbers because of default options?

Seems like R coerces big numbers and cannot compare them effectively: x = 123412415124231251233213 x == 123412415124231251233214  TRUE x == 123412415124231251233217  TRUE Any idea why (maybe a ...
339views

### First (smallest) even number not representable with IEEE 754 floating point?

I'm not sure how to go about this problem. I know that the smallest integer not representable by IEEE 754 would be 2^(mantissa+1) + 1 but how would I take that info and change it to an even number? ...
232views

### What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
105views

### What is a significance of 9.536743e-7?

I realized that outputs of a continuous function (takes a vector, returns a scalar) that I wrote in python are discretized at the resolution of 9.536743e-7. I googled this number and learned that some ...
86views

### python search float return wrong because of precision problem?

I want to find a float in a arry like this: arr = np.asarray([1351.1 , 1351.11, 1351.14, 1351.16, 1351.17]) index = np.searchsorted(arr, 1351.14, side="right") - 1 # return 2 But I find ...
99views

### Is precision of pow in C dependent on scale of parameters?

Legacy code I am to maintain has a function with a variable x of type double. This variable contains a position measured in meters. It is expected to hold a value in the mm range, let's say 0.0001 <...
1kviews

### Understanding sys.float_info for maximum floats in python

I'm trying to understand the information given by sys.float_info to understand what the maximum floats in Python are. On my computer, this gives me the following: >>> import sys >>> ...
1kviews

### How to get the maximum number of decimal places in each column of a pandas dataframe?

I need to set the global float precision to the minimum value possible Also, I need to get the precision for each column, in part to get the global precision and on the other hand, I would like to ...
52views

### How much precision is needed to represent n/2^k, given a boundary of -2 to +2?

Apologies for the poorly phrased title. I tried to do better, but perhaps someone can suggest a better title. As a test case for another project, I'm working with the Mandelbrot set, using only value ...
100views

### Is there a way for Python to read __float128 from a binary file?

I am trying to read a binary file that stores some __float128 variables from other C project using Python. I've been looking in the numpy.float128 but it seems that it is implemented in a different ...
95views

### Is there options to set on Visual Studio to keep deterministic floating-point results between different multi-thread setup?

I am working with OpenMp with VS2017 in order to compute our statistics with multi-thread process. But, for few values, we found out that results can be differents with the number of threads set. For ...
568views

### Casting from (long)double to size_t

I'm trying to implement Sieve of Eratosthenes as efficient as possible. I'd like to set length of my prime array to upper bond of pi(n) < 1.25506n / ln n but I'm not sure how to proceed with ...
205views

### Floating-point precision problems

Using Microsoft Excel 2010. Why do these two formulae produce different results? = (0.5 - 0.4 - 0.1) produces -2.775E-17 while = 0.5 - 0.4 - 0.1 produces exactly 0. The only difference is in the ...
657views

### What MySQL "DECIMAL" precision and scale is required to store a C# decimal value?

I want to store some decimal values in my MySQL database. The obvious MySQL database type for that would be DECIMAL I guess. What precision "a" and scale "b" do I have to set for DECIMAL(a,b ) ...
253views

### How to evaluate an alternating series when addends contain rounding errors?

I want to numerically evaluate the transition probability of a linear Birth and Death process where is the binomial coefficient and I am able to evaluate it with an acceptable numerical error (...
118views

### Finding smallest float x such that x+d=y

How to find the smallest float/double number x which satisfies x + d = y given d and y? (iiuc this is theoretically solved via setting fesetround (FE_DOWNWARD) and just doing y - d but in clang/Xcode ...
39views

### precison loss in floating point multiply

The following mentions http://floating-point-gui.de/errors/propagation/ that floating point multiply and divide are safe. Empirically testing across various chains of multiplication also suggest that ...
565views

### How to print scientific notation with all significant digits moved after decimal point in python

I am trying to print scientific numbers with all the significant digits after the decimal point. ` Python's scientific notation prints with one digit before the dot like 1.2345678E+03, '%E' % ...
106views

### How can I specify tolerance for floating point operation in CGAL library?

Is there a way to specify tolerance for CGAL geometry operations? I use exact_predicates_inexact_constructions kernel for computational speed. However sometimes operations such as determining a point ...
59views

### Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x < x < x < ... &...
120views

### Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
99views

### Floating Point Number issue JavaScript

I'm struggling to resolve a Floating Point Number issue where var change returns as 0.0999 recurring, and i need to return 0.01 (one penny). The code works fine, except the very last penny, because of ...
422views

### 16-bit floating point on fpga

I try to use Altera's floating point IP to generate half precision instead of single (32-bit) blocks for addition , multiplication etc. However when configuring the IP it seems that half precision fp ...
234views

### Can computing tan(x)=sin(x)/cos(x) cause a loss of precision?

After a call to sincos(x,&s,&c) from the unix m math library it would be natural to get the tangent as s/c. Is this safe or there may be (ill) cases in which the (supposedly) more expensive ...
136views

### Using Matlab to read binary data from file

I'm currently trying to read data from a .surf file using Matlab. (I realise there will probably be quite a lot of other questions similar to this, but each is specific to its own problem, and I wasn'...
146views

### Why are these two calculations which are exactly same giving different results in Fortran using gfortran?

real, dimension(3), parameter :: boxlen = [4.0, 5.0, 7.0] real, parameter :: mindist = 0.1 integer ::i write(*,"(A)") "Operation on array" print*, floor(boxlen/mindist) write(*,"(/A)") "...
668views

### numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
94views

### Floating point addition and division

For this code segment double count = 0.0; while( count != 1.0) {count += 1.0/3;} I was wondering what parts of (IEEE 754) would cause count 3: to be 1.0 instead of .9999999999999999. I realize that ...
1vote
198views

### Exact same WebGL code renders differently on different devices

I'm fairly new to shaders, just started playing with them in VisionsOfChaos for Android around a year ago. I implemented my 4D fractal viewer in it (as far as I know, a completely new approach, which ...
1vote
60views

### Python is automatically rounding large numbers with decimals

I need to perform the following calculation in Python: (10^16) + (9 * 54945054708534.5) The output it prints is: 5494505492376810.0 but this is incorrect as the solution should end in 0.5. I have ...
1vote
67views

### JavaScript: parseFloat() strips trailing zeros

I have been trying to find a solution to the following problem. I have a string that is a floating-point number like, var value = '12.30'; When I try to cast it to be a floating number parseFloat('...
1vote
58views

### floating point arithmetics: interpolating two half-precision floating point values with one full-precision interpolation

let's say I have two IEEE 32bit floating point values a and b plus one IEEE 32bit fp interpolation value x (something between 0 and 1) so I get the interpolated result: float a = 123.456f; float b = ...
1vote
79views

### Why do I get (slightly) different results everytime I run the same query on Google Bigquery?

I have a query on BQ that that produces slightly different results everytime I run it. Not big differences, e.g. 4421.10399999999 vs 4421.104 which looks like it is simply inconsistent rounding, but I ...
1vote
215views

### Pandas Float Precision - Apparently Identical Numbers Showing as Not Equal

Hopefully a very simple solution to this. I have tried the solutions for two similar questions on SO, but these haven't worked for me. Essentially I have a process which evaluates whether numbers in ...
1vote
46views

### Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
1vote
68views

### Determine precision of number, including trailing zeros

I suspect that this isn't possible, but giving it a try: Is it possible to determine the number of decimal points of a number, whether or not those decimal points are trailing zeros? getPrecision(3) ...
1vote
1kviews

### How to create a linspace like pytorch tensor which ensures that all values are between start and stop?

I want to create a 1d tensor of values similar to torch.linspace or torch.arange in pytorch, which ensures that all values of the created tensor are between start and stop. Example values which break ...
1vote
84views

### Why there's no precision error when printing 0.1 as a double with many digits after point?

The function (written in Kotlin) below fun main() { println ("%.20f".format(0.1f)) print ("%.100f".format(0.1)) } prints 0.10000000149011612000 0....
1vote
140views

### Float equality comparison using epsilon and hash code in Swift?

I want to create Set of floats, compared using epsilon. However if some values can produce different hash codes, but be equal is you compare them this way: func isEqual(to other: Float, usingEpsilon ...
1vote
141views

### How to find accuracy of matrix multiplication with floating-point numbers?

I am trying to analyze how floating-point computation becomes more inaccurate when the data size decreases. In order to do that, I wanted to perform simple matrix operations on different variations of ...
1vote
411views

### Node Buffer.readFloatBE() extends data to 64-bit

When converting data to/from a Buffer using 32-bit floating point numbers, it seems that nodejs extends the float to double using some criteria that makes the transformation not reversible: > f=3....
1vote
123views

### How to print glm vector with manually specified precion?

I know about glm::to_string but I cannot find away to set precision of the string conversion. A few issues come up because of this. If I want to serialize glm structures, I won't be able to store ...
1vote
162views

### How to preserve the accuracy of a floating point literal in Rust when using generics?

What's a reasonable strategy for preserving the accuracy of a floating point literal in Rust when using generics? As an example, consider the following code: // External functions use std::fmt::...
1vote
248views

### Conversion Precision Error when converting IEE Half Precision Floating Point to Decimal

I have some precision error during the conversion from 16 bit half precision floating point format to decimal. It is able to accurately convert certain numbers while at the same time not accurate for ...
1vote
59views

### How can we work on fixed amount of precision that is less than double but more than float?

I'm just trying to do something. There is a some computation to be done using double-type variables. We have two parameters "x" and "y" where we need to store log2(x) with y-bit precision. First ...
1vote
187views

### Key-space analysis of an algorithm

I have an algorithm that requires a set of keys to encrypt a text (image of size N x N). It is an image encryption algorithm for a project. The key can be thought of as a tuple x_0, lambda, d, i, j, ...
1vote
885views

### truncate float values to 8 decimal places with pandas.to_excel

I have a data frame with columns containing float values. One of the cells contain a value of 0.021893287 I have set the pandas option to: pd.set_option('precision', 8) When I try to export the ...
1vote