All Questions

194 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
11votes
0answers
610views

problem with vulkan floating point behavior

I try to implement the paper Extended-Precision Floating-Point Numbers for GPU Computation by Andrew Thall, Alma College in a GLSL Vulkan compute shader. I need this because some of my devices don't ...
user avatar
  • 111
3votes
1answer
62views

Dealing with floating point point inaccuracy in very small numbers efficiently

The program I am working with takes OpenStreetMap data to render a map. The data consists of 4 coordinates, that make up the bounds of the data. I am drawing lines, that sometimes exceed these bounds ...
user avatar
3votes
0answers
85views

R not working properly with big numbers because of default options?

Seems like R coerces big numbers and cannot compare them effectively: x = 123412415124231251233213 x == 123412415124231251233214 [1] TRUE x == 123412415124231251233217 [1] TRUE Any idea why (maybe a ...
user avatar
3votes
1answer
339views

First (smallest) even number not representable with IEEE 754 floating point?

I'm not sure how to go about this problem. I know that the smallest integer not representable by IEEE 754 would be 2^(mantissa+1) + 1 but how would I take that info and change it to an even number? ...
user avatar
  • 2,429
3votes
1answer
232views

What range of numbers can I represent using scaled_float in Elasticsearch?

I'm trying to figure out what numbers I can represent using scaled_float. In the documentation here https://www.elastic.co/guide/en/elasticsearch/reference/master/number.html I first read: ...
user avatar
2votes
1answer
105views

What is a significance of 9.536743e-7?

I realized that outputs of a continuous function (takes a vector, returns a scalar) that I wrote in python are discretized at the resolution of 9.536743e-7. I googled this number and learned that some ...
user avatar
  • 121
2votes
2answers
86views

python search float return wrong because of precision problem?

I want to find a float in a arry like this: arr = np.asarray([1351.1 , 1351.11, 1351.14, 1351.16, 1351.17]) index = np.searchsorted(arr, 1351.14, side="right") - 1 # return 2 But I find ...
user avatar
  • 7,635
2votes
0answers
99views

Is precision of pow in C dependent on scale of parameters?

Legacy code I am to maintain has a function with a variable x of type double. This variable contains a position measured in meters. It is expected to hold a value in the mm range, let's say 0.0001 <...
user avatar
2votes
1answer
1kviews

Understanding sys.float_info for maximum floats in python

I'm trying to understand the information given by sys.float_info to understand what the maximum floats in Python are. On my computer, this gives me the following: >>> import sys >>> ...
user avatar
2votes
0answers
1kviews

How to get the maximum number of decimal places in each column of a pandas dataframe?

I need to set the global float precision to the minimum value possible Also, I need to get the precision for each column, in part to get the global precision and on the other hand, I would like to ...
user avatar
  • 8,950
2votes
0answers
52views

How much precision is needed to represent n/2^k, given a boundary of -2 to +2?

Apologies for the poorly phrased title. I tried to do better, but perhaps someone can suggest a better title. As a test case for another project, I'm working with the Mandelbrot set, using only value ...
user avatar
  • 527
2votes
0answers
100views

Is there a way for Python to read __float128 from a binary file?

I am trying to read a binary file that stores some __float128 variables from other C project using Python. I've been looking in the numpy.float128 but it seems that it is implemented in a different ...
user avatar
2votes
0answers
95views

Is there options to set on Visual Studio to keep deterministic floating-point results between different multi-thread setup?

I am working with OpenMp with VS2017 in order to compute our statistics with multi-thread process. But, for few values, we found out that results can be differents with the number of threads set. For ...
user avatar
2votes
1answer
568views

Casting from (long)double to size_t

I'm trying to implement Sieve of Eratosthenes as efficient as possible. I'd like to set length of my prime array to upper bond of pi(n) < 1.25506n / ln n but I'm not sure how to proceed with ...
user avatar
  • 177
2votes
0answers
205views

Floating-point precision problems

Using Microsoft Excel 2010. Why do these two formulae produce different results? = (0.5 - 0.4 - 0.1) produces -2.775E-17 while = 0.5 - 0.4 - 0.1 produces exactly 0. The only difference is in the ...
user avatar
2votes
0answers
657views

What MySQL "DECIMAL" precision and scale is required to store a C# decimal value?

I want to store some decimal values in my MySQL database. The obvious MySQL database type for that would be DECIMAL I guess. What precision "a" and scale "b" do I have to set for DECIMAL(a,b ) ...
user avatar
  • 5,504
2votes
1answer
253views

How to evaluate an alternating series when addends contain rounding errors?

I want to numerically evaluate the transition probability of a linear Birth and Death process where is the binomial coefficient and I am able to evaluate it with an acceptable numerical error (...
user avatar
2votes
0answers
118views

Finding smallest float x such that x+d=y

How to find the smallest float/double number x which satisfies x + d = y given d and y? (iiuc this is theoretically solved via setting fesetround (FE_DOWNWARD) and just doing y - d but in clang/Xcode ...
user avatar
  • 22.2k
2votes
0answers
39views

precison loss in floating point multiply

The following mentions http://floating-point-gui.de/errors/propagation/ that floating point multiply and divide are safe. Empirically testing across various chains of multiplication also suggest that ...
user avatar
2votes
0answers
565views

How to print scientific notation with all significant digits moved after decimal point in python

I am trying to print scientific numbers with all the significant digits after the decimal point. ` Python's scientific notation prints with one digit before the dot like 1.2345678E+03, '%E' % ...
user avatar
  • 494
2votes
0answers
106views

How can I specify tolerance for floating point operation in CGAL library?

Is there a way to specify tolerance for CGAL geometry operations? I use exact_predicates_inexact_constructions kernel for computational speed. However sometimes operations such as determining a point ...
user avatar
2votes
0answers
59views

Solving the strict inequality with integer math and accounting for rounding errors

I have an array of floating point numbers of type T, where T could be either float or double T x[n]; These numbers are strictly positive and sorted, i.e. 0 < x[0] < x[1] < x[2] < ... &...
user avatar
  • 1,945
2votes
0answers
120views

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p ...
user avatar
2votes
1answer
99views

Floating Point Number issue JavaScript

I'm struggling to resolve a Floating Point Number issue where var change returns as 0.0999 recurring, and i need to return 0.01 (one penny). The code works fine, except the very last penny, because of ...
user avatar
2votes
0answers
422views

16-bit floating point on fpga

I try to use Altera's floating point IP to generate half precision instead of single (32-bit) blocks for addition , multiplication etc. However when configuring the IP it seems that half precision fp ...
user avatar
2votes
1answer
234views

Can computing tan(x)=sin(x)/cos(x) cause a loss of precision?

After a call to sincos(x,&s,&c) from the unix m math library it would be natural to get the tangent as s/c. Is this safe or there may be (ill) cases in which the (supposedly) more expensive ...
user avatar
  • 5,175
2votes
0answers
136views

Using Matlab to read binary data from file

I'm currently trying to read data from a .surf file using Matlab. (I realise there will probably be quite a lot of other questions similar to this, but each is specific to its own problem, and I wasn'...
user avatar
  • 358
2votes
1answer
146views

Why are these two calculations which are exactly same giving different results in Fortran using gfortran?

real, dimension(3), parameter :: boxlen = [4.0, 5.0, 7.0] real, parameter :: mindist = 0.1 integer ::i write(*,"(A)") "Operation on array" print*, floor(boxlen/mindist) write(*,"(/A)") "...
user avatar
2votes
1answer
668views

numpy/pandas: test float64 arrays are equal up to significant digits

I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this ...
user avatar
  • 6,788
2votes
1answer
94views

Floating point addition and division

For this code segment double count = 0.0; while( count != 1.0) {count += 1.0/3;} I was wondering what parts of (IEEE 754) would cause count 3: to be 1.0 instead of .9999999999999999. I realize that ...
user avatar
1vote
0answers
198views

Exact same WebGL code renders differently on different devices

I'm fairly new to shaders, just started playing with them in VisionsOfChaos for Android around a year ago. I implemented my 4D fractal viewer in it (as far as I know, a completely new approach, which ...
user avatar
1vote
0answers
60views

Python is automatically rounding large numbers with decimals

I need to perform the following calculation in Python: (10^16) + (9 * 54945054708534.5) The output it prints is: 5494505492376810.0 but this is incorrect as the solution should end in 0.5. I have ...
user avatar
1vote
2answers
67views

JavaScript: parseFloat() strips trailing zeros

I have been trying to find a solution to the following problem. I have a string that is a floating-point number like, var value = '12.30'; When I try to cast it to be a floating number parseFloat('...
user avatar
  • 1,824
1vote
0answers
58views

floating point arithmetics: interpolating two half-precision floating point values with one full-precision interpolation

let's say I have two IEEE 32bit floating point values a and b plus one IEEE 32bit fp interpolation value x (something between 0 and 1) so I get the interpolated result: float a = 123.456f; float b = ...
user avatar
1vote
0answers
79views

Why do I get (slightly) different results everytime I run the same query on Google Bigquery?

I have a query on BQ that that produces slightly different results everytime I run it. Not big differences, e.g. 4421.10399999999 vs 4421.104 which looks like it is simply inconsistent rounding, but I ...
user avatar
1vote
1answer
215views

Pandas Float Precision - Apparently Identical Numbers Showing as Not Equal

Hopefully a very simple solution to this. I have tried the solutions for two similar questions on SO, but these haven't worked for me. Essentially I have a process which evaluates whether numbers in ...
user avatar
1vote
0answers
46views

Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
user avatar
1vote
1answer
68views

Determine precision of number, including trailing zeros

I suspect that this isn't possible, but giving it a try: Is it possible to determine the number of decimal points of a number, whether or not those decimal points are trailing zeros? getPrecision(3) ...
user avatar
  • 1,649
1vote
1answer
1kviews

How to create a linspace like pytorch tensor which ensures that all values are between start and stop?

I want to create a 1d tensor of values similar to torch.linspace or torch.arange in pytorch, which ensures that all values of the created tensor are between start and stop. Example values which break ...
user avatar
1vote
1answer
84views

Why there's no precision error when printing 0.1 as a double with many digits after point?

The function (written in Kotlin) below fun main() { println ("%.20f".format(0.1f)) print ("%.100f".format(0.1)) } prints 0.10000000149011612000 0....
user avatar
1vote
0answers
140views

Float equality comparison using epsilon and hash code in Swift?

I want to create Set of floats, compared using epsilon. However if some values can produce different hash codes, but be equal is you compare them this way: func isEqual(to other: Float, usingEpsilon ...
user avatar
  • 3,500
1vote
2answers
141views

How to find accuracy of matrix multiplication with floating-point numbers?

I am trying to analyze how floating-point computation becomes more inaccurate when the data size decreases. In order to do that, I wanted to perform simple matrix operations on different variations of ...
user avatar
1vote
1answer
411views

Node Buffer.readFloatBE() extends data to 64-bit

When converting data to/from a Buffer using 32-bit floating point numbers, it seems that nodejs extends the float to double using some criteria that makes the transformation not reversible: > f=3....
user avatar
  • 673
1vote
0answers
123views

How to print glm vector with manually specified precion?

I know about glm::to_string but I cannot find away to set precision of the string conversion. A few issues come up because of this. If I want to serialize glm structures, I won't be able to store ...
user avatar
  • 3,881
1vote
0answers
162views

How to preserve the accuracy of a floating point literal in Rust when using generics?

What's a reasonable strategy for preserving the accuracy of a floating point literal in Rust when using generics? As an example, consider the following code: // External functions use std::fmt::...
user avatar
  • 5,670
1vote
0answers
248views

Conversion Precision Error when converting IEE Half Precision Floating Point to Decimal

I have some precision error during the conversion from 16 bit half precision floating point format to decimal. It is able to accurately convert certain numbers while at the same time not accurate for ...
user avatar
  • 31
1vote
0answers
59views

How can we work on fixed amount of precision that is less than double but more than float?

I'm just trying to do something. There is a some computation to be done using double-type variables. We have two parameters "x" and "y" where we need to store log2(x) with y-bit precision. First ...
user avatar
1vote
0answers
187views

Key-space analysis of an algorithm

I have an algorithm that requires a set of keys to encrypt a text (image of size N x N). It is an image encryption algorithm for a project. The key can be thought of as a tuple x_0, lambda, d, i, j, ...
user avatar
1vote
1answer
885views

truncate float values to 8 decimal places with pandas.to_excel

I have a data frame with columns containing float values. One of the cells contain a value of 0.021893287 I have set the pandas option to: pd.set_option('precision', 8) When I try to export the ...
user avatar
1vote
0answers
64views

Visual Studio C compiler custom floating point precision

I have a program running on an archaic Infineon C166 processor, compiled with Keil C166. The Keil floating point arithmetic library has an intricacy where it reduces the single-precision float ...
user avatar

15 30 50 per page