# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

1,276 questions
Filter by
Sorted by
Tagged with
24views

### go/golang: how to define user-specific precision for channel?

I want to use a communication channel in go/golang to transmit a value of the type big.Float. However, for an iteration process I need the precision of that value to be user-defined like in var a big....
26views

### Why console gives different results [duplicate]

When I write this in the console I get these answers which are unlogic a = 0.2 b = 0.3 c = 0.4 x = a+b+c gives me 0.9 y = b+c+a gives me 0.8999999999999999 x = c+a+b gives me 0.9000000000000001
19views

### Why does python truncate this float 1646313339.123456789 to 1646313339.1234567? [duplicate]

Why is the float 1646313339.123456789 truncated in Python? print(1646313339.123456789) prints 1646313339.1234567 And print('{0:.9f}'.format(1646313339.123456789)) prints 1646313339.123456717
78views

### How to avoid roundoff errors in numpy.random.choice?

Say x_1, x_2, ..., x_n are n objects and one wants to pick one of them so that the probability of choosing x_i is proportional to some number u_i. x, u = np.array([x_1, x_2, ..., x_n]), np.array([u_1, ...
• 125
33views

### Addressing polynomial multiplication and division "overflow" issue

I have a list of the coefficient to degree 1 polynomials, with a[i][0]*x^1 + a[i][1] a = np.array([[ 1. , 77.48514702], [ 1. , 0. ], [ 1. , 2....
• 251
64views

### What is the difference between accuracy_score and clf.score in sklearn?

from sklearn.svm import SVC from sklearn.datasets import make_blobs from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split X, y = make_blobs(n_samples=500,...
• 97
74views

### Lua float errors

I'm fairly new to lua, but I read that it does not have integers, so all numbers are floats. I noticed in my log that some numbers get a slight inaccuracy added to it. For instance 0.14 is written as ...
• 97
81views

### wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
• 441
33views

### How does numpy manage to divide float32 by 2**63?

Here Daniel mentions ... you pick any integer in [0, 2²⁴), and divide it by 2²⁴, then you can recover your original integer by multiplying the result again by 2²⁴. This works with 2²⁴ but not with 2²⁵...
52views

### How can I change my C program to give correct output for differentiation calculation?

I am trying to calculate the absolute error in the calculation of f'(1) (where f(x)=sin(x)) caused, due to direct calculation i.e.(cos(1)), and that done with the formula f'(x) = (f(x+h)-f(x))/h i....
48views

### Lua floating point calculations not precise [duplicate]

I was working on my project which visualises sine and cosine but sometimes stuff like this happens : Problem I checked the code bunch of times but it has only simple math operations just to change the ...
60views

### golang How to get the correct result? floating point arithmetic issues

testA, testB, testC := big.NewFloat(0), big.NewFloat(0), big.NewFloat(0) testA.SetPrec(500) testB.SetPrec(500) testA.SetString("0.081531021188798896") testB.SetString("0.9975") ...
• 1,152
33views

### Reduce or solve calculative errors in Javascript [duplicate]

Sometimes simple algebra can result in floating accuracy errors. in this case, I encountered 12 * 1.6 which resulted in 19.20000000003 Is there a way to catch this issue or prevent it from happening? ...
• 3,678
47views

### Calculate the order of precision limit of floating type representation [duplicate]

Let's say that I have a Single-precision floating-point format variable in my machine and I want to assign on it the result of a given operation. From Wikipedia: The IEEE 754 standard specifies a ...
36views

### Excel Bug: Why some cells in a time interval cells cannot be compared with accuracy?

When one drag a fill handle to fill times in a interval, some cells with times cannot be compared in trustable way. The bug reproduction is simple a) Format Column A as Hour type (like hh:mm:ss) b) ...
• 2,113
38views

### How to not lose accuracy when saving a DataFrame to csv

I have the following DataFrame: Direction Date Market Quantity Price Commission Charges 0 S 29/12/2020 IBTM 38.46538070 12.242512 0.0 0 1 ...
• 93
1vote
76views

### Unexpected integer division vs. floating-point division result in Python

Running the following code in Python produces a somewhat unexpected result. print(10000 // 0.1) # prints 99999.0 print(10000 / 0.1) # prints 100000.0 Now, I might have understood the discrepancy if ...
54views

### How to compare float and decimal in python?

I have a (decimal.Decimal) column in a dataframe as below: dt = pd.DataFrame({"OPEN": [-0.00010,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005, 0.0005,-0.0037, -...
• 6,767
1vote
33views

### what is the easiest way to subtract 2 columns of float in dataframe?

Let's say I have a dataframe with 2 columns of floats with 4 or 5 digits of pricision as below: dt = pd.DataFrame({"OPEN": [-0.00011,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005, ...
• 6,767
89views

### Why doesn't the same generated assembler code lead to the same output?

Sample code (t0.c): #include <stdio.h> float f(float a, float b, float c) __attribute__((noinline)); float f(float a, float b, float c) { return a * c + b * c; } int main(void) { void* ...
• 4,132
76views

### how many numbers between 0-1 can a 32bit floating point represent?

I'm trying to figure out how many numbers between 0-1 can a 32bit floating point represent? is it just the same as in an interval? (2**24)-1
56views

### as.character() with numeric loses precision when a string is transformed into numeric

In R I want to run following code: str <- "2811.1019192874087" val <- as.numeric(str) str2 <- as.character(val) print(str2) print(val) It prints: "2811.10191928741" 2811....
• 1,526
32views

### Expected value wrong with output with using long double [duplicate]

I am working on my knowledge in C++ and I am expecting a value which I did not receive. Here is my code #include <iostream> #include <float.h> using std:: cout; using std:: endl; using ...
• 15
113views

### Why does a subtype of AbstractArray result in imprecise matrix operations in Julia?

I'm currently working on creating a subtype of AbstractArray in Julia, which allows you to store a vector in addition to an Array itself. You can think of it as the column "names", with ...
• 53
60views

### i need more decimal places for pi calculation [closed]

I'm trying to make a Pi calculator in python but I need more decimal places. it would help a lot if someone edited my code and carefully explained what they did. this is the code I'm using. import ...
66views

### Float precision when dealing with logarithms in musical pitch calculations

I'm writing a simple program to determine the difference between two musical pitches in cents; one cent is equal to 1/100th of a semitone. Dealing in cents is preferable for comparing musical pitches ...
101views

### comparing 2 floating numbers in matlab

I was tortured by the floating point comparison issue in Matlab in the previous a couple of weeks. My code involves tons of a<b or c<=d. What drives me crazy is that dw = 0.001; W1 = [0:dw:1]; ...
• 145
35views

### R: difference between apply(object, 1, function(x) sum(x-a)/b) and rowsums((object-a)/b)

I'm new to R and am struggling with the apply function. It is really slow to execute and I was trying to optimize some code I received. I am trying to do some matrix operations (element-wise ...
91views

### Floating point in google colab for verification

I'm implementing an algorithm in Python (GoogleColab) and it outputs two lists; now, I would like to verify if this lists are equal, actualy I know that they should be equal. The problem is that ...
• 129
143views

### Is there any way to not lose the precision and still get the value?

First off, I'm a total beginner at C, with prior experience of programming in Java and Python. The goal of the program was to add 2 numbers. While I was playing with the code, I encountered an issue ...
32views

### Python replace integers in a Counter with their permillages of the total [duplicate]

I am trying to generate readable word-like random strings not found in any dictionaries using Markov Chain. I have pulled a large amount of data of ngram frequencies from a total of 105230 words ...
• 371
77views

### Calculating the angle between two vectors using a needle-like triangle

I implemented a function (angle_between) to calculate the angle between two vectors. It makes use of needle-like triangles and is based on Miscalculating Area and Angles of a Needle-like Triangle and ...
• 1,652
1vote
74views

### Convert double number to digits and vice versa?

I'm trying to convert a double number into array of digits Input: double num Output: int[] arrDigit int dotIdx bool isMinus for example: Input: double num = -69.69777 Output: ...
• 2,005
133views

### Program to count number of digits in a double following decimal point in Java

Below is a program that is supposed to count the number of digits in a double value following the decimal point. On entering some double values, the program tends to start an infinite loop (possibly ...
77views

### Is the remainder operator susceptible to floating point errors?

I want to create a setter for a double variable num, but I would only like to update it if the input is a multiple of 0.5. Here's what I have, but I'm worried about floating-point errors. public void ...
• 163
149views

### pandas mess up multi level index parquet float accuracy

I have some data that has multi level index & column. My goal is to store the data as float32 instead of double/float64 into parquet file, to save disk space. I don't care about the accuracy of ...
• 511
533views

### Euler-Mascheroni Constant

In programming, I used only Integers. But this time for some calculations. I need to calculate Euler-Mascheroni Constant γ . up to n-th decimal.{Though n ∈ [30, 150] is enough for me. [x] = gif(x) = ...
56views

### zsh loses precision when miltiplying two floats of significantly different magnitude

I'm trying to do some floating-point math in a zsh script. I'm seeing the following behavior: \$ (( a = 1.23456789 * 0.00000001 )); printf "a = %g\n" \$a a = 1.23e-08 \$ (( a = 1.23456789 * 0....
• 23
1vote
42views

### "Approximately equal to" operand in Python [duplicate]

Do you of you know what the "approximately equal to" operand is? I need to find numbers in a list that are approximately equal to a fixed number that I have defined. I tried doing =~ and == ~...
87views

### Python Decimal Library is Imprecise? [duplicate]

I'm reading up on the Python Decimal module. I have a need to make a large number of precise calculations, often with lots of decimal places, where being off by a small amount adds up over time. Enter ...
131views

### Precision limits of numpy floats?

Given the sequence 1/1, 1/2, 1/3, ... , 1/n How can I calculate at what point I will not be able to distinguish with precision E between two consecutive elements 1/i and 1/i+1 if I use numpy.float16 ?...
• 6,076
326views

### How many different sums can we get from very few floats?

Someone just asked why sum(myfloats) differed from sum(reversed(myfloats)). Quickly got duped to Is floating point math broken? and deleted. But it made me curious: How many different sums can we get ...
• 6,049
38views

### Convert a continuous range of doubles to a discrete int range

i want to map a continuous range of double values [10.0,20.0] to a byte in the range of [100,200]; The following must apply (even when converting back and forth multiple times): convertToByte(...
• 95
118views

### C - Float Imprecision, How to lock the decimal places for calculations?

I was making a little program to test floats in the C: The program itself is very simple, I just want to based on the User's input, return how much Dollar(s), Quarter(s)... etc, his number has. //-----...
112views

### Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
• 571
68views

### What are the details of the Juniper floating-point scenario/bug wherein the sticky bit is lost?

On some Juniper MX routers, floats are not handled correctly: The sticky bit is lost if it is shifted more that 8 bits to the right (underflow) during a calculation. Is there a workaround for this? ...
• 571
1vote
62views

### When have enough bits of my series with non-negative terms been calculated?

I have a power series with all terms non-negative which I want to evaluate to some arbitrarily set precision p (the length in binary digits of a MPFR floating-point mantissa). The result should be ...
• 131
52views

### Simple addition in Perl causing result to be off by small fraction [duplicate]

#!/usr/bin/perl \$a = 1; while ( \$a <= 50 ) { print "\$a\n"; \$a += 0.025; } Very simple code. But here is how the output looks like... 2.225 2.25 2.275 2.3 2.325 2.35 2.375 ...
1vote
160views

### Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC: #include <iostream> namespace { template<typename Out> constexpr auto caster{[](auto x) constexpr ...
• 131