# Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

1,276
questions

-1votes

0answers

24views

### go/golang: how to define user-specific precision for channel?

I want to use a communication channel in go/golang to transmit a value of the type big.Float. However, for an iteration process I need the precision of that value to be user-defined like in
var a big....

0votes

0answers

26views

### Why console gives different results [duplicate]

When I write this in the console I get these answers which are unlogic
a = 0.2
b = 0.3
c = 0.4
x = a+b+c gives me 0.9
y = b+c+a gives me 0.8999999999999999
x = c+a+b gives me 0.9000000000000001

-2votes

0answers

19views

### Why does python truncate this float 1646313339.123456789 to 1646313339.1234567? [duplicate]

Why is the float 1646313339.123456789 truncated in Python?
print(1646313339.123456789)
prints
1646313339.1234567
And
print('{0:.9f}'.format(1646313339.123456789))
prints
1646313339.123456717

0votes

2answers

78views

### How to avoid roundoff errors in numpy.random.choice?

Say x_1, x_2, ..., x_n are n objects and one wants to pick one of them so that the probability of choosing x_i is proportional to some number u_i.
x, u = np.array([x_1, x_2, ..., x_n]), np.array([u_1, ...

0votes

0answers

33views

### Addressing polynomial multiplication and division "overflow" issue

I have a list of the coefficient to degree 1 polynomials, with a[i][0]*x^1 + a[i][1]
a = np.array([[ 1. , 77.48514702],
[ 1. , 0. ],
[ 1. , 2....

0votes

1answer

64views

### What is the difference between accuracy_score and clf.score in sklearn?

from sklearn.svm import SVC
from sklearn.datasets import make_blobs
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
X, y = make_blobs(n_samples=500,...

0votes

1answer

74views

### Lua float errors

I'm fairly new to lua, but I read that it does not have integers, so all numbers are floats.
I noticed in my log that some numbers get a slight inaccuracy added to it. For instance 0.14 is written as ...

3votes

0answers

81views

### wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one.
I'm getting 1 instead of 1+epsilon when I perform the addition and print the result.
I've implemented a getEpsilon function. I ...

0votes

1answer

33views

### How does numpy manage to divide float32 by 2**63?

Here Daniel mentions
... you pick any integer in [0, 2²⁴), and divide it by 2²⁴, then you can recover your original integer by multiplying the result again by 2²⁴. This works with 2²⁴ but not with 2²⁵...

0votes

1answer

52views

### How can I change my C program to give correct output for differentiation calculation?

I am trying to calculate the absolute error in the calculation of f'(1) (where f(x)=sin(x)) caused, due to direct calculation i.e.(cos(1)), and that done with the formula
f'(x) = (f(x+h)-f(x))/h
i....

-1votes

1answer

48views

### Lua floating point calculations not precise [duplicate]

I was working on my project which visualises sine and cosine but sometimes stuff like this happens : Problem
I checked the code bunch of times but it has only simple math operations just to change the ...

0votes

1answer

60views

### golang How to get the correct result? floating point arithmetic issues

testA, testB, testC := big.NewFloat(0), big.NewFloat(0), big.NewFloat(0)
testA.SetPrec(500)
testB.SetPrec(500)
testA.SetString("0.081531021188798896")
testB.SetString("0.9975")
...

-1votes

1answer

33views

### Reduce or solve calculative errors in Javascript [duplicate]

Sometimes simple algebra can result in floating accuracy errors.
in this case, I encountered 12 * 1.6 which resulted in 19.20000000003
Is there a way to catch this issue or prevent it from happening?
...

0votes

1answer

47views

### Calculate the order of precision limit of floating type representation [duplicate]

Let's say that I have a Single-precision floating-point format variable in my machine and I want to assign on it the result of a given operation.
From Wikipedia:
The IEEE 754 standard specifies a ...

0votes

0answers

36views

### Excel Bug: Why some cells in a time interval cells cannot be compared with accuracy?

When one drag a fill handle to fill times in a interval, some
cells with times cannot be compared in trustable way.
The bug reproduction is simple
a) Format Column A as Hour type (like hh:mm:ss)
b) ...

0votes

0answers

38views

### How to not lose accuracy when saving a DataFrame to csv

I have the following DataFrame:
Direction Date Market Quantity Price Commission Charges
0 S 29/12/2020 IBTM 38.46538070 12.242512 0.0 0
1 ...

1vote

2answers

76views

### Unexpected integer division vs. floating-point division result in Python

Running the following code in Python produces a somewhat unexpected result.
print(10000 // 0.1) # prints 99999.0
print(10000 / 0.1) # prints 100000.0
Now, I might have understood the discrepancy if ...

0votes

1answer

54views

### How to compare float and decimal in python?

I have a (decimal.Decimal) column in a dataframe as below:
dt = pd.DataFrame({"OPEN": [-0.00010,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005,
0.0005,-0.0037, -...

1vote

1answer

33views

### what is the easiest way to subtract 2 columns of float in dataframe?

Let's say I have a dataframe with 2 columns of floats with 4 or 5 digits of pricision as below:
dt = pd.DataFrame({"OPEN": [-0.00011,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005,
...

5votes

1answer

89views

### Why doesn't the same generated assembler code lead to the same output?

Sample code (t0.c):
#include <stdio.h>
float f(float a, float b, float c) __attribute__((noinline));
float f(float a, float b, float c)
{
return a * c + b * c;
}
int main(void)
{
void* ...

-1votes

1answer

76views

### how many numbers between 0-1 can a 32bit floating point represent?

I'm trying to figure out how many numbers between 0-1 can a 32bit floating point represent?
is it just the same as in an interval? (2**24)-1

0votes

0answers

56views

### as.character() with numeric loses precision when a string is transformed into numeric

In R I want to run following code:
str <- "2811.1019192874087"
val <- as.numeric(str)
str2 <- as.character(val)
print(str2)
print(val)
It prints:
"2811.10191928741"
2811....

0votes

0answers

32views

### Expected value wrong with output with using long double [duplicate]

I am working on my knowledge in C++ and I am expecting a value which I did not receive. Here is my code
#include <iostream>
#include <float.h>
using std:: cout;
using std:: endl;
using ...

5votes

1answer

113views

### Why does a subtype of AbstractArray result in imprecise matrix operations in Julia?

I'm currently working on creating a subtype of AbstractArray in Julia, which allows you to store a vector in addition to an Array itself. You can think of it as the column "names", with ...

-3votes

2answers

60views

### i need more decimal places for pi calculation [closed]

I'm trying to make a Pi calculator in python but I need more decimal places.
it would help a lot if someone edited my code and carefully explained what they did.
this is the code I'm using.
import ...

0votes

3answers

66views

### Float precision when dealing with logarithms in musical pitch calculations

I'm writing a simple program to determine the difference between two musical pitches in cents; one cent is equal to 1/100th of a semitone. Dealing in cents is preferable for comparing musical pitches ...

0votes

2answers

101views

### comparing 2 floating numbers in matlab

I was tortured by the floating point comparison issue in Matlab in the previous a couple of weeks.
My code involves tons of a<b or c<=d.
What drives me crazy is that
dw = 0.001;
W1 = [0:dw:1];
...

0votes

0answers

35views

### R: difference between apply(object, 1, function(x) sum(x-a)/b) and rowsums((object-a)/b)

I'm new to R and am struggling with the apply function. It is really slow to execute and I was trying to optimize some code I received.
I am trying to do some matrix operations (element-wise ...

0votes

0answers

91views

### Floating point in google colab for verification

I'm implementing an algorithm in Python (GoogleColab) and it outputs two lists; now, I would like to verify if this lists are equal, actualy I know that they should be equal.
The problem is that ...

0votes

2answers

143views

### Is there any way to not lose the precision and still get the value?

First off, I'm a total beginner at C, with prior experience of programming in Java and Python. The goal of the program was to add 2 numbers. While I was playing with the code, I encountered an issue ...

0votes

1answer

32views

### Python replace integers in a Counter with their permillages of the total [duplicate]

I am trying to generate readable word-like random strings not found in any dictionaries using Markov Chain.
I have pulled a large amount of data of ngram frequencies from a total of 105230 words ...

3votes

1answer

77views

### Calculating the angle between two vectors using a needle-like triangle

I implemented a function (angle_between) to calculate the angle between two vectors. It makes use of needle-like triangles and is based on Miscalculating Area and Angles of a Needle-like Triangle and ...

1vote

1answer

74views

### Convert double number to digits and vice versa?

I'm trying to convert a double number into array of digits
Input:
double num
Output:
int[] arrDigit
int dotIdx
bool isMinus
for example:
Input:
double num = -69.69777
Output:
...

0votes

1answer

133views

### Program to count number of digits in a double following decimal point in Java

Below is a program that is supposed to count the number of digits in a double value following the decimal point. On entering some double values, the program tends to start an infinite loop (possibly ...

4votes

2answers

77views

### Is the remainder operator susceptible to floating point errors?

I want to create a setter for a double variable num, but I would only like to update it if the input is a multiple of 0.5.
Here's what I have, but I'm worried about floating-point errors.
public void ...

-1votes

1answer

149views

### pandas mess up multi level index parquet float accuracy

I have some data that has multi level index & column. My goal is to store the data as float32 instead of double/float64 into parquet file, to save disk space. I don't care about the accuracy of ...

3votes

2answers

533views

### Euler-Mascheroni Constant

In programming, I used only Integers. But this time for some calculations. I need to calculate Euler-Mascheroni Constant γ . up to n-th decimal.{Though n ∈ [30, 150] is enough for me.
[x] = gif(x) = ...

2votes

2answers

56views

### zsh loses precision when miltiplying two floats of significantly different magnitude

I'm trying to do some floating-point math in a zsh script. I'm seeing the following behavior:
$ (( a = 1.23456789 * 0.00000001 )); printf "a = %g\n" $a
a = 1.23e-08
$ (( a = 1.23456789 * 0....

1vote

0answers

42views

### "Approximately equal to" operand in Python [duplicate]

Do you of you know what the "approximately equal to" operand is? I need to find numbers in a list that are approximately equal to a fixed number that I have defined. I tried doing =~ and == ~...

-2votes

1answer

87views

### Python Decimal Library is Imprecise? [duplicate]

I'm reading up on the Python Decimal module. I have a need to make a large number of precise calculations, often with lots of decimal places, where being off by a small amount adds up over time. Enter ...

2votes

2answers

131views

### Precision limits of numpy floats?

Given the sequence
1/1, 1/2, 1/3, ... , 1/n
How can I calculate at what point I will not be able to distinguish with precision E between two consecutive elements 1/i and 1/i+1 if I use numpy.float16 ?...

8votes

2answers

326views

### How many different sums can we get from very few floats?

Someone just asked why sum(myfloats) differed from sum(reversed(myfloats)). Quickly got duped to Is floating point math broken? and deleted.
But it made me curious: How many different sums can we get ...

-1votes

1answer

38views

### Convert a continuous range of doubles to a discrete int range

i want to map a continuous range of double values [10.0,20.0] to a byte in the range of [100,200];
The following must apply (even when converting back and forth multiple times):
convertToByte(...

0votes

1answer

118views

### C - Float Imprecision, How to lock the decimal places for calculations?

I was making a little program to test floats in the C: The program itself is very simple, I just want to based on the User's input, return how much Dollar(s), Quarter(s)... etc, his number has.
//-----...

4votes

0answers

112views

### Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...

0votes

1answer

68views

### What are the details of the Juniper floating-point scenario/bug wherein the sticky bit is lost?

On some Juniper MX routers, floats are not handled correctly: The sticky bit is lost if it is shifted more that 8 bits to the right (underflow) during a calculation. Is there a workaround for this? ...

1vote

0answers

62views

### When have enough bits of my series with non-negative terms been calculated?

I have a power series with all terms non-negative which I want to evaluate to some arbitrarily set precision p (the length in binary digits of a MPFR floating-point mantissa). The result should be ...

0votes

0answers

52views

### Simple addition in Perl causing result to be off by small fraction [duplicate]

#!/usr/bin/perl
$a = 1;
while ( $a <= 50 ) {
print "$a\n";
$a += 0.025;
}
Very simple code.
But here is how the output looks like...
2.225
2.25
2.275
2.3
2.325
2.35
2.375
...

1vote

0answers

160views

### Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC:
#include <iostream>
namespace {
template<typename Out>
constexpr auto caster{[](auto x) constexpr ...

0votes

1answer

66views

### Problem of difference in between matrix multiplication done by * and by components by components by hand (MATLAB)

I have a problem with the accuracy of matrix multiplication on Matlab. I will explain this with an example.
Let
A=[0.1256, 0.25789; 0.00018, 0.68741];
B=[0.1578, 0.24471; 0.12801, 0.99701];
s=0;
...