Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Filter by
Sorted by
Tagged with
-1votes
0answers
24views

go/golang: how to define user-specific precision for channel?

I want to use a communication channel in go/golang to transmit a value of the type big.Float. However, for an iteration process I need the precision of that value to be user-defined like in var a big....
user avatar
  • 1
0votes
0answers
26views

Why console gives different results [duplicate]

When I write this in the console I get these answers which are unlogic a = 0.2 b = 0.3 c = 0.4 x = a+b+c gives me 0.9 y = b+c+a gives me 0.8999999999999999 x = c+a+b gives me 0.9000000000000001
user avatar
-2votes
0answers
19views

Why does python truncate this float 1646313339.123456789 to 1646313339.1234567? [duplicate]

Why is the float 1646313339.123456789 truncated in Python? print(1646313339.123456789) prints 1646313339.1234567 And print('{0:.9f}'.format(1646313339.123456789)) prints 1646313339.123456717
user avatar
0votes
2answers
78views

How to avoid roundoff errors in numpy.random.choice?

Say x_1, x_2, ..., x_n are n objects and one wants to pick one of them so that the probability of choosing x_i is proportional to some number u_i. x, u = np.array([x_1, x_2, ..., x_n]), np.array([u_1, ...
user avatar
0votes
0answers
33views

Addressing polynomial multiplication and division "overflow" issue

I have a list of the coefficient to degree 1 polynomials, with a[i][0]*x^1 + a[i][1] a = np.array([[ 1. , 77.48514702], [ 1. , 0. ], [ 1. , 2....
user avatar
  • 251
0votes
1answer
64views

What is the difference between accuracy_score and clf.score in sklearn?

from sklearn.svm import SVC from sklearn.datasets import make_blobs from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split X, y = make_blobs(n_samples=500,...
user avatar
0votes
1answer
74views

Lua float errors

I'm fairly new to lua, but I read that it does not have integers, so all numbers are floats. I noticed in my log that some numbers get a slight inaccuracy added to it. For instance 0.14 is written as ...
user avatar
  • 97
3votes
0answers
81views

wrong result on adition of numbers larger than epsilon using numpy.float128

Considering that epsilon is the smallest number that you can add to one. I'm getting 1 instead of 1+epsilon when I perform the addition and print the result. I've implemented a getEpsilon function. I ...
user avatar
  • 441
0votes
1answer
33views

How does numpy manage to divide float32 by 2**63?

Here Daniel mentions ... you pick any integer in [0, 2²⁴), and divide it by 2²⁴, then you can recover your original integer by multiplying the result again by 2²⁴. This works with 2²⁴ but not with 2²⁵...
user avatar
0votes
1answer
52views

How can I change my C program to give correct output for differentiation calculation?

I am trying to calculate the absolute error in the calculation of f'(1) (where f(x)=sin(x)) caused, due to direct calculation i.e.(cos(1)), and that done with the formula f'(x) = (f(x+h)-f(x))/h i....
user avatar
-1votes
1answer
48views

Lua floating point calculations not precise [duplicate]

I was working on my project which visualises sine and cosine but sometimes stuff like this happens : Problem I checked the code bunch of times but it has only simple math operations just to change the ...
user avatar
0votes
1answer
60views

golang How to get the correct result? floating point arithmetic issues

testA, testB, testC := big.NewFloat(0), big.NewFloat(0), big.NewFloat(0) testA.SetPrec(500) testB.SetPrec(500) testA.SetString("0.081531021188798896") testB.SetString("0.9975") ...
user avatar
  • 1,152
-1votes
1answer
33views

Reduce or solve calculative errors in Javascript [duplicate]

Sometimes simple algebra can result in floating accuracy errors. in this case, I encountered 12 * 1.6 which resulted in 19.20000000003 Is there a way to catch this issue or prevent it from happening? ...
user avatar
  • 3,678
0votes
1answer
47views

Calculate the order of precision limit of floating type representation [duplicate]

Let's say that I have a Single-precision floating-point format variable in my machine and I want to assign on it the result of a given operation. From Wikipedia: The IEEE 754 standard specifies a ...
user avatar
0votes
0answers
36views

Excel Bug: Why some cells in a time interval cells cannot be compared with accuracy?

When one drag a fill handle to fill times in a interval, some cells with times cannot be compared in trustable way. The bug reproduction is simple a) Format Column A as Hour type (like hh:mm:ss) b) ...
user avatar
0votes
0answers
38views

How to not lose accuracy when saving a DataFrame to csv

I have the following DataFrame: Direction Date Market Quantity Price Commission Charges 0 S 29/12/2020 IBTM 38.46538070 12.242512 0.0 0 1 ...
user avatar
  • 93
1vote
2answers
76views

Unexpected integer division vs. floating-point division result in Python

Running the following code in Python produces a somewhat unexpected result. print(10000 // 0.1) # prints 99999.0 print(10000 / 0.1) # prints 100000.0 Now, I might have understood the discrepancy if ...
user avatar
0votes
1answer
54views

How to compare float and decimal in python?

I have a (decimal.Decimal) column in a dataframe as below: dt = pd.DataFrame({"OPEN": [-0.00010,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005, 0.0005,-0.0037, -...
user avatar
  • 6,767
1vote
1answer
33views

what is the easiest way to subtract 2 columns of float in dataframe?

Let's say I have a dataframe with 2 columns of floats with 4 or 5 digits of pricision as below: dt = pd.DataFrame({"OPEN": [-0.00011,-0.0114, 0.0066,-0.0044,-0.0012,-0.0005, ...
user avatar
  • 6,767
5votes
1answer
89views

Why doesn't the same generated assembler code lead to the same output?

Sample code (t0.c): #include <stdio.h> float f(float a, float b, float c) __attribute__((noinline)); float f(float a, float b, float c) { return a * c + b * c; } int main(void) { void* ...
user avatar
  • 4,132
-1votes
1answer
76views

how many numbers between 0-1 can a 32bit floating point represent?

I'm trying to figure out how many numbers between 0-1 can a 32bit floating point represent? is it just the same as in an interval? (2**24)-1
user avatar
0votes
0answers
56views

as.character() with numeric loses precision when a string is transformed into numeric

In R I want to run following code: str <- "2811.1019192874087" val <- as.numeric(str) str2 <- as.character(val) print(str2) print(val) It prints: "2811.10191928741" 2811....
user avatar
  • 1,526
0votes
0answers
32views

Expected value wrong with output with using long double [duplicate]

I am working on my knowledge in C++ and I am expecting a value which I did not receive. Here is my code #include <iostream> #include <float.h> using std:: cout; using std:: endl; using ...
user avatar
  • 15
5votes
1answer
113views

Why does a subtype of AbstractArray result in imprecise matrix operations in Julia?

I'm currently working on creating a subtype of AbstractArray in Julia, which allows you to store a vector in addition to an Array itself. You can think of it as the column "names", with ...
user avatar
  • 53
-3votes
2answers
60views

i need more decimal places for pi calculation [closed]

I'm trying to make a Pi calculator in python but I need more decimal places. it would help a lot if someone edited my code and carefully explained what they did. this is the code I'm using. import ...
user avatar
0votes
3answers
66views

Float precision when dealing with logarithms in musical pitch calculations

I'm writing a simple program to determine the difference between two musical pitches in cents; one cent is equal to 1/100th of a semitone. Dealing in cents is preferable for comparing musical pitches ...
user avatar
0votes
2answers
101views

comparing 2 floating numbers in matlab

I was tortured by the floating point comparison issue in Matlab in the previous a couple of weeks. My code involves tons of a<b or c<=d. What drives me crazy is that dw = 0.001; W1 = [0:dw:1]; ...
user avatar
  • 145
0votes
0answers
35views

R: difference between apply(object, 1, function(x) sum(x-a)/b) and rowsums((object-a)/b)

I'm new to R and am struggling with the apply function. It is really slow to execute and I was trying to optimize some code I received. I am trying to do some matrix operations (element-wise ...
user avatar
0votes
0answers
91views

Floating point in google colab for verification

I'm implementing an algorithm in Python (GoogleColab) and it outputs two lists; now, I would like to verify if this lists are equal, actualy I know that they should be equal. The problem is that ...
user avatar
  • 129
0votes
2answers
143views

Is there any way to not lose the precision and still get the value?

First off, I'm a total beginner at C, with prior experience of programming in Java and Python. The goal of the program was to add 2 numbers. While I was playing with the code, I encountered an issue ...
user avatar
0votes
1answer
32views

Python replace integers in a Counter with their permillages of the total [duplicate]

I am trying to generate readable word-like random strings not found in any dictionaries using Markov Chain. I have pulled a large amount of data of ngram frequencies from a total of 105230 words ...
user avatar
  • 371
3votes
1answer
77views

Calculating the angle between two vectors using a needle-like triangle

I implemented a function (angle_between) to calculate the angle between two vectors. It makes use of needle-like triangles and is based on Miscalculating Area and Angles of a Needle-like Triangle and ...
user avatar
1vote
1answer
74views

Convert double number to digits and vice versa?

I'm trying to convert a double number into array of digits Input: double num Output: int[] arrDigit int dotIdx bool isMinus for example: Input: double num = -69.69777 Output: ...
user avatar
  • 2,005
0votes
1answer
133views

Program to count number of digits in a double following decimal point in Java

Below is a program that is supposed to count the number of digits in a double value following the decimal point. On entering some double values, the program tends to start an infinite loop (possibly ...
user avatar
4votes
2answers
77views

Is the remainder operator susceptible to floating point errors?

I want to create a setter for a double variable num, but I would only like to update it if the input is a multiple of 0.5. Here's what I have, but I'm worried about floating-point errors. public void ...
user avatar
  • 163
-1votes
1answer
149views

pandas mess up multi level index parquet float accuracy

I have some data that has multi level index & column. My goal is to store the data as float32 instead of double/float64 into parquet file, to save disk space. I don't care about the accuracy of ...
user avatar
  • 511
3votes
2answers
533views

Euler-Mascheroni Constant

In programming, I used only Integers. But this time for some calculations. I need to calculate Euler-Mascheroni Constant γ . up to n-th decimal.{Though n ∈ [30, 150] is enough for me. [x] = gif(x) = ...
user avatar
2votes
2answers
56views

zsh loses precision when miltiplying two floats of significantly different magnitude

I'm trying to do some floating-point math in a zsh script. I'm seeing the following behavior: $ (( a = 1.23456789 * 0.00000001 )); printf "a = %g\n" $a a = 1.23e-08 $ (( a = 1.23456789 * 0....
user avatar
  • 23
1vote
0answers
42views

"Approximately equal to" operand in Python [duplicate]

Do you of you know what the "approximately equal to" operand is? I need to find numbers in a list that are approximately equal to a fixed number that I have defined. I tried doing =~ and == ~...
user avatar
-2votes
1answer
87views

Python Decimal Library is Imprecise? [duplicate]

I'm reading up on the Python Decimal module. I have a need to make a large number of precise calculations, often with lots of decimal places, where being off by a small amount adds up over time. Enter ...
user avatar
2votes
2answers
131views

Precision limits of numpy floats?

Given the sequence 1/1, 1/2, 1/3, ... , 1/n How can I calculate at what point I will not be able to distinguish with precision E between two consecutive elements 1/i and 1/i+1 if I use numpy.float16 ?...
user avatar
  • 6,076
8votes
2answers
326views

How many different sums can we get from very few floats?

Someone just asked why sum(myfloats) differed from sum(reversed(myfloats)). Quickly got duped to Is floating point math broken? and deleted. But it made me curious: How many different sums can we get ...
user avatar
  • 6,049
-1votes
1answer
38views

Convert a continuous range of doubles to a discrete int range

i want to map a continuous range of double values [10.0,20.0] to a byte in the range of [100,200]; The following must apply (even when converting back and forth multiple times): convertToByte(...
user avatar
0votes
1answer
118views

C - Float Imprecision, How to lock the decimal places for calculations?

I was making a little program to test floats in the C: The program itself is very simple, I just want to based on the User's input, return how much Dollar(s), Quarter(s)... etc, his number has. //-----...
user avatar
4votes
0answers
112views

Is there a way to force numpy.set_printoptions to show the exact float value?

Following question 59674518, is there a way for numpy.set_printoptions to ensure the EXACT float value is displayed, without displaying trailing zeros, and without knowing the value a priori? I have ...
user avatar
  • 571
0votes
1answer
68views

What are the details of the Juniper floating-point scenario/bug wherein the sticky bit is lost?

On some Juniper MX routers, floats are not handled correctly: The sticky bit is lost if it is shifted more that 8 bits to the right (underflow) during a calculation. Is there a workaround for this? ...
user avatar
  • 571
1vote
0answers
62views

When have enough bits of my series with non-negative terms been calculated?

I have a power series with all terms non-negative which I want to evaluate to some arbitrarily set precision p (the length in binary digits of a MPFR floating-point mantissa). The result should be ...
user avatar
0votes
0answers
52views

Simple addition in Perl causing result to be off by small fraction [duplicate]

#!/usr/bin/perl $a = 1; while ( $a <= 50 ) { print "$a\n"; $a += 0.025; } Very simple code. But here is how the output looks like... 2.225 2.25 2.275 2.3 2.325 2.35 2.375 ...
user avatar
1vote
0answers
160views

Inconsistent results between GCC and Clang for simple floating point calculation

On my Intel x86_64 machine, this C++ code generates different sequences on Clang vs GCC: #include <iostream> namespace { template<typename Out> constexpr auto caster{[](auto x) constexpr ...
user avatar
0votes
1answer
66views

Problem of difference in between matrix multiplication done by * and by components by components by hand (MATLAB)

I have a problem with the accuracy of matrix multiplication on Matlab. I will explain this with an example. Let A=[0.1256, 0.25789; 0.00018, 0.68741]; B=[0.1578, 0.24471; 0.12801, 0.99701]; s=0; ...
user avatar
  • 111

15 30 50 per page
1
2 3 4 5
26