All Questions
Tagged with floating-point rounding
44
questions with no upvoted or accepted answers
4votes
3answers
115views
Is there a bug in controlled rounding using `exp`?
I'm observing incorrect (IMO) rounding behaviour on some platforms as follows:
Calculate the value of log(2) under rounding modes to FE_DOWNWARD and FE_UPWARD (see <fenv.h>). In all cases I've ...
- 41
4votes
0answers
384views
C# Change FPU rounding mode
I'm attempting to write an interval arithmetic library in C# .NET, but in order to do this accurately I need to be able to control the rounding mode of floating point operations.
After a bit of ...
- 639
3votes
0answers
151views
Correctly rounding a trigonometric function for single-precision
I want a correctly rounded (round to nearest ties to even) single-precision trigonometric function (0.5 ulp error).
I can use either the CORDIC algorithm or one of the polynomial approximation ...
- 440
3votes
0answers
169views
Type-preserving rounding in Haskell
Is there a built-in function in Haskell that rounds a real floating-point number to the nearest whole number, without changing the type of said number?
sameTypeRound f == fromIntegral (round f)
- 1,551
3votes
1answer
312views
Is it safe to cast Math.Round result to float?
A colleague has written some code along these lines:
var roundedNumber = (float) Math.Round(someFloat, 2);
Console.WriteLine(roundedNumber);
I have an uncertainty about this code - is the number ...
- 124k
2votes
1answer
96views
How to compute trunc(a/b) with only the nearest-to-even rounding mode?
Given two IEEE-754 double-precision floating-point numbers a and b, I want to get the exact quotient a/b rounded to an integer towards zero.
A C99 program to do that could look like this:
#include <...
- 5,437
2votes
1answer
253views
Stochastically rounding a float to an integer
I want a function (using Python 3.6+ if it's relevant) that will stochastically round a floating-point number to an integer in the following manner:
Given a real number, x, let a = floor(x) and let b ...
- 533
2votes
2answers
735views
Simple alternatives to floating-point when performing arithmetic on short string-encoded rationals
I am creating unit tests for a function that rounds "rational" numbers stored as strings. The current rounding implementation casts the strings to a floating point type:
#include <boost/...
- 23
2votes
0answers
81views
Inconsistent rounding in OPA
There seems to be a different in rounding in OPA10 versus OPA12.
The number 27,775 gives an amount of 27,77 in OPA12 whereas OPA10 gives a result of 27,78.
Another variable wih value gives 83,325 ...
- 29
2votes
0answers
1kviews
Truncate (not round) decimal in SQL
I'm doing astored procedure using a MySQL database.
How could I round or truncate a FLOAT variable with diferent decimals?
I've found this answer: Truncate (not round) decimal places in SQL Server
...
- 131
1vote
0answers
41views
Is it considered normal that under FE_DOWNWARD or FE_TOWARDZERO expression FLT_MAX * FLT_MAX evaluates to FLT_MAX?
Sample code:
#include <float.h>
#include <fenv.h>
#pragma STDC FENV_ACCESS ON
int main(void)
{
if (fesetround(RM) != 0) return 2;
return ((FLT_MAX * FLT_MAX) == FLT_MAX) ? 0 : 1;
...
- 4,132
1vote
3answers
102views
1vote
1answer
428views
How does arithmetic floating point rounding in RISC-V work?
I am currently working on my own RISC-V (rv64gc) emulator. Everything went smoothly so far, however the floating point rounding mode is giving me a headache.
The RV ISA comprises of the following 5 ...
1vote
1answer
163views
SQL Server truncating values of a whole number in a float field when cast to CHAR
I have a rather strange problem.
Due to sourcing one column in a large query from both FLOAT and CHAR columns I'm casting the resulting FLOAT to CHAR so that the data types for the resulting query ...
- 424
1vote
3answers
297views
How to show amount in round figure for Odoo 10
I have a float field and i can calculate amount within this field. I want that amount should show in round figure. For eg. when price is 64.80 then it updated to 65 and if price 64.40 then it updated ...
- 1,156
1vote
1answer
720views
In MIPS assembly language, how does one round a floating point value to the nearest .01?
If I had the floating point (saved in some $f register) 1684.714 how could I round that to 1684.71? If I had the floating point 1684.716, how could I round that to 1684.72?
- 15
1vote
0answers
330views
Why is Round Nearest Ties Away Rounding mode optional in the IEEE754 standard for Binary formats?
As Quoted from IEEE Standard for Floating-Point Arithmetic (2008) Section 4.3.3
An implementation of this standard shall provide roundTiesToEven and the three directed rounding attributes. A ...
- 11
1vote
2answers
57views
How many bits does IEEE floating point uses in the normalization step?
Using Matlab, for example. Suppose I give an input
a=0.1
The first step is to convert this string in the binary notation. In this step, how many bits the language takes into account to go to the ...
- 11
1vote
1answer
1kviews
php round number - When echo it displays 2 decimals, if print to CSV it displays .0999999
Using http://www.maatwebsite.nl/laravel-excel/ to export to CSV and Excel.
I have a $rows array wich contains all rows with it's columns.
Some columns have a formatted number using number_format(...
- 183
1vote
0answers
90views
Why might IE give a wrong result for Math.floor()
I've got logs from my client-side app which show IE occasionally having trouble with Math.floor.
Here's an example of one of these logs:
var d = 1442167625975
d / 16 = 90135476623.4375
Math.floor(d ...
- 1,239
1vote
2answers
254views
Function to truncate values of resultset in java
I have to truncate the values obtained from database upto two fractional parts and put that values in the resultset I'm adding values from resultset into list in following way
while(rs.next())
...
- 526
1vote
0answers
1kviews
mysql float data type round off's values after 6 decimal places
I have a table (say table1) with one field of data type float (say field1). The values in this field are all unique with upto 8 decimal places.
I executed following queries on this table:
SELECT ...
- 1,880
1vote
0answers
88views
Blitting with a camera leads to seams (due to floating point errors?)
I'm working on a tilemap implementation with blitting (in Flash but I don't think it's language relevant). The problem I'm having is that as I pan my camera around, 1 pixel seams start to become ...
0votes
1answer
24views
Rounding issue for float type in Snowflake
I have two columns (float type) as below:
MRR Standard_MRR
629.0625 687.5
I need to do the calculation - (1-MRR/Standard_MRR)=0.085000
If I round the result to 2 decimal place and it will ...
- 11
0votes
1answer
39views
changing rounding mode via importing libm in python 3
My environment: Ubuntu 18.04, Anaconda, Python 3.6
I am using following code to import libm in python via ctypes in order to change floating type environment such as rounding.
import numpy as np
...
- 35
0votes
0answers
108views
How to calculate or estimate floating point rounding error for a specific number
I'm converting 64 bit numbers to 32 bit numbers and in doing the conversion I want to ensure that my numbers only get bigger in the conversion and not smaller. I know that the rounding error for ...
- 2,148
0votes
1answer
38views
Python rouding float with decimal module working, but not when using values in list afterwards
I know that there are a lot of similar questions, but I was not able to find a solution in any of them.
I rounding a float to 2 decimal digits and am using the decimal module for this purpose - as ...
- 1
0votes
0answers
94views
Best practices to avoid Python float rounding errors?
So I encountered my first noticeable issue with float representation in Python.
The error came from multiplying the float:2.55 by the float:100. The error being I expect 255.0, but python returns 254....
- 11
0votes
1answer
75views
Rounding IEEE floating point numbers
I'm currently doing an assignment where I have to find the value of a given IEEE floating point representation. For the assignment, there is 1 singed bit, 5 exponent bits and 3 frac bits.
The bits ...
- 19
0votes
0answers
58views
Why does 0.32 round up to 1 when multiplying
this is not about floating point math in general being broken... it IS NOT THE SAME ISSUE - this is likely an automatic type conversion - not float precision issue:
So I'm trying to write an a small ...
- 751
0votes
1answer
188views
Template Multiplication Function
I would like to make a general function that will take two parameters of any data type (although they will always be some sort of a number) and then multiply them together, round them down, and return ...
- 87
0votes
2answers
552views
How to round a float in the model so you don't have to round it elsewhere (it is just saved in rounded form)
I was wondering if there was a way to round a number in the model, so that I would not have to round the number to my specified decimal places anywhere else but there.
I looked at these two posts, ...
- 205
0votes
1answer
697views
Rounding Off function to given decimal without using round function
I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
- 1
0votes
0answers
32views
Rounding floating point number
I have a number with an exponent field of 4 and a fraction field of 5.
The number I need to round is −(1.0110101)2, and I have to round down, round up, round to zero, round nearest.
So I have -(1....
- 119
0votes
1answer
62views
Avoid rounding to 0 when a result is very little
I've searched for this but i didn't find anything, i hope this is not a doubled question.
I'm doing a formula in TSQL like this:
@Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1))
+ ((@...
- 45
0votes
0answers
1kviews
Rounding off floats in Python after json.dumps(), urlencode
This is a follow-up question to this and I looked at this one as well. My Python script collects data during every run and sends data to a URL. The URL should display data/values up to 2 decimal ...
- 917
0votes
0answers
1kviews
Saving float number in SQLite issue?
I tried saving some floats into SQLite tables (for the first time) but the stored values are different from what I wanted to save. They were nearly exactly the numbers I saved. This requires me to ...
- 59k
0votes
1answer
432views
Nicest way to solve stringified floating rounding issue through algorithm? (Java, JEval)
I have an output operation from JEval that is an string with the float imprecision, something like "1.56700000001". I need some approach to hold the maximum precision but correcting the float ...
- 6,041
0votes
1answer
46views
Float rounded to Multiple then changed to string with python?
I need to round down to the nearest multiple, for a lot of pairs of numbers. And then get the result as a string (rather than scientific notation that floats output with). Considering just two, a and ...
-1votes
1answer
303views
adding two float values by using python 3
I am trying to add a two float values(6.66,12.208) in python3 but it return 19.868000000000002. is it right? can someone please tell me the reason
-2votes
1answer
180views
How can you accurately round a floating point number in javascript?
I have looked in many places for an answer for this including but not
limited to https://stackoverflow.com, and I am not here
attempting to duplicate a question. I'm here to get a definitive
...
- 85
-3votes
4answers
74views
How to round all elements of a list using a list comprehension?
Considering the following list of float values
aList = ['3.32', '1.08', '4.23']
How do I use a list comprehension to round it and obtain the following [3.0, 1.0, 4.0]?
-5votes
1answer
420views
why Float 999999.99 auto round to 1000000.0, but 99999.99 ,99999999, wont auto round
java. i have known "A float is generally subject to round-offs in its 8th significant digit",and using double will avoid that problem.
can anyone tell me
why it will round-offs in its 8th ...
