All Questions

44 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4votes
3answers
115views

Is there a bug in controlled rounding using `exp`?

I'm observing incorrect (IMO) rounding behaviour on some platforms as follows: Calculate the value of log(2) under rounding modes to FE_DOWNWARD and FE_UPWARD (see <fenv.h>). In all cases I've ...
user avatar
4votes
0answers
384views

C# Change FPU rounding mode

I'm attempting to write an interval arithmetic library in C# .NET, but in order to do this accurately I need to be able to control the rounding mode of floating point operations. After a bit of ...
user avatar
  • 639
3votes
0answers
151views

Correctly rounding a trigonometric function for single-precision

I want a correctly rounded (round to nearest ties to even) single-precision trigonometric function (0.5 ulp error). I can use either the CORDIC algorithm or one of the polynomial approximation ...
user avatar
  • 440
3votes
0answers
169views

Type-preserving rounding in Haskell

Is there a built-in function in Haskell that rounds a real floating-point number to the nearest whole number, without changing the type of said number? sameTypeRound f == fromIntegral (round f)
user avatar
3votes
1answer
312views

Is it safe to cast Math.Round result to float?

A colleague has written some code along these lines: var roundedNumber = (float) Math.Round(someFloat, 2); Console.WriteLine(roundedNumber); I have an uncertainty about this code - is the number ...
user avatar
  • 124k
2votes
1answer
96views

How to compute trunc(a/b) with only the nearest-to-even rounding mode?

Given two IEEE-754 double-precision floating-point numbers a and b, I want to get the exact quotient a/b rounded to an integer towards zero. A C99 program to do that could look like this: #include <...
user avatar
  • 5,437
2votes
1answer
253views

Stochastically rounding a float to an integer

I want a function (using Python 3.6+ if it's relevant) that will stochastically round a floating-point number to an integer in the following manner: Given a real number, x, let a = floor(x) and let b ...
user avatar
  • 533
2votes
2answers
735views

Simple alternatives to floating-point when performing arithmetic on short string-encoded rationals

I am creating unit tests for a function that rounds "rational" numbers stored as strings. The current rounding implementation casts the strings to a floating point type: #include <boost/...
user avatar
2votes
0answers
81views

Inconsistent rounding in OPA

There seems to be a different in rounding in OPA10 versus OPA12. The number 27,775 gives an amount of 27,77 in OPA12 whereas OPA10 gives a result of 27,78. Another variable wih value gives 83,325 ...
user avatar
  • 29
2votes
0answers
1kviews

Truncate (not round) decimal in SQL

I'm doing astored procedure using a MySQL database. How could I round or truncate a FLOAT variable with diferent decimals? I've found this answer: Truncate (not round) decimal places in SQL Server ...
user avatar
  • 131
1vote
0answers
41views

Is it considered normal that under FE_DOWNWARD or FE_TOWARDZERO expression FLT_MAX * FLT_MAX evaluates to FLT_MAX?

Sample code: #include <float.h> #include <fenv.h> #pragma STDC FENV_ACCESS ON int main(void) { if (fesetround(RM) != 0) return 2; return ((FLT_MAX * FLT_MAX) == FLT_MAX) ? 0 : 1; ...
user avatar
  • 4,132
1vote
3answers
102views

How to round double / float in vala?

How can I round var x = 0.10000000000000001 to 0.10?
user avatar
  • 11
1vote
1answer
428views

How does arithmetic floating point rounding in RISC-V work?

I am currently working on my own RISC-V (rv64gc) emulator. Everything went smoothly so far, however the floating point rounding mode is giving me a headache. The RV ISA comprises of the following 5 ...
user avatar
1vote
1answer
163views

SQL Server truncating values of a whole number in a float field when cast to CHAR

I have a rather strange problem. Due to sourcing one column in a large query from both FLOAT and CHAR columns I'm casting the resulting FLOAT to CHAR so that the data types for the resulting query ...
user avatar
1vote
3answers
297views

How to show amount in round figure for Odoo 10

I have a float field and i can calculate amount within this field. I want that amount should show in round figure. For eg. when price is 64.80 then it updated to 65 and if price 64.40 then it updated ...
user avatar
  • 1,156
1vote
1answer
720views

In MIPS assembly language, how does one round a floating point value to the nearest .01?

If I had the floating point (saved in some $f register) 1684.714 how could I round that to 1684.71? If I had the floating point 1684.716, how could I round that to 1684.72?
user avatar
1vote
0answers
330views

Why is Round Nearest Ties Away Rounding mode optional in the IEEE754 standard for Binary formats?

As Quoted from IEEE Standard for Floating-Point Arithmetic (2008) Section 4.3.3 An implementation of this standard shall provide roundTiesToEven and the three directed rounding attributes. A ...
user avatar
1vote
2answers
57views

How many bits does IEEE floating point uses in the normalization step?

Using Matlab, for example. Suppose I give an input a=0.1 The first step is to convert this string in the binary notation. In this step, how many bits the language takes into account to go to the ...
user avatar
1vote
1answer
1kviews

php round number - When echo it displays 2 decimals, if print to CSV it displays .0999999

Using http://www.maatwebsite.nl/laravel-excel/ to export to CSV and Excel. I have a $rows array wich contains all rows with it's columns. Some columns have a formatted number using number_format(...
user avatar
  • 183
1vote
0answers
90views

Why might IE give a wrong result for Math.floor()

I've got logs from my client-side app which show IE occasionally having trouble with Math.floor. Here's an example of one of these logs: var d = 1442167625975 d / 16 = 90135476623.4375 Math.floor(d ...
user avatar
  • 1,239
1vote
2answers
254views

Function to truncate values of resultset in java

I have to truncate the values obtained from database upto two fractional parts and put that values in the resultset I'm adding values from resultset into list in following way while(rs.next()) ...
user avatar
1vote
0answers
1kviews

mysql float data type round off's values after 6 decimal places

I have a table (say table1) with one field of data type float (say field1). The values in this field are all unique with upto 8 decimal places. I executed following queries on this table: SELECT ...
user avatar
  • 1,880
1vote
0answers
88views

Blitting with a camera leads to seams (due to floating point errors?)

I'm working on a tilemap implementation with blitting (in Flash but I don't think it's language relevant). The problem I'm having is that as I pan my camera around, 1 pixel seams start to become ...
user avatar
0votes
1answer
24views

Rounding issue for float type in Snowflake

I have two columns (float type) as below: MRR Standard_MRR 629.0625 687.5 I need to do the calculation - (1-MRR/Standard_MRR)=0.085000 If I round the result to 2 decimal place and it will ...
user avatar
  • 11
0votes
1answer
39views

changing rounding mode via importing libm in python 3

My environment: Ubuntu 18.04, Anaconda, Python 3.6 I am using following code to import libm in python via ctypes in order to change floating type environment such as rounding. import numpy as np ...
user avatar
  • 35
0votes
0answers
108views

How to calculate or estimate floating point rounding error for a specific number

I'm converting 64 bit numbers to 32 bit numbers and in doing the conversion I want to ensure that my numbers only get bigger in the conversion and not smaller. I know that the rounding error for ...
user avatar
0votes
1answer
38views

Python rouding float with decimal module working, but not when using values in list afterwards

I know that there are a lot of similar questions, but I was not able to find a solution in any of them. I rounding a float to 2 decimal digits and am using the decimal module for this purpose - as ...
user avatar
  • 1
0votes
0answers
94views

Best practices to avoid Python float rounding errors?

So I encountered my first noticeable issue with float representation in Python. The error came from multiplying the float:2.55 by the float:100. The error being I expect 255.0, but python returns 254....
user avatar
0votes
1answer
75views

Rounding IEEE floating point numbers

I'm currently doing an assignment where I have to find the value of a given IEEE floating point representation. For the assignment, there is 1 singed bit, 5 exponent bits and 3 frac bits. The bits ...
user avatar
0votes
0answers
58views

Why does 0.32 round up to 1 when multiplying

this is not about floating point math in general being broken... it IS NOT THE SAME ISSUE - this is likely an automatic type conversion - not float precision issue: So I'm trying to write an a small ...
user avatar
  • 751
0votes
1answer
188views

Template Multiplication Function

I would like to make a general function that will take two parameters of any data type (although they will always be some sort of a number) and then multiply them together, round them down, and return ...
user avatar
  • 87
0votes
2answers
552views

How to round a float in the model so you don't have to round it elsewhere (it is just saved in rounded form)

I was wondering if there was a way to round a number in the model, so that I would not have to round the number to my specified decimal places anywhere else but there. I looked at these two posts, ...
user avatar
0votes
1answer
697views

Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
user avatar
  • 1
0votes
0answers
32views

Rounding floating point number

I have a number with an exponent field of 4 and a fraction field of 5. The number I need to round is −(1.0110101)2, and I have to round down, round up, round to zero, round nearest. So I have -(1....
user avatar
  • 119
0votes
1answer
62views

Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question. I'm doing a formula in TSQL like this: @Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1)) + ((@...
user avatar
  • 45
0votes
0answers
1kviews

Rounding off floats in Python after json.dumps(), urlencode

This is a follow-up question to this and I looked at this one as well. My Python script collects data during every run and sends data to a URL. The URL should display data/values up to 2 decimal ...
user avatar
  • 917
0votes
0answers
1kviews

Saving float number in SQLite issue?

I tried saving some floats into SQLite tables (for the first time) but the stored values are different from what I wanted to save. They were nearly exactly the numbers I saved. This requires me to ...
user avatar
  • 59k
0votes
1answer
432views

Nicest way to solve stringified floating rounding issue through algorithm? (Java, JEval)

I have an output operation from JEval that is an string with the float imprecision, something like "1.56700000001". I need some approach to hold the maximum precision but correcting the float ...
user avatar
  • 6,041
0votes
1answer
46views

Float rounded to Multiple then changed to string with python?

I need to round down to the nearest multiple, for a lot of pairs of numbers. And then get the result as a string (rather than scientific notation that floats output with). Considering just two, a and ...
user avatar
-1votes
1answer
303views

adding two float values by using python 3

I am trying to add a two float values(6.66,12.208) in python3 but it return 19.868000000000002. is it right? can someone please tell me the reason
user avatar
-2votes
1answer
180views

How can you accurately round a floating point number in javascript?

I have looked in many places for an answer for this including but not limited to https://stackoverflow.com, and I am not here attempting to duplicate a question. I'm here to get a definitive ...
user avatar
  • 85
-3votes
4answers
74views

How to round all elements of a list using a list comprehension?

Considering the following list of float values aList = ['3.32', '1.08', '4.23'] How do I use a list comprehension to round it and obtain the following [3.0, 1.0, 4.0]?
user avatar
-5votes
1answer
420views

why Float 999999.99 auto round to 1000000.0, but 99999.99 ,99999999, wont auto round

java. i have known "A float is generally subject to round-offs in its 8th significant digit",and using double will avoid that problem. can anyone tell me why it will round-offs in its 8th ...
user avatar