# All Questions

44 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
115views

### Is there a bug in controlled rounding using `exp`?

I'm observing incorrect (IMO) rounding behaviour on some platforms as follows: Calculate the value of log(2) under rounding modes to FE_DOWNWARD and FE_UPWARD (see <fenv.h>). In all cases I've ...
384views

### C# Change FPU rounding mode

I'm attempting to write an interval arithmetic library in C# .NET, but in order to do this accurately I need to be able to control the rounding mode of floating point operations. After a bit of ...
• 639
151views

### Correctly rounding a trigonometric function for single-precision

I want a correctly rounded (round to nearest ties to even) single-precision trigonometric function (0.5 ulp error). I can use either the CORDIC algorithm or one of the polynomial approximation ...
• 440
169views

Is there a built-in function in Haskell that rounds a real floating-point number to the nearest whole number, without changing the type of said number? sameTypeRound f == fromIntegral (round f)
• 1,551
312views

### Is it safe to cast Math.Round result to float?

A colleague has written some code along these lines: var roundedNumber = (float) Math.Round(someFloat, 2); Console.WriteLine(roundedNumber); I have an uncertainty about this code - is the number ...
• 124k
96views

### How to compute trunc(a/b) with only the nearest-to-even rounding mode?

Given two IEEE-754 double-precision floating-point numbers a and b, I want to get the exact quotient a/b rounded to an integer towards zero. A C99 program to do that could look like this: #include <...
• 5,437
253views

### Stochastically rounding a float to an integer

I want a function (using Python 3.6+ if it's relevant) that will stochastically round a floating-point number to an integer in the following manner: Given a real number, x, let a = floor(x) and let b ...
• 533
735views

### Simple alternatives to floating-point when performing arithmetic on short string-encoded rationals

I am creating unit tests for a function that rounds "rational" numbers stored as strings. The current rounding implementation casts the strings to a floating point type: #include <boost/...
• 23
81views

### Inconsistent rounding in OPA

There seems to be a different in rounding in OPA10 versus OPA12. The number 27,775 gives an amount of 27,77 in OPA12 whereas OPA10 gives a result of 27,78. Another variable wih value gives 83,325 ...
• 29
1kviews

### Truncate (not round) decimal in SQL

I'm doing astored procedure using a MySQL database. How could I round or truncate a FLOAT variable with diferent decimals? I've found this answer: Truncate (not round) decimal places in SQL Server ...
• 131
1vote
41views

### Is it considered normal that under FE_DOWNWARD or FE_TOWARDZERO expression FLT_MAX * FLT_MAX evaluates to FLT_MAX?

Sample code: #include <float.h> #include <fenv.h> #pragma STDC FENV_ACCESS ON int main(void) { if (fesetround(RM) != 0) return 2; return ((FLT_MAX * FLT_MAX) == FLT_MAX) ? 0 : 1; ...
• 4,132
1vote
102views

### How to round double / float in vala?

How can I round var x = 0.10000000000000001 to 0.10?
• 11
1vote
428views

### How does arithmetic floating point rounding in RISC-V work?

I am currently working on my own RISC-V (rv64gc) emulator. Everything went smoothly so far, however the floating point rounding mode is giving me a headache. The RV ISA comprises of the following 5 ...
1vote
163views

### SQL Server truncating values of a whole number in a float field when cast to CHAR

I have a rather strange problem. Due to sourcing one column in a large query from both FLOAT and CHAR columns I'm casting the resulting FLOAT to CHAR so that the data types for the resulting query ...
• 424
1vote
297views

### How to show amount in round figure for Odoo 10

I have a float field and i can calculate amount within this field. I want that amount should show in round figure. For eg. when price is 64.80 then it updated to 65 and if price 64.40 then it updated ...
• 1,156
1vote
720views

### In MIPS assembly language, how does one round a floating point value to the nearest .01?

If I had the floating point (saved in some \$f register) 1684.714 how could I round that to 1684.71? If I had the floating point 1684.716, how could I round that to 1684.72?
1vote
330views

### Why is Round Nearest Ties Away Rounding mode optional in the IEEE754 standard for Binary formats?

As Quoted from IEEE Standard for Floating-Point Arithmetic (2008) Section 4.3.3 An implementation of this standard shall provide roundTiesToEven and the three directed rounding attributes. A ...
• 11
1vote
57views

### How many bits does IEEE floating point uses in the normalization step?

Using Matlab, for example. Suppose I give an input a=0.1 The first step is to convert this string in the binary notation. In this step, how many bits the language takes into account to go to the ...
1vote
1kviews

### php round number - When echo it displays 2 decimals, if print to CSV it displays .0999999

Using http://www.maatwebsite.nl/laravel-excel/ to export to CSV and Excel. I have a \$rows array wich contains all rows with it's columns. Some columns have a formatted number using number_format(...
• 183
1vote
90views

### Why might IE give a wrong result for Math.floor()

I've got logs from my client-side app which show IE occasionally having trouble with Math.floor. Here's an example of one of these logs: var d = 1442167625975 d / 16 = 90135476623.4375 Math.floor(d ...
• 1,239
1vote
254views

### Function to truncate values of resultset in java

I have to truncate the values obtained from database upto two fractional parts and put that values in the resultset I'm adding values from resultset into list in following way while(rs.next()) ...
1vote
1kviews

### mysql float data type round off's values after 6 decimal places

I have a table (say table1) with one field of data type float (say field1). The values in this field are all unique with upto 8 decimal places. I executed following queries on this table: SELECT ...
• 1,880
1vote
88views

### Blitting with a camera leads to seams (due to floating point errors?)

I'm working on a tilemap implementation with blitting (in Flash but I don't think it's language relevant). The problem I'm having is that as I pan my camera around, 1 pixel seams start to become ...
24views

### Rounding issue for float type in Snowflake

I have two columns (float type) as below: MRR Standard_MRR 629.0625 687.5 I need to do the calculation - (1-MRR/Standard_MRR)=0.085000 If I round the result to 2 decimal place and it will ...
• 11
39views

### changing rounding mode via importing libm in python 3

My environment: Ubuntu 18.04, Anaconda, Python 3.6 I am using following code to import libm in python via ctypes in order to change floating type environment such as rounding. import numpy as np ...
• 35
108views

### How to calculate or estimate floating point rounding error for a specific number

I'm converting 64 bit numbers to 32 bit numbers and in doing the conversion I want to ensure that my numbers only get bigger in the conversion and not smaller. I know that the rounding error for ...
• 2,148
38views

### Python rouding float with decimal module working, but not when using values in list afterwards

I know that there are a lot of similar questions, but I was not able to find a solution in any of them. I rounding a float to 2 decimal digits and am using the decimal module for this purpose - as ...
94views

### Best practices to avoid Python float rounding errors?

So I encountered my first noticeable issue with float representation in Python. The error came from multiplying the float:2.55 by the float:100. The error being I expect 255.0, but python returns 254....
75views

### Rounding IEEE floating point numbers

I'm currently doing an assignment where I have to find the value of a given IEEE floating point representation. For the assignment, there is 1 singed bit, 5 exponent bits and 3 frac bits. The bits ...
58views

### Why does 0.32 round up to 1 when multiplying

this is not about floating point math in general being broken... it IS NOT THE SAME ISSUE - this is likely an automatic type conversion - not float precision issue: So I'm trying to write an a small ...
• 751
188views

### Template Multiplication Function

I would like to make a general function that will take two parameters of any data type (although they will always be some sort of a number) and then multiply them together, round them down, and return ...
• 87
552views

### How to round a float in the model so you don't have to round it elsewhere (it is just saved in rounded form)

I was wondering if there was a way to round a number in the model, so that I would not have to round the number to my specified decimal places anywhere else but there. I looked at these two posts, ...
697views

### Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
32views

### Rounding floating point number

I have a number with an exponent field of 4 and a fraction field of 5. The number I need to round is −(1.0110101)2, and I have to round down, round up, round to zero, round nearest. So I have -(1....
• 119
62views

### Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question. I'm doing a formula in TSQL like this: @Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1)) + ((@...
• 45
1kviews

### Rounding off floats in Python after json.dumps(), urlencode

This is a follow-up question to this and I looked at this one as well. My Python script collects data during every run and sends data to a URL. The URL should display data/values up to 2 decimal ...
• 917
1kviews

### Saving float number in SQLite issue?

I tried saving some floats into SQLite tables (for the first time) but the stored values are different from what I wanted to save. They were nearly exactly the numbers I saved. This requires me to ...
• 59k
432views

### Nicest way to solve stringified floating rounding issue through algorithm? (Java, JEval)

I have an output operation from JEval that is an string with the float imprecision, something like "1.56700000001". I need some approach to hold the maximum precision but correcting the float ...
• 6,041
46views

### Float rounded to Multiple then changed to string with python?

I need to round down to the nearest multiple, for a lot of pairs of numbers. And then get the result as a string (rather than scientific notation that floats output with). Considering just two, a and ...
303views

### adding two float values by using python 3

I am trying to add a two float values(6.66,12.208) in python3 but it return 19.868000000000002. is it right? can someone please tell me the reason
180views

### How can you accurately round a floating point number in javascript?

I have looked in many places for an answer for this including but not limited to https://stackoverflow.com, and I am not here attempting to duplicate a question. I'm here to get a definitive ...
• 85