# All Questions

Tagged with floating-point decimal

57
questions with no upvoted or accepted answers

9votes

1answer

596views

### Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism:
Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...

3votes

0answers

2kviews

### How to format float to 4 decimal places within json dumps?

Have difficulty in converting pf_stats output to have 4 decimal places:
import json
import numpy as np
def run_simulation(H, P, B, C, mu, sigma, T, L):
x = normal(mu, sigma, (L, T))
pf_all ...

2votes

0answers

655views

### Handle money and currencies in javascript multiplying and dividing by 100

I'm writing an app for myself. I need to handle money and currencies.
I read this and I want to try the second approach:
You can multiply floats into integers before you calculate, then divide them ...

2votes

0answers

3kviews

### Python - How can I change 1E-8 to 0.00000001 without changing type?

Python's decimal.Decimal automatically converts the following number (7 zeros and a one, after the decimal point) into scientific notation:
d = decimal.Decimal('0.00000001')
import simplejson; ...

2votes

0answers

3kviews

### Python float types vs Decimal

I am an intern at TCD, Physics.
I wrote a code to perform some data analysis on random particle packings. The code is written in Python.
The code reads in columns of data from a .txt file, provided ...

2votes

1answer

228views

### How to convert string value to Number for exact calculation ? Javascript

Below is the code :
var a ="10.9549999999999141";
console.log(a);
console.log(typeof a);
console.log(Number(a));
Now the above code results as below :
"10.9549999999999141"
"string"
10....

2votes

5answers

2kviews

### Join two strings into a single decimal value

I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation?
decimal value; //like 500.45
string txt1, txt2;
txt1 = "500";
txt2 = "45";
value = Convert....

1vote

1answer

38views

### Getting different values for Decimal LocalContext Type

I have written following code to determine type of decimal localcontext:
print('decimal.localcontext() type: ', type(decimal.localcontext()))
Output:
decimal.localcontext() type: <class 'decimal....

1vote

1answer

40views

### How to get first 4 digits after the 0s in a decimal but keeping 0s • Python

I need the first 4 digits after the front 0s in a decimal while also keeping the 0s in the output without getting scientific notation.
Would like to take
0.0000000000000000000000634546534
and get
0....

1vote

1answer

64views

### Confused by loss of float precision in financial application with python Decimal package

I'm having a problem in financial application when calculate 70000.0*5.65500*18.0/36000.0 and compare the result with another number.
The accurate result is 197.925
When use Decimal, the results ...

1vote

0answers

100views

### How to use Decimal64 DPD with gcc

I have been trying to figure out how to integrate Decimal Floating Types with GCC 10.2.0.
I built gcc with --enable-decimal-float=dpd and --with-backend=libdecnumber flags, and specified DFP_ENABLE=1 ...

1vote

0answers

69views

### 9 Bit Floating Point to Hex

I've been running into an issue where I'm trying to convert a 9-bit floating point number to hex, where the floating point scheme is 1 sign-bit, followed by 4 bit exponent, and then a 4-bit mantissa. ...

1vote

1answer

35views

### Decimal-module is not working on binary level? How?

I'm a beginner when it comes to what's going on in the background when I start the program. Right now my focus is on the difference between "decimal" and "float" in "Python&...

1vote

0answers

39views

### How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding?
For instance:
import math
def truncate(f, n):
return math.floor(f * 10 ** n) / 10 ** n
r = lambda f,p: f - f % p
size = 3245.249999999999853146309903
y ...

1vote

1answer

180views

### How to get the number of decimal places of a float value?

I want to express the final product rounded down according to scientific notation.
So if 80.55 and 879.5689 give 70849.281250 the final print functions outputs 70849.28.
Is there a way to get the ...

1vote

0answers

57views

### Why does `decimal.Decimal` still give rounding errors?

I am using decimal.Decimal instead of float explicitly to avoid rounding errors, but am still getting them:
>>> from decimal import Decimal
>>> Decimal('9') * (Decimal('10') / ...

1vote

0answers

73views

### Highly precise division, multiplication, and exponentiation of large complex numbers

I am working on a project that requires highly precise division of large numbers which will sometimes be complex numbers. I need to do this in python, preferably python 3.7, but everything I have ...

1vote

1answer

318views

### Python:computing the number of digits after decimal gives wrong answer

I have a floating point field in one of my form, consider it as field_x. Based on that field_x i have some computation.
After all if field_x have n digits after decimal result also should have n ...

1vote

1answer

745views

### Serialize a decimal with simplejson

I have been trying to serialize a decimal value with a string key but so far i only managed to make a successful 1 decimal digit (like '16.1'). I tried both json and simplejson libraries, subclassing ...

1vote

2answers

148views

### Reversible decimal to double conversion

I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...

1vote

1answer

131views

### How is this number expressed as a multiple of 2^-22*(1/10)? How is that derived?

I was refreshing on binary and floats and I run into the following example:
0.1 is represented as: 0.0001100110011001100110011[0011] with the part in the brackets repeating.
If we round up this ...

1vote

1answer

43views

### python hack converting floats to decimals

I've written a large program, with dependencies on libraries written in my lab. I'm getting wrong (and somewhat random) results, which are caused by floating-point errors.
I would like to do some ...

1vote

1answer

261views

### python float addition results with unexpected decimal places

I have the following code to detect stock trades by counting orders:
import json
orders = '[{"side": "SELL", "executedQty": "0.046"}, {"side": "SELL", "executedQty": "0.041"}, {"side": "SELL", "...

0votes

1answer

52views

### Accelerometer data from hex to float

I am trying to get accelerometer data off a poorly documented BLE beacon. Here is what I receive (as NSDictionary):
{
"x-Data" = 0020;
"y-Data" = 0000;
"z-Data&...

0votes

1answer

410views

### Logical operation with float number (with C & python)

I'm a Python newbie and currently comparing python with C language using logical operations.
My question is
(I could solve Q1 (logical operation in C), thank you for your comments!)
Q1: Logical ...

0votes

1answer

47views

### how to restrict float number to 1 digit after decimal point

I have a float number 242.1356783 and -87.10234512,
I need to restrict it it to 242.1
I tried below on cortex m4 microcontroller
1.data[i] = (int)(data[i] *10);
data[i] = data[i]/10;
2.data[i]= ceilf(...

0votes

0answers

63views

### C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database

I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...

0votes

2answers

77views

### The decimal.Decimal() function is producing an unrounded result

I am new to Python and I am having trouble with the Decimal function in the decimal module in Python. I understand that due to the way that Python stores float values in memory, in order to perform ...

0votes

0answers

48views

### String to float conversion not working php

I've a string value of the type "x.yy" which I want to convert to double/float.
I've tried floatval(), doubleval() and float and double type casting as well but it only gives me "X"...

0votes

0answers

83views

### How to deal with decimal operation errors in dart?

I am trying to fix the decimal error generated in my program.
For example I have the code:-
void main() {
double a = 300;
double b = 150;
double c = 33.33;
double d = 100;
//sum is exact ...

0votes

0answers

1kviews

### Determine the length of a float in SQL Server

I want to change float columns to decimal columns, so I use max(x) to determine the numbers of digits I need before the decimal, but how can I check the length after the decimal?
For example:
float ...

0votes

0answers

351views

### Convert periodic binary number to decimal

I have a simple question that is confusing me.
Convert the periodic binary number (0.1011)_2 (imagine a bar over the digits after the comma) to a decimal representation.
If it weren't periodic, no ...

0votes

0answers

302views

### MIPS assembly language programming

Hello I am currently learning MIPS assembly language programming. I am trying to print the answer truncated to only one decimal place. For example 127.0 instead of 127.000000 but it is still printing ...

0votes

0answers

154views

### How to always display at least two decimal places?

I need to format the values of a TFloatField in order to always display at least two decimal places.
Example:
1
1.2
1.23
1.234
Output:
1.00
1.20
1.23
1.234
I've tried setting the DisplayFormat ...

0votes

1answer

200views

### Variable return types of DecimalField

I have a model thats something like this:
class Item(models.Model):
value_a = models.DecimalField(default=0.0, decimal_places=4, max_digits=20)
value_b = models.DecimalField(default=0.0, ...

0votes

0answers

582views

### Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...

0votes

2answers

2kviews

### Very large float in python

I'm trying to construct a neural network for the Mnist database. When computing the softmax function I receive an error to the same ends as "you can't store a float that size"
code is as follows:
...

0votes

0answers

45views

### String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...

0votes

0answers

137views

### Limit digits of float values in HTML

I have a reference to a field in Autodesk Infraworks which shows the area number. Field contains float numbers, but they are displayed incorrectly. I.e. instead of 20.8 it shows 20.790000000000000.
...

0votes

1answer

256views

### Inconsistent Decimal arithmetic in Python (rounding errors)

>>> from decimal import Decimal
>>> Decimal('0.3637') * (Decimal('1') / Decimal('0.9323')) == Decimal('0.3637') / Decimal('0.9323')
False
Why?
I thought decimal module solves ...

0votes

1answer

679views

### How to convert decimal(with float point) to binary in Swift 3? (self written code without third-party library and Foundation)

I am looking for a simple way to convert a decimal with floating point to binary with floating point in Swift 3. For example, this code converts decimal to binary without any problems.
func ...

0votes

0answers

84views

### Taking a Floating point number to 30th decimal point in C#

I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002
0.010000000000000000000000000003
0....

0votes

0answers

47views

### what is the result of (54.125) - (184)10

I am practicing for midterm and apprently there's no answer key for it.
However, I practiced and got a result but not sure if this is correct since the solution is really long.
perfrom the following ...

0votes

1answer

697views

### Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...

0votes

0answers

227views

### Accurate javascript decimal number processing with Google Closure Compiler & Library

I'm 6 months into a project using google closure library and compiler. I'm finding I'd like more accurate decimal number handling (handling some simple currency calculations)
one of the common ...

0votes

2answers

342views

### More Decimal Places In Python

Here is my python code:
import math
import decimal as dec
import numpy as np
import matplotlib.pyplot as plt
c = 3e8
wave = np.array([253.6e-9,283e-9,303.9e-9,330.2e-9,366.3e-9,435.8e-9])
freq =...

0votes

1answer

62views

### Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question.
I'm doing a formula in TSQL like this:
@Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1))
+ ((@...

0votes

1answer

1kviews

### Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand.
The significand is normalized so that the first digit to the right of the radix point must be a 1.
Exponent is signed 2's ...

0votes

2answers

146views

### Understanding machine precision based on bit representation

I am trying to understand how this is calculated:
A computer program that represents numbers as follows: 1 bit for the overall
sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...

0votes

2answers

813views

### PHP: string with decimals strips decimals when converting to float

With the function floatval() I try to convert a string with decimals to a float, but it doesn´t give me the desired result. Here´s the code:
$price_calc = str_replace(',','.',$price);
$...