All Questions
Tagged with floating-point decimal
57
questions with no upvoted or accepted answers
9votes
1answer
596views
Are .NET Decimal type computations deterministic?
I have two questions regarding .NET's decimal data type determinism:
Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
- 460
3votes
0answers
2kviews
How to format float to 4 decimal places within json dumps?
Have difficulty in converting pf_stats output to have 4 decimal places:
import json
import numpy as np
def run_simulation(H, P, B, C, mu, sigma, T, L):
x = normal(mu, sigma, (L, T))
pf_all ...
- 31
2votes
0answers
655views
Handle money and currencies in javascript multiplying and dividing by 100
I'm writing an app for myself. I need to handle money and currencies.
I read this and I want to try the second approach:
You can multiply floats into integers before you calculate, then divide them ...
- 1,325
2votes
0answers
3kviews
Python - How can I change 1E-8 to 0.00000001 without changing type?
Python's decimal.Decimal automatically converts the following number (7 zeros and a one, after the decimal point) into scientific notation:
d = decimal.Decimal('0.00000001')
import simplejson; ...
- 8,926
2votes
0answers
3kviews
Python float types vs Decimal
I am an intern at TCD, Physics.
I wrote a code to perform some data analysis on random particle packings. The code is written in Python.
The code reads in columns of data from a .txt file, provided ...
- 321
2votes
1answer
228views
How to convert string value to Number for exact calculation ? Javascript
Below is the code :
var a ="10.9549999999999141";
console.log(a);
console.log(typeof a);
console.log(Number(a));
Now the above code results as below :
"10.9549999999999141"
"string"
10....
- 1,952
2votes
5answers
2kviews
Join two strings into a single decimal value
I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation?
decimal value; //like 500.45
string txt1, txt2;
txt1 = "500";
txt2 = "45";
value = Convert....
- 1,348
1vote
1answer
38views
Getting different values for Decimal LocalContext Type
I have written following code to determine type of decimal localcontext:
print('decimal.localcontext() type: ', type(decimal.localcontext()))
Output:
decimal.localcontext() type: <class 'decimal....
- 11.2k
1vote
1answer
40views
How to get first 4 digits after the 0s in a decimal but keeping 0s • Python
I need the first 4 digits after the front 0s in a decimal while also keeping the 0s in the output without getting scientific notation.
Would like to take
0.0000000000000000000000634546534
and get
0....
- 11
1vote
1answer
64views
Confused by loss of float precision in financial application with python Decimal package
I'm having a problem in financial application when calculate 70000.0*5.65500*18.0/36000.0 and compare the result with another number.
The accurate result is 197.925
When use Decimal, the results ...
- 11
1vote
0answers
100views
How to use Decimal64 DPD with gcc
I have been trying to figure out how to integrate Decimal Floating Types with GCC 10.2.0.
I built gcc with --enable-decimal-float=dpd and --with-backend=libdecnumber flags, and specified DFP_ENABLE=1 ...
- 312
1vote
0answers
69views
9 Bit Floating Point to Hex
I've been running into an issue where I'm trying to convert a 9-bit floating point number to hex, where the floating point scheme is 1 sign-bit, followed by 4 bit exponent, and then a 4-bit mantissa. ...
- 11
1vote
1answer
35views
Decimal-module is not working on binary level? How?
I'm a beginner when it comes to what's going on in the background when I start the program. Right now my focus is on the difference between "decimal" and "float" in "Python&...
- 27
1vote
0answers
39views
How to handle precision Decimals in Python and overcome binary representation limitations in computers?
How to obtain precision without rounding?
For instance:
import math
def truncate(f, n):
return math.floor(f * 10 ** n) / 10 ** n
r = lambda f,p: f - f % p
size = 3245.249999999999853146309903
y ...
- 1,231
1vote
1answer
180views
How to get the number of decimal places of a float value?
I want to express the final product rounded down according to scientific notation.
So if 80.55 and 879.5689 give 70849.281250 the final print functions outputs 70849.28.
Is there a way to get the ...
- 11
1vote
0answers
57views
Why does `decimal.Decimal` still give rounding errors?
I am using decimal.Decimal instead of float explicitly to avoid rounding errors, but am still getting them:
>>> from decimal import Decimal
>>> Decimal('9') * (Decimal('10') / ...
- 1,062
1vote
0answers
73views
Highly precise division, multiplication, and exponentiation of large complex numbers
I am working on a project that requires highly precise division of large numbers which will sometimes be complex numbers. I need to do this in python, preferably python 3.7, but everything I have ...
- 192
1vote
1answer
318views
Python:computing the number of digits after decimal gives wrong answer
I have a floating point field in one of my form, consider it as field_x. Based on that field_x i have some computation.
After all if field_x have n digits after decimal result also should have n ...
- 793
1vote
1answer
745views
Serialize a decimal with simplejson
I have been trying to serialize a decimal value with a string key but so far i only managed to make a successful 1 decimal digit (like '16.1'). I tried both json and simplejson libraries, subclassing ...
- 77
1vote
2answers
148views
Reversible decimal to double conversion
I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...
- 3,621
1vote
1answer
131views
How is this number expressed as a multiple of 2^-22*(1/10)? How is that derived?
I was refreshing on binary and floats and I run into the following example:
0.1 is represented as: 0.0001100110011001100110011[0011] with the part in the brackets repeating.
If we round up this ...
- 51.2k
1vote
1answer
43views
python hack converting floats to decimals
I've written a large program, with dependencies on libraries written in my lab. I'm getting wrong (and somewhat random) results, which are caused by floating-point errors.
I would like to do some ...
1vote
1answer
261views
python float addition results with unexpected decimal places
I have the following code to detect stock trades by counting orders:
import json
orders = '[{"side": "SELL", "executedQty": "0.046"}, {"side": "SELL", "executedQty": "0.041"}, {"side": "SELL", "...
- 109
0votes
1answer
52views
Accelerometer data from hex to float
I am trying to get accelerometer data off a poorly documented BLE beacon. Here is what I receive (as NSDictionary):
{
"x-Data" = 0020;
"y-Data" = 0000;
"z-Data&...
- 311
0votes
1answer
410views
Logical operation with float number (with C & python)
I'm a Python newbie and currently comparing python with C language using logical operations.
My question is
(I could solve Q1 (logical operation in C), thank you for your comments!)
Q1: Logical ...
0votes
1answer
47views
how to restrict float number to 1 digit after decimal point
I have a float number 242.1356783 and -87.10234512,
I need to restrict it it to 242.1
I tried below on cortex m4 microcontroller
1.data[i] = (int)(data[i] *10);
data[i] = data[i]/10;
2.data[i]= ceilf(...
- 1
0votes
0answers
63views
C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database
I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...
- 201
0votes
2answers
77views
The decimal.Decimal() function is producing an unrounded result
I am new to Python and I am having trouble with the Decimal function in the decimal module in Python. I understand that due to the way that Python stores float values in memory, in order to perform ...
0votes
0answers
48views
String to float conversion not working php
I've a string value of the type "x.yy" which I want to convert to double/float.
I've tried floatval(), doubleval() and float and double type casting as well but it only gives me "X"...
- 115
0votes
0answers
83views
How to deal with decimal operation errors in dart?
I am trying to fix the decimal error generated in my program.
For example I have the code:-
void main() {
double a = 300;
double b = 150;
double c = 33.33;
double d = 100;
//sum is exact ...
- 575
0votes
0answers
1kviews
Determine the length of a float in SQL Server
I want to change float columns to decimal columns, so I use max(x) to determine the numbers of digits I need before the decimal, but how can I check the length after the decimal?
For example:
float ...
0votes
0answers
351views
Convert periodic binary number to decimal
I have a simple question that is confusing me.
Convert the periodic binary number (0.1011)_2 (imagine a bar over the digits after the comma) to a decimal representation.
If it weren't periodic, no ...
- 1
0votes
0answers
302views
MIPS assembly language programming
Hello I am currently learning MIPS assembly language programming. I am trying to print the answer truncated to only one decimal place. For example 127.0 instead of 127.000000 but it is still printing ...
- 131
0votes
0answers
154views
How to always display at least two decimal places?
I need to format the values of a TFloatField in order to always display at least two decimal places.
Example:
1
1.2
1.23
1.234
Output:
1.00
1.20
1.23
1.234
I've tried setting the DisplayFormat ...
- 7,128
0votes
1answer
200views
Variable return types of DecimalField
I have a model thats something like this:
class Item(models.Model):
value_a = models.DecimalField(default=0.0, decimal_places=4, max_digits=20)
value_b = models.DecimalField(default=0.0, ...
0votes
0answers
582views
Python: mysql double and float
In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...
- 109
0votes
2answers
2kviews
Very large float in python
I'm trying to construct a neural network for the Mnist database. When computing the softmax function I receive an error to the same ends as "you can't store a float that size"
code is as follows:
...
- 327
0votes
0answers
45views
String Formatting with Decimal Number Python
I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
- 2,453
0votes
0answers
137views
Limit digits of float values in HTML
I have a reference to a field in Autodesk Infraworks which shows the area number. Field contains float numbers, but they are displayed incorrectly. I.e. instead of 20.8 it shows 20.790000000000000.
...
- 389
0votes
1answer
256views
Inconsistent Decimal arithmetic in Python (rounding errors)
>>> from decimal import Decimal
>>> Decimal('0.3637') * (Decimal('1') / Decimal('0.9323')) == Decimal('0.3637') / Decimal('0.9323')
False
Why?
I thought decimal module solves ...
0votes
1answer
679views
How to convert decimal(with float point) to binary in Swift 3? (self written code without third-party library and Foundation)
I am looking for a simple way to convert a decimal with floating point to binary with floating point in Swift 3. For example, this code converts decimal to binary without any problems.
func ...
- 77
0votes
0answers
84views
Taking a Floating point number to 30th decimal point in C#
I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002
0.010000000000000000000000000003
0....
0votes
0answers
47views
what is the result of (54.125) - (184)10
I am practicing for midterm and apprently there's no answer key for it.
However, I practiced and got a result but not sure if this is correct since the solution is really long.
perfrom the following ...
user5638088
0votes
1answer
697views
Rounding Off function to given decimal without using round function
I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
- 1
0votes
0answers
227views
Accurate javascript decimal number processing with Google Closure Compiler & Library
I'm 6 months into a project using google closure library and compiler. I'm finding I'd like more accurate decimal number handling (handling some simple currency calculations)
one of the common ...
0votes
2answers
342views
More Decimal Places In Python
Here is my python code:
import math
import decimal as dec
import numpy as np
import matplotlib.pyplot as plt
c = 3e8
wave = np.array([253.6e-9,283e-9,303.9e-9,330.2e-9,366.3e-9,435.8e-9])
freq =...
- 635
0votes
1answer
62views
Avoid rounding to 0 when a result is very little
I've searched for this but i didn't find anything, i hope this is not a doubled question.
I'm doing a formula in TSQL like this:
@Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1))
+ ((@...
- 45
0votes
1answer
1kviews
Smallest positive number with the following format
12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand.
The significand is normalized so that the first digit to the right of the radix point must be a 1.
Exponent is signed 2's ...
- 1
0votes
2answers
146views
Understanding machine precision based on bit representation
I am trying to understand how this is calculated:
A computer program that represents numbers as follows: 1 bit for the overall
sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...
0votes
2answers
813views
PHP: string with decimals strips decimals when converting to float
With the function floatval() I try to convert a string with decimals to a float, but it doesn´t give me the desired result. Here´s the code:
$price_calc = str_replace(',','.',$price);
$...
