All Questions

Tagged with
57 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
596views

Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism: Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
• 460
2kviews

How to format float to 4 decimal places within json dumps?

Have difficulty in converting pf_stats output to have 4 decimal places: import json import numpy as np def run_simulation(H, P, B, C, mu, sigma, T, L): x = normal(mu, sigma, (L, T)) pf_all ...
• 31
655views

Handle money and currencies in javascript multiplying and dividing by 100

I'm writing an app for myself. I need to handle money and currencies. I read this and I want to try the second approach: You can multiply floats into integers before you calculate, then divide them ...
• 1,325
3kviews

Python - How can I change 1E-8 to 0.00000001 without changing type?

Python's decimal.Decimal automatically converts the following number (7 zeros and a one, after the decimal point) into scientific notation: d = decimal.Decimal('0.00000001') import simplejson; ...
• 8,926
3kviews

Python float types vs Decimal

I am an intern at TCD, Physics. I wrote a code to perform some data analysis on random particle packings. The code is written in Python. The code reads in columns of data from a .txt file, provided ...
228views

How to convert string value to Number for exact calculation ? Javascript

Below is the code : var a ="10.9549999999999141"; console.log(a); console.log(typeof a); console.log(Number(a)); Now the above code results as below : "10.9549999999999141" "string" 10....
• 1,952
2kviews

Join two strings into a single decimal value

I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation? decimal value; //like 500.45 string txt1, txt2; txt1 = "500"; txt2 = "45"; value = Convert....
• 1,348
1vote
38views

Getting different values for Decimal LocalContext Type

I have written following code to determine type of decimal localcontext: print('decimal.localcontext() type: ', type(decimal.localcontext())) Output: decimal.localcontext() type: <class 'decimal....
• 11.2k
1vote
40views

How to get first 4 digits after the 0s in a decimal but keeping 0s • Python

I need the first 4 digits after the front 0s in a decimal while also keeping the 0s in the output without getting scientific notation. Would like to take 0.0000000000000000000000634546534 and get 0....
• 11
1vote
64views

Confused by loss of float precision in financial application with python Decimal package

I'm having a problem in financial application when calculate 70000.0*5.65500*18.0/36000.0 and compare the result with another number. The accurate result is 197.925 When use Decimal, the results ...
• 11
1vote
100views

How to use Decimal64 DPD with gcc

I have been trying to figure out how to integrate Decimal Floating Types with GCC 10.2.0. I built gcc with --enable-decimal-float=dpd and --with-backend=libdecnumber flags, and specified DFP_ENABLE=1 ...
• 312
1vote
69views

9 Bit Floating Point to Hex

I've been running into an issue where I'm trying to convert a 9-bit floating point number to hex, where the floating point scheme is 1 sign-bit, followed by 4 bit exponent, and then a 4-bit mantissa. ...
1vote
35views

Decimal-module is not working on binary level? How?

I'm a beginner when it comes to what's going on in the background when I start the program. Right now my focus is on the difference between "decimal" and "float" in "Python&...
1vote
39views

How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding? For instance: import math def truncate(f, n): return math.floor(f * 10 ** n) / 10 ** n r = lambda f,p: f - f % p size = 3245.249999999999853146309903 y ...
• 1,231
1vote
180views

How to get the number of decimal places of a float value?

I want to express the final product rounded down according to scientific notation. So if 80.55 and 879.5689 give 70849.281250 the final print functions outputs 70849.28. Is there a way to get the ...
1vote
57views

Why does `decimal.Decimal` still give rounding errors?

I am using decimal.Decimal instead of float explicitly to avoid rounding errors, but am still getting them: >>> from decimal import Decimal >>> Decimal('9') * (Decimal('10') / ...
• 1,062
1vote
73views

Highly precise division, multiplication, and exponentiation of large complex numbers

I am working on a project that requires highly precise division of large numbers which will sometimes be complex numbers. I need to do this in python, preferably python 3.7, but everything I have ...
• 192
1vote
318views

Python:computing the number of digits after decimal gives wrong answer

I have a floating point field in one of my form, consider it as field_x. Based on that field_x i have some computation. After all if field_x have n digits after decimal result also should have n ...
• 793
1vote
745views

Serialize a decimal with simplejson

I have been trying to serialize a decimal value with a string key but so far i only managed to make a successful 1 decimal digit (like '16.1'). I tried both json and simplejson libraries, subclassing ...
• 77
1vote
148views

Reversible decimal to double conversion

I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...
• 3,621
1vote
131views

How is this number expressed as a multiple of 2^-22*(1/10)? How is that derived?

I was refreshing on binary and floats and I run into the following example: 0.1 is represented as: 0.0001100110011001100110011[0011] with the part in the brackets repeating. If we round up this ...
• 51.2k
1vote
43views

python hack converting floats to decimals

I've written a large program, with dependencies on libraries written in my lab. I'm getting wrong (and somewhat random) results, which are caused by floating-point errors. I would like to do some ...
1vote
261views

python float addition results with unexpected decimal places

I have the following code to detect stock trades by counting orders: import json orders = '[{"side": "SELL", "executedQty": "0.046"}, {"side": "SELL", "executedQty": "0.041"}, {"side": "SELL", "...
• 109
52views

Accelerometer data from hex to float

I am trying to get accelerometer data off a poorly documented BLE beacon. Here is what I receive (as NSDictionary): { "x-Data" = 0020; "y-Data" = 0000; "z-Data&...
• 311
410views

Logical operation with float number (with C & python)

I'm a Python newbie and currently comparing python with C language using logical operations. My question is (I could solve Q1 (logical operation in C), thank you for your comments!) Q1: Logical ...
47views

how to restrict float number to 1 digit after decimal point

I have a float number 242.1356783 and -87.10234512, I need to restrict it it to 242.1 I tried below on cortex m4 microcontroller 1.data[i] = (int)(data[i] *10); data[i] = data[i]/10; 2.data[i]= ceilf(...
• 1
63views

C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database

I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...
• 201
77views

The decimal.Decimal() function is producing an unrounded result

I am new to Python and I am having trouble with the Decimal function in the decimal module in Python. I understand that due to the way that Python stores float values in memory, in order to perform ...
48views

String to float conversion not working php

I've a string value of the type "x.yy" which I want to convert to double/float. I've tried floatval(), doubleval() and float and double type casting as well but it only gives me "X"...
• 115
83views

How to deal with decimal operation errors in dart?

I am trying to fix the decimal error generated in my program. For example I have the code:- void main() { double a = 300; double b = 150; double c = 33.33; double d = 100; //sum is exact ...
• 575
1kviews

Determine the length of a float in SQL Server

I want to change float columns to decimal columns, so I use max(x) to determine the numbers of digits I need before the decimal, but how can I check the length after the decimal? For example: float ...
351views

Convert periodic binary number to decimal

I have a simple question that is confusing me. Convert the periodic binary number (0.1011)_2 (imagine a bar over the digits after the comma) to a decimal representation. If it weren't periodic, no ...
• 1
302views

MIPS assembly language programming

Hello I am currently learning MIPS assembly language programming. I am trying to print the answer truncated to only one decimal place. For example 127.0 instead of 127.000000 but it is still printing ...
154views

How to always display at least two decimal places?

I need to format the values of a TFloatField in order to always display at least two decimal places. Example: 1 1.2 1.23 1.234 Output: 1.00 1.20 1.23 1.234 I've tried setting the DisplayFormat ...
• 7,128
200views

Variable return types of DecimalField

I have a model thats something like this: class Item(models.Model): value_a = models.DecimalField(default=0.0, decimal_places=4, max_digits=20) value_b = models.DecimalField(default=0.0, ...
582views

Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...
• 109
2kviews

Very large float in python

I'm trying to construct a neural network for the Mnist database. When computing the softmax function I receive an error to the same ends as "you can't store a float that size" code is as follows: ...
45views

String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
• 2,453
137views

Limit digits of float values in HTML

I have a reference to a field in Autodesk Infraworks which shows the area number. Field contains float numbers, but they are displayed incorrectly. I.e. instead of 20.8 it shows 20.790000000000000. ...
256views

Inconsistent Decimal arithmetic in Python (rounding errors)

>>> from decimal import Decimal >>> Decimal('0.3637') * (Decimal('1') / Decimal('0.9323')) == Decimal('0.3637') / Decimal('0.9323') False Why? I thought decimal module solves ...
679views

How to convert decimal(with float point) to binary in Swift 3? (self written code without third-party library and Foundation)

I am looking for a simple way to convert a decimal with floating point to binary with floating point in Swift 3. For example, this code converts decimal to binary without any problems. func ...
• 77
84views

Taking a Floating point number to 30th decimal point in C#

I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002 0.010000000000000000000000000003 0....
47views

what is the result of (54.125) - (184)10

I am practicing for midterm and apprently there's no answer key for it. However, I practiced and got a result but not sure if this is correct since the solution is really long. perfrom the following ...
697views

Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
227views

Accurate javascript decimal number processing with Google Closure Compiler & Library

I'm 6 months into a project using google closure library and compiler. I'm finding I'd like more accurate decimal number handling (handling some simple currency calculations) one of the common ...
342views

More Decimal Places In Python

Here is my python code: import math import decimal as dec import numpy as np import matplotlib.pyplot as plt c = 3e8 wave = np.array([253.6e-9,283e-9,303.9e-9,330.2e-9,366.3e-9,435.8e-9]) freq =...
• 635
62views

Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question. I'm doing a formula in TSQL like this: @Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1)) + ((@...
• 45
1kviews

Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand. The significand is normalized so that the first digit to the right of the radix point must be a 1. Exponent is signed 2's ...
• 1
146views

Understanding machine precision based on bit representation

I am trying to understand how this is calculated: A computer program that represents numbers as follows: 1 bit for the overall sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...