# All Questions

Tagged with
57 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
596views

### Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism: Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
2kviews

### How to format float to 4 decimal places within json dumps?

Have difficulty in converting pf_stats output to have 4 decimal places: import json import numpy as np def run_simulation(H, P, B, C, mu, sigma, T, L): x = normal(mu, sigma, (L, T)) pf_all ...
655views

### Handle money and currencies in javascript multiplying and dividing by 100

I'm writing an app for myself. I need to handle money and currencies. I read this and I want to try the second approach: You can multiply floats into integers before you calculate, then divide them ...
3kviews

### Python - How can I change 1E-8 to 0.00000001 without changing type?

Python's decimal.Decimal automatically converts the following number (7 zeros and a one, after the decimal point) into scientific notation: d = decimal.Decimal('0.00000001') import simplejson; ...
3kviews

### Python float types vs Decimal

I am an intern at TCD, Physics. I wrote a code to perform some data analysis on random particle packings. The code is written in Python. The code reads in columns of data from a .txt file, provided ...
228views

### How to convert string value to Number for exact calculation ? Javascript

Below is the code : var a ="10.9549999999999141"; console.log(a); console.log(typeof a); console.log(Number(a)); Now the above code results as below : "10.9549999999999141" "string" 10....
2kviews

### Join two strings into a single decimal value

I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation? decimal value; //like 500.45 string txt1, txt2; txt1 = "500"; txt2 = "45"; value = Convert....
1vote
38views

### Getting different values for Decimal LocalContext Type

I have written following code to determine type of decimal localcontext: print('decimal.localcontext() type: ', type(decimal.localcontext())) Output: decimal.localcontext() type: <class 'decimal....
1vote
40views

### How to get first 4 digits after the 0s in a decimal but keeping 0s • Python

I need the first 4 digits after the front 0s in a decimal while also keeping the 0s in the output without getting scientific notation. Would like to take 0.0000000000000000000000634546534 and get 0....
1vote
64views

### Confused by loss of float precision in financial application with python Decimal package

I'm having a problem in financial application when calculate 70000.0*5.65500*18.0/36000.0 and compare the result with another number. The accurate result is 197.925 When use Decimal, the results ...
1vote
100views

### How to use Decimal64 DPD with gcc

I have been trying to figure out how to integrate Decimal Floating Types with GCC 10.2.0. I built gcc with --enable-decimal-float=dpd and --with-backend=libdecnumber flags, and specified DFP_ENABLE=1 ...
1vote
69views

### 9 Bit Floating Point to Hex

I've been running into an issue where I'm trying to convert a 9-bit floating point number to hex, where the floating point scheme is 1 sign-bit, followed by 4 bit exponent, and then a 4-bit mantissa. ...
1vote
35views

### Decimal-module is not working on binary level? How?

I'm a beginner when it comes to what's going on in the background when I start the program. Right now my focus is on the difference between "decimal" and "float" in "Python&...
1vote
39views

### How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding? For instance: import math def truncate(f, n): return math.floor(f * 10 ** n) / 10 ** n r = lambda f,p: f - f % p size = 3245.249999999999853146309903 y ...
1vote
180views

### How to get the number of decimal places of a float value?

I want to express the final product rounded down according to scientific notation. So if 80.55 and 879.5689 give 70849.281250 the final print functions outputs 70849.28. Is there a way to get the ...
1vote
57views

### Why does `decimal.Decimal` still give rounding errors?

I am using decimal.Decimal instead of float explicitly to avoid rounding errors, but am still getting them: >>> from decimal import Decimal >>> Decimal('9') * (Decimal('10') / ...
1vote
73views

### Highly precise division, multiplication, and exponentiation of large complex numbers

I am working on a project that requires highly precise division of large numbers which will sometimes be complex numbers. I need to do this in python, preferably python 3.7, but everything I have ...
1vote
318views

### Python:computing the number of digits after decimal gives wrong answer

I have a floating point field in one of my form, consider it as field_x. Based on that field_x i have some computation. After all if field_x have n digits after decimal result also should have n ...
1vote
745views

### Serialize a decimal with simplejson

I have been trying to serialize a decimal value with a string key but so far i only managed to make a successful 1 decimal digit (like '16.1'). I tried both json and simplejson libraries, subclassing ...
1vote
148views

### Reversible decimal to double conversion

I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...
1vote
131views

### How is this number expressed as a multiple of 2^-22*(1/10)? How is that derived?

I was refreshing on binary and floats and I run into the following example: 0.1 is represented as: 0.0001100110011001100110011 with the part in the brackets repeating. If we round up this ...
1vote
43views

### python hack converting floats to decimals

I've written a large program, with dependencies on libraries written in my lab. I'm getting wrong (and somewhat random) results, which are caused by floating-point errors. I would like to do some ...
1vote
261views

### python float addition results with unexpected decimal places

I have the following code to detect stock trades by counting orders: import json orders = '[{"side": "SELL", "executedQty": "0.046"}, {"side": "SELL", "executedQty": "0.041"}, {"side": "SELL", "...
52views

### Accelerometer data from hex to float

I am trying to get accelerometer data off a poorly documented BLE beacon. Here is what I receive (as NSDictionary): { "x-Data" = 0020; "y-Data" = 0000; "z-Data&...
410views

### Logical operation with float number (with C & python)

I'm a Python newbie and currently comparing python with C language using logical operations. My question is (I could solve Q1 (logical operation in C), thank you for your comments!) Q1: Logical ...
47views

### how to restrict float number to 1 digit after decimal point

I have a float number 242.1356783 and -87.10234512, I need to restrict it it to 242.1 I tried below on cortex m4 microcontroller 1.data[i] = (int)(data[i] *10); data[i] = data[i]/10; 2.data[i]= ceilf(...
63views

### C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database

I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...
77views

### The decimal.Decimal() function is producing an unrounded result

I am new to Python and I am having trouble with the Decimal function in the decimal module in Python. I understand that due to the way that Python stores float values in memory, in order to perform ...
48views

### String to float conversion not working php

I've a string value of the type "x.yy" which I want to convert to double/float. I've tried floatval(), doubleval() and float and double type casting as well but it only gives me "X"...
83views

### How to deal with decimal operation errors in dart?

I am trying to fix the decimal error generated in my program. For example I have the code:- void main() { double a = 300; double b = 150; double c = 33.33; double d = 100; //sum is exact ...
1kviews

### Determine the length of a float in SQL Server

I want to change float columns to decimal columns, so I use max(x) to determine the numbers of digits I need before the decimal, but how can I check the length after the decimal? For example: float ...
351views

### Convert periodic binary number to decimal

I have a simple question that is confusing me. Convert the periodic binary number (0.1011)_2 (imagine a bar over the digits after the comma) to a decimal representation. If it weren't periodic, no ...
302views

### MIPS assembly language programming

Hello I am currently learning MIPS assembly language programming. I am trying to print the answer truncated to only one decimal place. For example 127.0 instead of 127.000000 but it is still printing ...
154views

### How to always display at least two decimal places?

I need to format the values of a TFloatField in order to always display at least two decimal places. Example: 1 1.2 1.23 1.234 Output: 1.00 1.20 1.23 1.234 I've tried setting the DisplayFormat ...
200views

### Variable return types of DecimalField

I have a model thats something like this: class Item(models.Model): value_a = models.DecimalField(default=0.0, decimal_places=4, max_digits=20) value_b = models.DecimalField(default=0.0, ...
582views

### Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...
2kviews

### Very large float in python

I'm trying to construct a neural network for the Mnist database. When computing the softmax function I receive an error to the same ends as "you can't store a float that size" code is as follows: ...
45views

### String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
137views

### Limit digits of float values in HTML

I have a reference to a field in Autodesk Infraworks which shows the area number. Field contains float numbers, but they are displayed incorrectly. I.e. instead of 20.8 it shows 20.790000000000000. ...
256views

### Inconsistent Decimal arithmetic in Python (rounding errors)

>>> from decimal import Decimal >>> Decimal('0.3637') * (Decimal('1') / Decimal('0.9323')) == Decimal('0.3637') / Decimal('0.9323') False Why? I thought decimal module solves ...
679views

### How to convert decimal(with float point) to binary in Swift 3? (self written code without third-party library and Foundation)

I am looking for a simple way to convert a decimal with floating point to binary with floating point in Swift 3. For example, this code converts decimal to binary without any problems. func ...
84views

### Taking a Floating point number to 30th decimal point in C#

I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002 0.010000000000000000000000000003 0....
47views

### what is the result of (54.125) - (184)10

I am practicing for midterm and apprently there's no answer key for it. However, I practiced and got a result but not sure if this is correct since the solution is really long. perfrom the following ... 697views

### Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
227views

### Accurate javascript decimal number processing with Google Closure Compiler & Library

I'm 6 months into a project using google closure library and compiler. I'm finding I'd like more accurate decimal number handling (handling some simple currency calculations) one of the common ...
342views

### More Decimal Places In Python

Here is my python code: import math import decimal as dec import numpy as np import matplotlib.pyplot as plt c = 3e8 wave = np.array([253.6e-9,283e-9,303.9e-9,330.2e-9,366.3e-9,435.8e-9]) freq =...
62views

### Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question. I'm doing a formula in TSQL like this: @Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1)) + ((@...
1kviews

### Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand. The significand is normalized so that the first digit to the right of the radix point must be a 1. Exponent is signed 2's ...
146views

### Understanding machine precision based on bit representation

I am trying to understand how this is calculated: A computer program that represents numbers as follows: 1 bit for the overall sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...