All Questions

Tagged with
57 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9votes
1answer
596views

Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism: Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
user avatar
  • 460
3votes
0answers
2kviews

How to format float to 4 decimal places within json dumps?

Have difficulty in converting pf_stats output to have 4 decimal places: import json import numpy as np def run_simulation(H, P, B, C, mu, sigma, T, L): x = normal(mu, sigma, (L, T)) pf_all ...
user avatar
  • 31
2votes
0answers
655views

Handle money and currencies in javascript multiplying and dividing by 100

I'm writing an app for myself. I need to handle money and currencies. I read this and I want to try the second approach: You can multiply floats into integers before you calculate, then divide them ...
user avatar
  • 1,325
2votes
0answers
3kviews

Python - How can I change 1E-8 to 0.00000001 without changing type?

Python's decimal.Decimal automatically converts the following number (7 zeros and a one, after the decimal point) into scientific notation: d = decimal.Decimal('0.00000001') import simplejson; ...
user avatar
  • 8,926
2votes
0answers
3kviews

Python float types vs Decimal

I am an intern at TCD, Physics. I wrote a code to perform some data analysis on random particle packings. The code is written in Python. The code reads in columns of data from a .txt file, provided ...
user avatar
2votes
1answer
228views

How to convert string value to Number for exact calculation ? Javascript

Below is the code : var a ="10.9549999999999141"; console.log(a); console.log(typeof a); console.log(Number(a)); Now the above code results as below : "10.9549999999999141" "string" 10....
user avatar
2votes
5answers
2kviews

Join two strings into a single decimal value

I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation? decimal value; //like 500.45 string txt1, txt2; txt1 = "500"; txt2 = "45"; value = Convert....
user avatar
  • 1,348
1vote
1answer
38views

Getting different values for Decimal LocalContext Type

I have written following code to determine type of decimal localcontext: print('decimal.localcontext() type: ', type(decimal.localcontext())) Output: decimal.localcontext() type: <class 'decimal....
user avatar
  • 11.2k
1vote
1answer
40views

How to get first 4 digits after the 0s in a decimal but keeping 0s • Python

I need the first 4 digits after the front 0s in a decimal while also keeping the 0s in the output without getting scientific notation. Would like to take 0.0000000000000000000000634546534 and get 0....
user avatar
  • 11
1vote
1answer
64views

Confused by loss of float precision in financial application with python Decimal package

I'm having a problem in financial application when calculate 70000.0*5.65500*18.0/36000.0 and compare the result with another number. The accurate result is 197.925 When use Decimal, the results ...
user avatar
  • 11
1vote
0answers
100views

How to use Decimal64 DPD with gcc

I have been trying to figure out how to integrate Decimal Floating Types with GCC 10.2.0. I built gcc with --enable-decimal-float=dpd and --with-backend=libdecnumber flags, and specified DFP_ENABLE=1 ...
user avatar
1vote
0answers
69views

9 Bit Floating Point to Hex

I've been running into an issue where I'm trying to convert a 9-bit floating point number to hex, where the floating point scheme is 1 sign-bit, followed by 4 bit exponent, and then a 4-bit mantissa. ...
user avatar
1vote
1answer
35views

Decimal-module is not working on binary level? How?

I'm a beginner when it comes to what's going on in the background when I start the program. Right now my focus is on the difference between "decimal" and "float" in "Python&...
user avatar
1vote
0answers
39views

How to handle precision Decimals in Python and overcome binary representation limitations in computers?

How to obtain precision without rounding? For instance: import math def truncate(f, n): return math.floor(f * 10 ** n) / 10 ** n r = lambda f,p: f - f % p size = 3245.249999999999853146309903 y ...
user avatar
1vote
1answer
180views

How to get the number of decimal places of a float value?

I want to express the final product rounded down according to scientific notation. So if 80.55 and 879.5689 give 70849.281250 the final print functions outputs 70849.28. Is there a way to get the ...
user avatar
1vote
0answers
57views

Why does `decimal.Decimal` still give rounding errors?

I am using decimal.Decimal instead of float explicitly to avoid rounding errors, but am still getting them: >>> from decimal import Decimal >>> Decimal('9') * (Decimal('10') / ...
user avatar
  • 1,062
1vote
0answers
73views

Highly precise division, multiplication, and exponentiation of large complex numbers

I am working on a project that requires highly precise division of large numbers which will sometimes be complex numbers. I need to do this in python, preferably python 3.7, but everything I have ...
user avatar
  • 192
1vote
1answer
318views

Python:computing the number of digits after decimal gives wrong answer

I have a floating point field in one of my form, consider it as field_x. Based on that field_x i have some computation. After all if field_x have n digits after decimal result also should have n ...
user avatar
  • 793
1vote
1answer
745views

Serialize a decimal with simplejson

I have been trying to serialize a decimal value with a string key but so far i only managed to make a successful 1 decimal digit (like '16.1'). I tried both json and simplejson libraries, subclassing ...
user avatar
1vote
2answers
148views

Reversible decimal to double conversion

I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...
user avatar
1vote
1answer
131views

How is this number expressed as a multiple of 2^-22*(1/10)? How is that derived?

I was refreshing on binary and floats and I run into the following example: 0.1 is represented as: 0.0001100110011001100110011[0011] with the part in the brackets repeating. If we round up this ...
user avatar
  • 51.2k
1vote
1answer
43views

python hack converting floats to decimals

I've written a large program, with dependencies on libraries written in my lab. I'm getting wrong (and somewhat random) results, which are caused by floating-point errors. I would like to do some ...
user avatar
1vote
1answer
261views

python float addition results with unexpected decimal places

I have the following code to detect stock trades by counting orders: import json orders = '[{"side": "SELL", "executedQty": "0.046"}, {"side": "SELL", "executedQty": "0.041"}, {"side": "SELL", "...
user avatar
  • 109
0votes
1answer
52views

Accelerometer data from hex to float

I am trying to get accelerometer data off a poorly documented BLE beacon. Here is what I receive (as NSDictionary): { "x-Data" = 0020; "y-Data" = 0000; "z-Data&...
user avatar
  • 311
0votes
1answer
410views

Logical operation with float number (with C & python)

I'm a Python newbie and currently comparing python with C language using logical operations. My question is (I could solve Q1 (logical operation in C), thank you for your comments!) Q1: Logical ...
user avatar
0votes
1answer
47views

how to restrict float number to 1 digit after decimal point

I have a float number 242.1356783 and -87.10234512, I need to restrict it it to 242.1 I tried below on cortex m4 microcontroller 1.data[i] = (int)(data[i] *10); data[i] = data[i]/10; 2.data[i]= ceilf(...
user avatar
  • 1
0votes
0answers
63views

C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database

I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...
user avatar
  • 201
0votes
2answers
77views

The decimal.Decimal() function is producing an unrounded result

I am new to Python and I am having trouble with the Decimal function in the decimal module in Python. I understand that due to the way that Python stores float values in memory, in order to perform ...
user avatar
0votes
0answers
48views

String to float conversion not working php

I've a string value of the type "x.yy" which I want to convert to double/float. I've tried floatval(), doubleval() and float and double type casting as well but it only gives me "X"...
user avatar
0votes
0answers
83views

How to deal with decimal operation errors in dart?

I am trying to fix the decimal error generated in my program. For example I have the code:- void main() { double a = 300; double b = 150; double c = 33.33; double d = 100; //sum is exact ...
user avatar
0votes
0answers
1kviews

Determine the length of a float in SQL Server

I want to change float columns to decimal columns, so I use max(x) to determine the numbers of digits I need before the decimal, but how can I check the length after the decimal? For example: float ...
user avatar
0votes
0answers
351views

Convert periodic binary number to decimal

I have a simple question that is confusing me. Convert the periodic binary number (0.1011)_2 (imagine a bar over the digits after the comma) to a decimal representation. If it weren't periodic, no ...
user avatar
  • 1
0votes
0answers
302views

MIPS assembly language programming

Hello I am currently learning MIPS assembly language programming. I am trying to print the answer truncated to only one decimal place. For example 127.0 instead of 127.000000 but it is still printing ...
user avatar
0votes
0answers
154views

How to always display at least two decimal places?

I need to format the values of a TFloatField in order to always display at least two decimal places. Example: 1 1.2 1.23 1.234 Output: 1.00 1.20 1.23 1.234 I've tried setting the DisplayFormat ...
user avatar
  • 7,128
0votes
1answer
200views

Variable return types of DecimalField

I have a model thats something like this: class Item(models.Model): value_a = models.DecimalField(default=0.0, decimal_places=4, max_digits=20) value_b = models.DecimalField(default=0.0, ...
user avatar
0votes
0answers
582views

Python: mysql double and float

In MySQL table column balance has a type DOUBLE without specifying precision. And I can't change column type to DECIMAL or DOUBLE with precision (I know that is epic fail to store money in DOUBLE ...
user avatar
  • 109
0votes
2answers
2kviews

Very large float in python

I'm trying to construct a neural network for the Mnist database. When computing the softmax function I receive an error to the same ends as "you can't store a float that size" code is as follows: ...
user avatar
0votes
0answers
45views

String Formatting with Decimal Number Python

I am working on building a report. One figure on the report needs to be expressed in millions. I wrote some basic formatters to handle various types of formatting that need to be consistent throughout ...
user avatar
  • 2,453
0votes
0answers
137views

Limit digits of float values in HTML

I have a reference to a field in Autodesk Infraworks which shows the area number. Field contains float numbers, but they are displayed incorrectly. I.e. instead of 20.8 it shows 20.790000000000000. ...
user avatar
0votes
1answer
256views

Inconsistent Decimal arithmetic in Python (rounding errors)

>>> from decimal import Decimal >>> Decimal('0.3637') * (Decimal('1') / Decimal('0.9323')) == Decimal('0.3637') / Decimal('0.9323') False Why? I thought decimal module solves ...
user avatar
0votes
1answer
679views

How to convert decimal(with float point) to binary in Swift 3? (self written code without third-party library and Foundation)

I am looking for a simple way to convert a decimal with floating point to binary with floating point in Swift 3. For example, this code converts decimal to binary without any problems. func ...
user avatar
  • 77
0votes
0answers
84views

Taking a Floating point number to 30th decimal point in C#

I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002 0.010000000000000000000000000003 0....
user avatar
0votes
0answers
47views

what is the result of (54.125) - (184)10

I am practicing for midterm and apprently there's no answer key for it. However, I practiced and got a result but not sure if this is correct since the solution is really long. perfrom the following ...
user avatar
0votes
1answer
697views

Rounding Off function to given decimal without using round function

I am trying to write a function to round a floating point number up to n decimal places. The function can take one or two arguments. If there is only one argument the number should be rounded to two ...
user avatar
  • 1
0votes
0answers
227views

Accurate javascript decimal number processing with Google Closure Compiler & Library

I'm 6 months into a project using google closure library and compiler. I'm finding I'd like more accurate decimal number handling (handling some simple currency calculations) one of the common ...
user avatar
0votes
2answers
342views

More Decimal Places In Python

Here is my python code: import math import decimal as dec import numpy as np import matplotlib.pyplot as plt c = 3e8 wave = np.array([253.6e-9,283e-9,303.9e-9,330.2e-9,366.3e-9,435.8e-9]) freq =...
user avatar
  • 635
0votes
1answer
62views

Avoid rounding to 0 when a result is very little

I've searched for this but i didn't find anything, i hope this is not a doubled question. I'm doing a formula in TSQL like this: @Temp = SQRT((((@Base1 - 1) * (@StDev1 * @StDev1)) + ((@...
user avatar
  • 45
0votes
1answer
1kviews

Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand. The significand is normalized so that the first digit to the right of the radix point must be a 1. Exponent is signed 2's ...
user avatar
  • 1
0votes
2answers
146views

Understanding machine precision based on bit representation

I am trying to understand how this is calculated: A computer program that represents numbers as follows: 1 bit for the overall sign, 5 bits for the exponent, and 20 bits for the mantissa. Obviously ...
user avatar
0votes
2answers
813views

PHP: string with decimals strips decimals when converting to float

With the function floatval() I try to convert a string with decimals to a float, but it doesn´t give me the desired result. Here´s the code: $price_calc = str_replace(',','.',$price); $...
user avatar

15 30 50 per page