# All Questions

39 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
596views

### Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism: Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
• 460
47views

### Understanding IEEE 754: why convertFromInt and convertToIntegerXXX are categorized as arithmetic operations and not conversion operations?

Note: understanding IEEE 754. Please be patient. IEEE Std 754-2019: 5.4.1 Arithmetic operations ― formatOf-convertFromInt(int) ― intFormatOf-convertToIntegerXXX(source) Question: why convertFromInt ...
• 4,132
568views

### Casting from (long)double to size_t

I'm trying to implement Sieve of Eratosthenes as efficient as possible. I'd like to set length of my prime array to upper bond of pi(n) < 1.25506n / ln n but I'm not sure how to proceed with ...
• 177
2kviews

### Rcpp: Converting SEXP to float/double

SEXP callFunction1(List network, List words, Function testWordContinuity){ SEXP res = testWordContinuity(network, words); return res; } for(int i=0; i<(n_epochs); i++){ NumericVector ...
• 178
569views

### Facing undefined symbols linker issue with Diab compiler when I type cast array of data from float to long long

I wrote a small example code and executed in both GCC and DIAB compilers. #include<stdio.h> int main() { float a[10]; long long int b[10]; int i; for (i =0;i<10;i++) { ...
1kviews

### Inconsistent type conversion in python/numpy when using scalars or lists/arrays

I have a question about the strange way python/numpy performs type conversion. When I perform an arithmetic operation between a float32 and a float64 number, the lower precision is converted to ...
• 75
2kviews

### SSE: convert from const __m128 * to const float *

I'm trying to write a little SSE code but can't continue because of this error: error C2664: '_mm_loadu_ps' : cannot convert parameter 1 from 'const __m128 *' to 'const float *' I've to load ...
• 156
1vote
113views

### floor-then-convert_int vs convert_int_rtn - equivalent in OpenCL?

In OpenCL, is: convert_int(floor(my_float)) exactly equivalent to convert_int_rtn(my_float) semantically? Would I ever prefer the former command to the latter? If you answer depends on the fact ...
• 99.5k
1vote
125views

### Float.valueOf(String arg) rounding off issue in Java

People, I stumbled upon a basic issue in my application where am trying to convert a string value to it's corresponding Float value : Float val = obj.getFloat("220293.43") ; System.out.println(val); ...
• 113
1vote
1kviews

### C programming: creating a function to convert double to string

I am trying to code a function that converts a double into a string (a sort of dtoa function). I don't want to use any of the standard library that will do all the job for me (itoa is ok, strlen too, ...
• 71
1vote
614views

### Conversion of hex to Float in SciLab

I'm trying to read a float number in a hex representation f.ex 43079A11 which should give me 135.601822, (similar to what this site does http://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html) using ...
1vote
1kviews

### mysql float data type round off's values after 6 decimal places

I have a table (say table1) with one field of data type float (say field1). The values in this field are all unique with upto 8 decimal places. I executed following queries on this table: SELECT ...
• 1,880
1vote
47views

### Convert Long to Float Changing the number c#

I have long vairable long x = 231021578; and when I convert it to float like float y = x; the value of y will be 231021584 I want to know why this happen. Float is stored in 4 bytes and its range is ...
• 17
575views

### ValueError: could not convert string to float using Pandas

Here I would like to convert all column values to floating point numbers, I have tried the code below but getting error, I have tried and removed ',' (as you can see in comments) but still getting ...
• 151
60views

### How can i round a fraction properly to two places after the decimal?

I've written a simple program, which a user types an amount of money, and the computer returns some bills to change. When I type 99.99, the computer cannot handle properly with 0.01 fraction. roundf() ...
24views

### idx = int(float(img_id)) ValueError: could not convert string to float:

I am getting this error when I try to read my files from a given directory. My code is: box_dir = '/home/test/' img_id = img_metas[0]['filename'][34:40] idx = int(float(img_id)) ...
54views

### How to insert value as 'null' instead of 0 to numeric column in sql using c#?

I need to insert float value as 'NULL' for all columns in sql instead of 0.00000 or 0.I have written code as below.Is there any better alternative for this since I should do this for all columns ...
• 27
346views

### STM32L1 snprintf usage to convert float to string

I have problem with convert float to str on STM32, I have two simple functions to print Int and Float Int work fine, but float print cause program crash void LCD_PrintInt(int value) { char str[8]; ...
• 1
37views

### Need help converting string or int to float

So I have been stuck on this one for a while and could use some help. I have been trying to fix this code and I keep getting an error about invalid syntax. So here is the code I need help with need to ...
352views

### Data type conversion in GNURadio

I'm reading a stream of float digits from an external program to my GNURadio flowchart through "Socket PDU" bloc. However, the latter delivers a stream of strings instead of floats. I tried writting ...
299views

### What are the rules governing Narrowing conversion from double to float?

Is there a magnitude (decimal range) on which the conversion from double to float is based ? At the beginning I was thinking so but the exponent part confused me .. So what rules are governing that ...
142views

### JPQL number to string conversion disagreeing with Java

Due to some of the restrictions of JPQL (no ordering in subqueries, can't check array equality), I'm having to do some workarounds. Namely, I'm concatenating some numbers (and commas) into a string ...
• 3,891
35views

### Packing integer as floats, finding original ints

We are packing two 12 bits integers into three 8 bits RGB colors to convey values to a GPU. The process goes like this: Two 16 bits integers are generated (U and V) They are decimated to two 12 ...
• 713
73views

### Matlab structure with int32 and float values

I created a function that generates a structure to store parameters acquired during a simulation. Some parameters are int32, while other (actually, most of them) are float. The structure is generated ...
• 139
31views

### Type Conversion for data transmission

i have a problem converting data between different types. I know there are a lot of Questions and How-to's regarding this topic, but i wasn't able to succeed with any of those. I have a transmission ...
• 341
906views

### Objective-C convert 2 byte into a float

I have 2 byte temperature value like: [0x0C, 0xDE] and I need to convert it to a float, what I've tried to do is: NSData *payload <-- this contains the bytes in the "range" defined after [payload ...
• 2,342
337views

### Why cant we typecast Float(wrapper) to int(primitive) in java?

While I'm doing typecast from float to int type and typecast from Integer to int it is working. But while i am trying to typecast from Float to int type I'm getting the error as "incompatible type". ...
• 480
136views

### Something goes wrong when I convert NSString to float

Something goes wrong when I convert NSString to float. I got zero value, but why? NSString *polylineStr = @"51.351305,-0.024239;51.374855,-0.092412"; NSArray *strPolylineArr = [polylineStr ...
• 11.9k
2kviews

### converting float to fixed point in python

I think I am overthinking this probelem, but I can't seem to solve it. I am reading in little endian floats from a file (range (-1,1)), and want to convert them to fixed point 32b. If I read in the ...
• 341
222views

### How do I convert a double to a proprietary floating point format in C#?

I have a 20 bit proprietary floating point format defined as: Bits: 19 18 17 16 15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Use: s e e e e e m m m m m m m m m m m m m m ...
• 1,214
1kviews

### Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand. The significand is normalized so that the first digit to the right of the radix point must be a 1. Exponent is signed 2's ...
• 1
133views

### fixed point datatype with PHP 5

Am I wrong (I hope), but I don't find on PHP a fixed-point datatype !? Datatype like DECIMAL/NUMERIC from MySQL will be nice for doing operations on money without precision loss. Any ideas ?
• 3,526
2kviews

### inserting c# double values to sql server float field. (no fractional part)

I am trying to insert double values to my database via EF 5. I generated EF entity model from db. There is a price column in the table which is float, and naturally EF generated a double type for the ...
• 1,307
4kviews

### Converting a float to an integer

There is a catch! I have a IEEE 754 single precision (32 bit) float stored in two consecutive 16bit integers. The processor I am using doesn't have floating point maths or float data types! What I ...
• 1,858
104views

### converting doubles to long doubles, understanding of 'binary vs. decimal' ambiguity? and is there a standard solution?

[edit 2021-09-26] sorry!, I have to admit that I asked crap here, explanation follows. I don't think I should post this as an 'answer', so as an edit: I'm still curious how a 'double' value of 0.1 ...
212views

### float.Parse(string) without exponent in C#

Pretty simple situation, but I got stuck exactly here. So, I have next method: public static float? ToSingleNullable(string value) { if (string.IsNullOrEmpty(value)) return null; else ...
3kviews

### 64 Bit Hex conversion to decimal

I have a 16 Bit hex value: 4164ACFCE33404EA After conversion, the number is: 10840039.1001 I've tried both IEEE single and double precision 64-bit to decimal and a variety of 16 bit two ...