All Questions

39 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9votes
1answer
596views

Are .NET Decimal type computations deterministic?

I have two questions regarding .NET's decimal data type determinism: Are decimal type computations cross-platform deterministic? Or in other words, will math operations on decimal type produce ...
user avatar
  • 460
2votes
0answers
47views

Understanding IEEE 754: why convertFromInt and convertToIntegerXXX are categorized as arithmetic operations and not conversion operations?

Note: understanding IEEE 754. Please be patient. IEEE Std 754-2019: 5.4.1 Arithmetic operations ― formatOf-convertFromInt(int) ― intFormatOf-convertToIntegerXXX(source) Question: why convertFromInt ...
user avatar
  • 4,132
2votes
1answer
568views

Casting from (long)double to size_t

I'm trying to implement Sieve of Eratosthenes as efficient as possible. I'd like to set length of my prime array to upper bond of pi(n) < 1.25506n / ln n but I'm not sure how to proceed with ...
user avatar
  • 177
2votes
2answers
2kviews

Rcpp: Converting SEXP to float/double

SEXP callFunction1(List network, List words, Function testWordContinuity){ SEXP res = testWordContinuity(network, words); return res; } for(int i=0; i<(n_epochs); i++){ NumericVector ...
user avatar
  • 178
2votes
0answers
569views

Facing undefined symbols linker issue with Diab compiler when I type cast array of data from float to long long

I wrote a small example code and executed in both GCC and DIAB compilers. #include<stdio.h> int main() { float a[10]; long long int b[10]; int i; for (i =0;i<10;i++) { ...
user avatar
2votes
3answers
1kviews

Inconsistent type conversion in python/numpy when using scalars or lists/arrays

I have a question about the strange way python/numpy performs type conversion. When I perform an arithmetic operation between a float32 and a float64 number, the lower precision is converted to ...
user avatar
  • 75
2votes
1answer
2kviews

SSE: convert from const __m128 * to const float *

I'm trying to write a little SSE code but can't continue because of this error: error C2664: '_mm_loadu_ps' : cannot convert parameter 1 from 'const __m128 *' to 'const float *' I've to load ...
user avatar
  • 156
1vote
1answer
113views

floor-then-convert_int vs convert_int_rtn - equivalent in OpenCL?

In OpenCL, is: convert_int(floor(my_float)) exactly equivalent to convert_int_rtn(my_float) semantically? Would I ever prefer the former command to the latter? If you answer depends on the fact ...
user avatar
  • 99.5k
1vote
2answers
125views

Float.valueOf(String arg) rounding off issue in Java

People, I stumbled upon a basic issue in my application where am trying to convert a string value to it's corresponding Float value : Float val = obj.getFloat("220293.43") ; System.out.println(val); ...
user avatar
1vote
0answers
1kviews

C programming: creating a function to convert double to string

I am trying to code a function that converts a double into a string (a sort of dtoa function). I don't want to use any of the standard library that will do all the job for me (itoa is ok, strlen too, ...
user avatar
1vote
1answer
614views

Conversion of hex to Float in SciLab

I'm trying to read a float number in a hex representation f.ex 43079A11 which should give me 135.601822, (similar to what this site does http://babbage.cs.qc.cuny.edu/IEEE-754.old/32bit.html) using ...
user avatar
1vote
0answers
1kviews

mysql float data type round off's values after 6 decimal places

I have a table (say table1) with one field of data type float (say field1). The values in this field are all unique with upto 8 decimal places. I executed following queries on this table: SELECT ...
user avatar
  • 1,880
1vote
2answers
47views

Convert Long to Float Changing the number c#

I have long vairable long x = 231021578; and when I convert it to float like float y = x; the value of y will be 231021584 I want to know why this happen. Float is stored in 4 bytes and its range is ...
user avatar
  • 17
0votes
0answers
575views

ValueError: could not convert string to float using Pandas

Here I would like to convert all column values to floating point numbers, I have tried the code below but getting error, I have tried and removed ',' (as you can see in comments) but still getting ...
user avatar
0votes
3answers
60views

How can i round a fraction properly to two places after the decimal?

I've written a simple program, which a user types an amount of money, and the computer returns some bills to change. When I type 99.99, the computer cannot handle properly with 0.01 fraction. roundf() ...
user avatar
0votes
0answers
24views

idx = int(float(img_id)) ValueError: could not convert string to float:

I am getting this error when I try to read my files from a given directory. My code is: box_dir = '/home/test/' img_id = img_metas[0]['filename'][34:40] idx = int(float(img_id)) ...
user avatar
0votes
0answers
54views

How to insert value as 'null' instead of 0 to numeric column in sql using c#?

I need to insert float value as 'NULL' for all columns in sql instead of 0.00000 or 0.I have written code as below.Is there any better alternative for this since I should do this for all columns ...
user avatar
  • 27
0votes
2answers
346views

STM32L1 snprintf usage to convert float to string

I have problem with convert float to str on STM32, I have two simple functions to print Int and Float Int work fine, but float print cause program crash void LCD_PrintInt(int value) { char str[8]; ...
user avatar
  • 1
0votes
1answer
37views

Need help converting string or int to float

So I have been stuck on this one for a while and could use some help. I have been trying to fix this code and I keep getting an error about invalid syntax. So here is the code I need help with need to ...
user avatar
0votes
0answers
352views

Data type conversion in GNURadio

I'm reading a stream of float digits from an external program to my GNURadio flowchart through "Socket PDU" bloc. However, the latter delivers a stream of strings instead of floats. I tried writting ...
user avatar
0votes
1answer
299views

What are the rules governing Narrowing conversion from double to float?

Is there a magnitude (decimal range) on which the conversion from double to float is based ? At the beginning I was thinking so but the exponent part confused me .. So what rules are governing that ...
user avatar
0votes
0answers
142views

JPQL number to string conversion disagreeing with Java

Due to some of the restrictions of JPQL (no ordering in subqueries, can't check array equality), I'm having to do some workarounds. Namely, I'm concatenating some numbers (and commas) into a string ...
user avatar
  • 3,891
0votes
0answers
35views

Packing integer as floats, finding original ints

We are packing two 12 bits integers into three 8 bits RGB colors to convey values to a GPU. The process goes like this: Two 16 bits integers are generated (U and V) They are decimated to two 12 ...
user avatar
  • 713
0votes
0answers
73views

Matlab structure with int32 and float values

I created a function that generates a structure to store parameters acquired during a simulation. Some parameters are int32, while other (actually, most of them) are float. The structure is generated ...
user avatar
  • 139
0votes
0answers
31views

Type Conversion for data transmission

i have a problem converting data between different types. I know there are a lot of Questions and How-to's regarding this topic, but i wasn't able to succeed with any of those. I have a transmission ...
user avatar
  • 341
0votes
3answers
906views

Objective-C convert 2 byte into a float

I have 2 byte temperature value like: [0x0C, 0xDE] and I need to convert it to a float, what I've tried to do is: NSData *payload <-- this contains the bytes in the "range" defined after [payload ...
user avatar
  • 2,342
0votes
2answers
337views

Why cant we typecast Float(wrapper) to int(primitive) in java?

While I'm doing typecast from float to int type and typecast from Integer to int it is working. But while i am trying to typecast from Float to int type I'm getting the error as "incompatible type". ...
user avatar
  • 480
0votes
1answer
136views

Something goes wrong when I convert NSString to float

Something goes wrong when I convert NSString to float. I got zero value, but why? NSString *polylineStr = @"51.351305,-0.024239;51.374855,-0.092412"; NSArray *strPolylineArr = [polylineStr ...
user avatar
  • 11.9k
0votes
2answers
2kviews

converting float to fixed point in python

I think I am overthinking this probelem, but I can't seem to solve it. I am reading in little endian floats from a file (range (-1,1)), and want to convert them to fixed point 32b. If I read in the ...
user avatar
  • 341
0votes
1answer
222views

How do I convert a double to a proprietary floating point format in C#?

I have a 20 bit proprietary floating point format defined as: Bits: 19 18 17 16 15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00 Use: s e e e e e m m m m m m m m m m m m m m ...
user avatar
  • 1,214
0votes
1answer
1kviews

Smallest positive number with the following format

12 bit floating point number: 1 bit sign, 4 bit exponent, 7 bit significand. The significand is normalized so that the first digit to the right of the radix point must be a 1. Exponent is signed 2's ...
user avatar
  • 1
0votes
0answers
133views

fixed point datatype with PHP 5

Am I wrong (I hope), but I don't find on PHP a fixed-point datatype !? Datatype like DECIMAL/NUMERIC from MySQL will be nice for doing operations on money without precision loss. Any ideas ?
user avatar
  • 3,526
0votes
1answer
2kviews

inserting c# double values to sql server float field. (no fractional part)

I am trying to insert double values to my database via EF 5. I generated EF entity model from db. There is a price column in the table which is float, and naturally EF generated a double type for the ...
user avatar
  • 1,307
0votes
1answer
4kviews

Converting a float to an integer

There is a catch! I have a IEEE 754 single precision (32 bit) float stored in two consecutive 16bit integers. The processor I am using doesn't have floating point maths or float data types! What I ...
user avatar
  • 1,858
-1votes
1answer
104views

converting doubles to long doubles, understanding of 'binary vs. decimal' ambiguity? and is there a standard solution?

[edit 2021-09-26] sorry!, I have to admit that I asked crap here, explanation follows. I don't think I should post this as an 'answer', so as an edit: I'm still curious how a 'double' value of 0.1 ...
user avatar
-1votes
1answer
212views

float.Parse(string) without exponent in C#

Pretty simple situation, but I got stuck exactly here. So, I have next method: public static float? ToSingleNullable(string value) { if (string.IsNullOrEmpty(value)) return null; else ...
user avatar
-1votes
1answer
3kviews

64 Bit Hex conversion to decimal

I have a 16 Bit hex value: 4164ACFCE33404EA After conversion, the number is: 10840039.1001 I've tried both IEEE single and double precision 64-bit to decimal and a variety of 16 bit two ...
user avatar
-1votes
1answer
164views

Convert a float to an int

I have a price for a car, let's say 10000. I want to apply a 20% sale to this price. I have a struct in which auta->cena is a float. int year, i, n=0, pocet=1, sale; scanf(" %d", &year); ...
user avatar
-2votes
2answers
164views

How to check the input format in java

i need to check whether the number input is in decimal format or in floating point format in java coding. in simple terms how would this check be possible?
user avatar
  • 1