All Questions

Tagged with
67 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4votes
0answers
384views

C# Change FPU rounding mode

I'm attempting to write an interval arithmetic library in C# .NET, but in order to do this accurately I need to be able to control the rounding mode of floating point operations. After a bit of ...
user avatar
  • 639
4votes
2answers
1kviews

How to correctly pass a float from C# to C++ (dll)

I'm getting huge differences when I pass a float from C# to C++. I'm passing a dynamic float wich changes over time. With a debugger I get this: c++ lonVel -0.036019072 float c# lonVel -0....
user avatar
3votes
1answer
146views

What guarantees does System.Numerics.Vectors provide about size and bit order?

I have implemented a vector-based c# approximation of Log. It includes unsafe code. It's been working fine in a number of environments, but on a recent deployment has fallen over. The implementation ...
user avatar
3votes
1answer
312views

Is it safe to cast Math.Round result to float?

A colleague has written some code along these lines: var roundedNumber = (float) Math.Round(someFloat, 2); Console.WriteLine(roundedNumber); I have an uncertainty about this code - is the number ...
user avatar
  • 124k
3votes
0answers
196views

Calculating floating point error bound

I have geometrical algorithms and im struggling with floating point inaccuracies. For example, I'm calculating wether a point lies on the left/right/on a plane (C#): const double Epsilon = 1e-10; ...
user avatar
  • 610
3votes
2answers
1kviews

Incorrect float subtraction result

DataSet id qty CheckIn CheckOut 5 10 1 0 5 10 0 1 5 1.6 1 0 5 0.4 0 1 5 0.4 0 1 5 0.4 0 1 I am trying to ...
user avatar
2votes
1answer
126views

Double values comparison w/ tolerance for "point" equality in a DXF drawing

I have a simple algorithm that fails sometimes because it is comparing doubles. I look at a DXF drawing and get all the line segments, and also break it down into a series of points. When looping ...
user avatar
2votes
0answers
40views

Casting long.MaxValue -> float -> long results in overflown value

We noticed quite a problem in our codebase lately which caused our RNG to go bonkers. Currently all our game variables are saved as floats, among them the max long value from the game server. When ...
user avatar
2votes
0answers
657views

What MySQL "DECIMAL" precision and scale is required to store a C# decimal value?

I want to store some decimal values in my MySQL database. The obvious MySQL database type for that would be DECIMAL I guess. What precision "a" and scale "b" do I have to set for DECIMAL(a,b ) ...
user avatar
  • 5,504
2votes
3answers
3kviews

32-bit Grayscale Tiff with floating point pixel values to array using LibTIFF.NET C#

I just started using LibTIFF.NET in my c# application to read Tiff images as heightmaps obtained from ArcGIS servers. All I need is to populate an array with image's pixel values for terrain ...
user avatar
2votes
1answer
141views

How to fix floating point artifacts in StdDev calculation?

I trying to calculate standard deviation with a next method: private static double? StdDev(IReadOnlyCollection<double> items) { if(items == null) { throw new ArgumentNullException("items")...
user avatar
2votes
0answers
1kviews

Show Amount upto two decimal places without rounding off in DataGrid

Say for example i have a large number 12345678.88 right now when i display it in the Datagrid with the use of <asp:Label ID="txtlineAmt" runat="server" ReadOnly="true" Text='<%# Eval("...
user avatar
  • 4,442
2votes
2answers
272views

hashing a floating point number for test case purposes in the .Net runtime

I have complex test cases with geometric entities. It is easy for me to visually verify that the test case has passed by looking at the generated geometry in a viewport like so. When I am satisfied ...
user avatar
  • 385
2votes
5answers
2kviews

Join two strings into a single decimal value

I want to get a decimal value, for example 500.45, from two strings. How to achieve this operation? decimal value; //like 500.45 string txt1, txt2; txt1 = "500"; txt2 = "45"; value = Convert....
user avatar
  • 1,348
1vote
1answer
74views

Strange double subtraction result in .NET

I'm aware of the inaccuracies of floating point operations but the results of the below code seem rather odd to be. Can anybody please explain it? This is with .NET Framework 4.8 on Windows 10. The ...
user avatar
1vote
0answers
126views

Higher precision floating point calculations

I would like to find a way to make higher precision calculations in a compute shader using multiple floats to represent a single number. My goal is a mandelbrot renderer where you can zoom deeper ...
user avatar
1vote
0answers
46views

Is it possible to manually increase the precision of a double as a bit string through calculations

In order to create a floating point number from a decimal value, we need to have a sign bit, and exponent, and a mantissa. Double precision uses 11 exponent bits, and 52 mantissa bits. Let's say we ...
user avatar
1vote
0answers
74views

Puzzled over Float vs Double benchmark

After searching SO for performance differences between floats and doubles in .NET, I put together a simple benchmark to test 2 methods, each with floats and doubles, debug and release. After running ...
user avatar
1vote
0answers
83views

How do I set the default decimal number separator in Python?

I am working on two applications, one running in C#/.NET and the other with the Python 3 interpreter targeting Windows 10 only. The two applications communicate with each other using a local network ...
user avatar
  • 3,625
1vote
0answers
63views

Do .NET floating-point operations agree between x64 debug and release?

In this question, the asker wondered whether or not one could expect floating point operations to behave similarly between debug and release configurations. The answers given there show that this is ...
user avatar
  • 15.7k
1vote
0answers
246views

using float.Parse() vs casting to get a value from double

I wasn't able to find a question that directly answers this question, I came across this line of code in a project at work (I added the comment) // act.score is a double float score = float.Parse(act....
user avatar
  • 8,497
1vote
2answers
148views

Reversible decimal to double conversion

I'm curious to know if you could convert decimal to double, but still be able to come back to the same decimal I started with. Of course, assuming you use 15 or less significant digits. Basically, can ...
user avatar
1vote
1answer
1kviews

C# Decimal type rounding off inappropriately

I've got an in house banking solution for a summer camp that I'm a part of. The issue is coming up that some accounts are exactly a penny off at the end of the season. I'm using SQL Server Express, ...
user avatar
1vote
0answers
69views

Formatting numbers in vertically aligned textboxes in C# with the floating point also vertically aligned

On my UI, there are several textboxes that are vertically aligned. I'd like to show the numbers in them in a way that all floating points are also vertically aligned. I also need to show the sign of ...
user avatar
  • 889
1vote
1answer
179views

Could Mono.Simd or Microsoft.Bcl.Simd provide deterministic floating point arithmetic to C#?

C# cannot be coerced into deterministic floating point arithmetic, the designers seem to have considered the performance overhead too significant. I'm trying to assess whether it would be possible to ...
user avatar
  • 4,110
1vote
0answers
557views

Converting 4-byte OLE (VB6) to double/float in C#

Well, I have a 4 (four) bytes array that I need to convert to double data-type (or float) in C#. I believe that the 4 bytes array is an OLE Automation one. Tried the following, but no success: // ...
user avatar
  • 109
1vote
1answer
457views

Extjs float value is being passed as null in c# controller

I have a float value in extjs model. For some reason any value I enter and pass to c# controller, its value is always null. I checked the json data being pass and I can see the passed value. This is ...
user avatar
  • 1,357
1vote
0answers
147views

Byte[] to float conversion issue

I have a binary file written in big endian enviorment using c code. er On my PC a C# application is reading bytes from the file using BinaryReader and converting it to Managed struct using Marshalling....
user avatar
  • 247
1vote
2answers
578views

Floats changing when passing them from CLI to C#

EDIT : i wrote the same code that i had at work at home to recreate the bug. this is the exact code !! I'm trying to pass floating number from c++ to c# and i'm using c++ CLI as intermediate. I have ...
user avatar
  • 4,420
1vote
2answers
483views

C#: Store number in database with 2 places of decimal. Eliminate unwanted numbers

I have a field NumberValue1 declared like this public double NumberValue1 { get; set; } NumberValue1 has a datatype of Number in the oracle database I read in a value from an excel file which is 22....
user avatar
  • 1,657
1vote
0answers
2kviews

Getting significand and exponent value from floating point literal with C#

A literal of 0.25 can be expressed with 1*2^(-2) so that we can get 1 as significand and -2 as exponent. What function can I get the significand/exponent with C#? int significand = GetSignificand(0....
user avatar
  • 167k
1vote
2answers
47views

Convert Long to Float Changing the number c#

I have long vairable long x = 231021578; and when I convert it to float like float y = x; the value of y will be 231021584 I want to know why this happen. Float is stored in 4 bytes and its range is ...
user avatar
  • 17
1vote
2answers
2kviews

Error when trying to save data to a database with dbcontext. Not a valid instance of type float

I cant seem to figure this out. The error I get is "The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is >incorrect. Parameter 13 ("@10"): The supplied value is ...
user avatar
  • 697
0votes
0answers
55views

Need a .NET Framework solution for IEEE-754 float/double parse and ToString

.NET framework does not give me expected results for float/double parsing and ToString. It is fixed in .NET 6 (even in .NET Core 3.0). However, I cannot migrate my whole project from .NET framework 4....
user avatar
  • 34
0votes
0answers
61views

.NET 6.0 vs .NET Framework Number Rounding

As I was migrating a complex .NET Framework project to .NET 6.0, I noticed different results under some circumstances. I was able to trace down the issue to a basic inequality involving floats and a ...
user avatar
  • 338
0votes
0answers
33views

I am having trouble evaluating the height in c#, when i introduce a number smaller than 1.5 still says come in

using System; namespace Tarea_C_ { class Program { static void Main(string[] args) { Console.Write("Write your age: "); int age = Convert....
user avatar
0votes
0answers
58views

Is there a way to tell the compiler to compile with "fast-math" or something similar in C#?

In some other languages, you can allow the compiler to perform some floating point optimizations, one of which is rewriting your code into algebraically equivalent expressions. Here are some examples ...
user avatar
  • 122
0votes
0answers
54views

How to insert value as 'null' instead of 0 to numeric column in sql using c#?

I need to insert float value as 'NULL' for all columns in sql instead of 0.00000 or 0.I have written code as below.Is there any better alternative for this since I should do this for all columns ...
user avatar
  • 27
0votes
0answers
63views

C# - Converting data from Decimal to float is truncating after decimal points while retrieving data from database

I am trying to fetch percentage value from database and my database query gives 2 digits after decimal, but when i am trying to access that value in c# code, the decimal point and the 2 digits after ...
user avatar
  • 201
0votes
0answers
132views

Why does float.TryParse() accepts "1.000e-45" and rounds it to 1.401298E-45?

I try to validate that an input string is a float. To achieve this I pass the input string like "1.000e-45" which is closer to 0 than the c# float minimum allows (1.401298E-45). I expect ...
user avatar
  • 2,012
0votes
1answer
36views

Cutting corners with float-to-string representation?

I have a float I need to turn into a string with 5 decimals precision (X.XXXXX), which means I need to have at least 6 decimals for round up/down. The issue is that the operation to get integer ...
user avatar
  • 11
0votes
0answers
80views

Why .net C# float is truncated to thousand in string representation?

How come the following is possible ? immediate console: "{0:F20},{1:F20}, {2:F20}".format( res_ts, // float AppMgr.logon_info.res_ts, //float res_ts - AppMgr.logon_info.res_ts ) "{0},{...
user avatar
  • 946
0votes
0answers
106views

How to find the similarity (as a percentage) between two float[] arrays in c#

I need to get the similarity between two float[] arrays of the same lengths and return have this similarity be returned as a float which will be the percentage. How would you do this in c#?
user avatar
0votes
1answer
485views

Is it valid check for infinity with the equals operator?

c# how to check if a number is infinity Answer: Double.IsInfinity, Double.IsPosiviteInfinity, etc. So now comes my question: Is it valid to simply do: double d = Double.PositiveInfinity; if(d == ...
user avatar
0votes
0answers
58views

How to prevent Floating Point Excel Data loaded with SQL to be truncated

When I load floating point data from Excel in my C# software using SQL the data ends up being truncated. For example, instead of 0,054034 I end up with 0,054. The relevant part of my code is: ...
user avatar
0votes
1answer
82views

More elegant way to determine whether two floats have same signs in C#?

Is there any way that can compare the signs of the two floats quickly in C#? I doubt the performance of comparing a float with 0 again and again. However, it seems that bit operation of float is ...
user avatar
0votes
0answers
58views

Converting floats to ints reliably

I have an application that uses floats to handle distances, for example, 2.5f is 2.5 meters (250 centimeters). Now I want to interface with that application without using floats via network. I can't ...
user avatar
  • 872
0votes
0answers
84views

Taking a Floating point number to 30th decimal point in C#

I have a number which is 0.010000000000000000000000000001. I want this number to be stored as it is and want to increment to it like 0.010000000000000000000000000002 0.010000000000000000000000000003 0....
user avatar
0votes
1answer
56views

IEEE floating point addition

instead of float a = 32.342 , b = 193.132 float total = a + b How can I convert them to 32 bits and add using their 32 bits explicitly?
user avatar
  • 727
0votes
0answers
258views

Float Precision in the Visual Studio Immediate Window

Undertaking a code review I have come across some strange behaviour I don't understand in the C# immediate window. all the variables below are floats and calculate the gradient of a line. ...
user avatar
  • 2,950

15 30 50 per page