I stumbled upon a problem, where the result of the floating point operation depends on whether I subtract a value now (f) or later (f2). Why is that?

int main(int argc, char * argv[])
    const float fResolution = 0.501f;
    const int iDiff = -6; // some values lead to f == f2, like -1, -2, -3, -4 or -5
    float f = iDiff * fResolution - 10.0f;
    float f2 = iDiff * fResolution;
    f2 -= 10.0f;

    printf("Difference %f %f %f: %d\n", f, f2, f - f2, iDiff);
    return 0;

Output: Difference -13.006000 -13.006001 0.000001: -6

I guess, it has something to do with floating point precision, but there are only floats here, so what is the difference between f and f2? Or is it maybe a special behavior in Visual Studio 2010?

This is of course a minimal example. In the real world, I extracted the calculation of iDiff * fResolution into a method, and sometimes subtract -10. The output of my program differed after extracting the method because of the above problem.

Disassembly (Debug Mode):

    const float fResolution = 0.501f;
00CC7872  fld         dword ptr [__real@3f004189 (0D4AEACh)]  
00CC7878  fstp        dword ptr [ebp-0F8h]  
    const int iDiff = -6;
00CC787E  mov         dword ptr [ebp-104h],0FFFFFFFAh  
    float f = iDiff * fResolution - 10.0f;
00CC7888  fld         dword ptr [ebp-0F8h]  
00CC788E  fmul        qword ptr [__real@c018000000000000 (0D4AED8h)]  
00CC7894  fsub        qword ptr [__real@4024000000000000 (0D4AE58h)]  
00CC789A  fstp        dword ptr [ebp-110h]  
    float f2 = iDiff * fResolution;
00CC78A0  fld         dword ptr [ebp-0F8h]  
00CC78A6  fmul        qword ptr [__real@c018000000000000 (0D4AED8h)]  
00CC78AC  fstp        dword ptr [ebp-11Ch]  
    f2 = f2 - 10.0f;
00CC78B2  fld         dword ptr [ebp-11Ch]  
00CC78B8  fsub        qword ptr [__real@4024000000000000 (0D4AE58h)]  
00CC78BE  fstp        dword ptr [ebp-11Ch] 
  • 2
    Intermediate calculations can be stored in more precise types. When data is stored in a variable all extra precision is lost. Maybe. If compiler optimises away some variables, final value can change again. Fun fact: sin(x) == sin(x) might yield false (on older GCC compilers). Jul 8, 2016 at 14:00
  • 1
    It likely has to do with the intermediate storage of the floating-point values. If you're targeting x86-32 and telling it precision is not of utmost importance, the optimizer is probably generating code that leaves intermediate values on the 80-bit FPU stack, rather than spilling it to memory to preserve constant precision. No way to tell for sure without looking at the disassembly. Which is hardly worth doing, relying on floating point computations for precision is a fool's errand.
    – Cody Gray
    Jul 8, 2016 at 14:00
  • obligatory plugs: Is floating point math broken? and Floating point calculation gives different results with float than with double(how more precision can affect the results) Jul 8, 2016 at 14:12
  • 2
    The result of the instruction at address 00CC788E is kept stored inside the FPU at 64-bit precision. At address 00CC78AC you force it to be stored back to memory, truncating the result to 24-bit precision. Intel thought 36 years ago that this was a desirable feature, armies of programmers since have disagreed. Fwiw, the same thing will happen in the Release build of your program, but now it is no longer under your control. Do update your VS version, FPU usage was retired at VS2012. Jul 8, 2016 at 14:14
  • Relevant reading : docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html Jul 8, 2016 at 14:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.