如何在x86中将进位和溢出标志用于计算乘法

Question

How are the two flags used to correctly calculate the answer when two numbers that are multiplied overflow the register?

Eg if al holds 0xff and is multiplied by 0x2 , causing an overflow into ax , how do the flags help with this?

Answer 1

Multiplication on x86/x64 never overflows when using the one operand form.
This is because mul and its sibling imul produce an output twice as wide as their operands ¹ .
In your example, multiplying by al produces an output in ax and no overflow is generated.

The CF and OF are set when the result cannot fit in the operands size.
This can be used to perform multiplication with saturation, for example:

;Unsigned

mul ebx
sbb edx, edx      ;EDX = CF replicated along all the 32 bits
or eax, edx       ;EAX = 0ff..ffh if overflow, EAX*EBX otherwise

;Signed

imul ebx
cmovc eax, 7fffffffh  ;Signed positive max if overflow
and edx, 80000000h    ;Keep only the sign of the result
or eax, edx           ;If no overflow bit 31 of EAX = bit 31 of EDX
                      ;and this is a nop, otherwise set EAX to negative max
                      ;if the result was negative

However to implement a multi-precision multiplication, say 64x64-bit, those flags are not needed, in fact, denoting with 2 ³² with k we have:

(a· k +b) × (c· k +d) = a·c· k ² + (a·d+b·c)· k + b·d

where the 32-bit products produce 64-bit results that are added as below

.----.----.
|   b·d   |
'----'----'
       +
     .----.----.
     | a·d+b·c |
     '----'----'
            +
          .----.----.
          |   a·c   |
          '----'----'

 =

.----.----.----.----.
|    128-bit result |
'----'----'----'----' 

---

¹ And this suffices to prevent overflow.

Answer 2

The best way to answer questions like this is to read the manual.

Now I am going back to my paper copy of an 80188/186 (I don't know where my 8088/8086 programming manual is). But even back then, it is like folks are saying. The text says

If the source is a byte, then it is multiplied by register AL, and the double length result is returned in AH and AL.

And that is all fine: you can't overflow, as folks are saying, but folks don't normally write high-level language code that uses a result twice the size of the operands. It goes further to say:

If the upper half of the result is non-zero CF and OF are set; otherwise they are cleared.

So there you go: if you are doing an 8-bit operation and the result does not fit in 8 bits (0xFF * 2 = 0x1FE), then both CF and OF are set. At least, they were 20 years ago. It should be quite trivial to do an experiment to see what your current x86 does. You could have crafted this experiment before posting here, walk through every combination of operands and see what the flags do for each.