Packed Shuffle Bytes, reverse operation

Question

Need an expert on AVX:

given the instruction for shuffling

VPSHUFB ymm1, ymm2, ymm3/m256

Accordingly to docs, this "Shuffle bytes in ymm2 according to contents of ymm3/m256."

My question is: is it possible to reverse the operation, in other words, knowing ymm2 and the result of the shuffling, is possible to get ymm3?

For example vpshufb ymm0, ymm0, ymmword ptr ds:[0x100]

I know what is already loaded in ymm0, and the result, but I'd like to understand how to get the "control mask" ymmword....

Answer 1

is it possible to reverse the operation, in other words, knowing ymm2 and the result of the shuffling, is possible to get ymm3?

No, in general it's not. If the source data had some duplicate elements (within lanes), you can't uniquely determine what the shuffle control vector was because that output byte could have come from multiple places.

It's of course easy to come up with a shuffle control that would create that output from that input, though.

---

Of course if the shuffle input data was unique within each 16-byte lane, then yes you can uniquely determine the index that each output element must have come from.

Even then you can only determine the low 4 bits of each byte of the shuffle control . (And that the high bit was unset, otherwise vpshufb zeroes that byte of the output.) Bits 6:4 have no effect at all on the operation. (Look at the Operation section of the manual: https://www.felixcloutier.com/x86/pshufb . Bits 3:0 of the shuffle control vector elements are indices into the source data vector.)

The high bit of the shuffle control elements are another possible ambiguity: if the input data contained any zero bytes, an output zero could have come from indexing that source element or from that control element having its high bit set.

---

And BTW, there aren't any instructions that will help you do this efficiently . IDK if you can do better than searching for the source position of each byte 1 at a time. (Using a byte-broadcast and vpcmpeqb -> vpmovmskb -> bsf to search all 16 positions at once.)