在 Excel 中挂钩 ReadFile function

Question

I use XOR each byte with 'A' to generate encrypted file. And do that again to decrypt file -> Excel open decrypted file normally.

Now I want to open an encrypted file by Excel.

I hook the ReadFile API to decrypt the buffer before returning it.

But Excel displays

Excel cannot be open the file 'filename.xlsx' because the file format or file extension is not valid

after reading the first 8 bytes.

The decrypted 8 bytes are 50 4B 03 04 14 00 06 00 , which is the correct Open Office XML Signature and same with the original file.

Here is the myReadFile function:

BOOL WINAPI MyReadFile(
_In_ HANDLE hFile,
_Out_writes_bytes_to_opt_(nNumberOfBytesToRead, *lpNumberOfBytesRead) __out_data_source(FILE) LPVOID lpBuffer,
_In_ DWORD nNumberOfBytesToRead,
_Out_opt_ LPDWORD lpNumberOfBytesRead,
_Inout_opt_ LPOVERLAPPED lpOverlapped){

BOOL result = ReadFile(hFile, lpBuffer, nNumberOfBytesToRead, lpNumberOfBytesRead, lpOverlapped);
if (!result) {
    return result;
}

if (*lpNumberOfBytesRead == 0) {
    return result;
}
char* Buf = (char*)lpBuffer;
for (long i = 0; i < (*lpNumberOfBytesRead); i++)
{
    Buf[i] = Buf[i] ^ 'A';
}
return result;}

Does need hook other API to open encrypted file perfectly.

Answer 1

Does need hook other API to open encrypted file perfectly.

Might be. A file migt be read other way after reading by ReadFile you've observed.

There are ReadFileEx and ReadFileScatter , though the first is rarely called, the second is even more rare. An application might directly call NtReadFile , though it is not much likely too.

What is more likely, is creating file mapping using CreateFileMapping or equivalent, and then using MapViewOfFile or equivalent. In this case I don't see a clean way of doing it via hooking.

It is also possilbe that is is read from some other process.

My suggestion is to getprocmon tool, filter accesses to the target file path, and see all file operations. You can see call stacks of each operation and see what you need to hook.