linux 2.6.43，ext3、10K RPM SAS磁盘，在不同文件上的2个连续写入（direct io），就像随机写入一样

Question

I recently stall on this one problem:

"2 sequential write(direct io 4KB alignemnt block) on different file acting like random write, which yield poor write performance in 10K RPM SAS disk".

The thing confuse me most: I got batch of server, all equip with same kind of disk (raid 1 with 2 300GB 10K RPM disk), but response different.

• several servers seams ok with this kind of write pattern, disk happy accepted up to 50+MB/s; (same kernel version, same filesystem, with different lib (libc 2.4))
• others not so much, 100 op/s seams reach the limit of underlying disk, which confirm the random write performance of disk; ((same kernel version, same filesystem, with different lib (libc 2.12)))

[NOTE: I check the "pwrite" code of different libc, which tell nothing but simple "syscall"]

I have managed to rule out the possibly: 1. software bug in my own program; by a simple deamon(compile with no dynamic link), do sequcetial direct io write; 2. disk problem; switch 2 different version of linux system on one test machine, which perform well on my direct io write pattern, and a couple of day after switch to old lib version, the bad random write;

I try to compare:

1. /sys/block/sda/queue/*, which may different in both way;
2. filefrag show nothing but two different file interleaved sequenctial grow physical block id;

there must be some kind of write strategy lead to this problem, but i don't know where to start:

1. different kernel setting ?, may be related to how ext3 allocate disk block ?
2. raid cache(write back) or disk cache write strategy?
3. or underlying disk strategy to mapping logical block into real physical block ?

really appreciate

Answer 1

THE ANS IS: it's because of /sys/block/sda/queue/schedule setting:

1. MACHINE A: display schedule: cfq, but undlying, it's deadline;
2. MACHINE B: the schedule is consistent with cfq; //=> SINCE my server is db svr, deadline is my best option;