Non AWS server connecting to AWS RDS database is much slower

Question

I am migrating away from a hosting company who currently host my PHP-based web application and MySQL database on the same server. Their server specs are decent, with the server running on an SSD, 11GB RAM and 8-core CPU (GenuineIntel, QEMU Virtual CPU version (cpu64-rhel6), 2399.996 MHz,4096 KB cache).

I can't move everything all at once, so just want to move the database to AWS RDS and update the host in my PHP config to the AWS endpoint.

I have set this up with basic packages available on AWS for testing - t2.micro EC2 (for testing/benchmarking PHP scripts) and db.t2.micro RDS.

Everything works but it is very slow when I connect from my remote, non AWS server. The same database tables, running the same version of MariaDB, with the same number of rows in tables give the following results:

AWS EC2 -> AWS RDS (fully AWS)
Connected via PDO in 0.04 seconds
Queried 1000 rows in 0.63 seconds

Remote Server -> AWS RDS (part AWS)
Connected via PDO in 0.16 seconds
Queried 1000 rows in 5.49 seconds

Remote Server -> Database running on same server (non AWS)
Connected via PDO in 0.00 seconds
Queried 1000 rows in 2.70 seconds

As you can see, the fully AWS set-up is by far the quickest, even on their free-tier packages.

The fully remote server set-up is slower, but still not awful.

The remote server -> AWS set-up is the slowest.

As I am new to AWS, I would really appreciate it if someone could let me know if this is expected. Are non AWS calls to AWS RDS always going to be this much slower?

Could something on either server be throttling the speed?

PS. The remote server is in the UK and have the AWS region set to EU-West, so I don't think it's distance-latency related.

EDIT Rick James asked in the comments below about query cache etc. Caching is disabled but to be sure I have altered my script so that it does the following:

$mysqli = mysqli_connect($dbhost,$username,$password,$dbname) or die ("could not connect to mysql");

$tableCreate = "CREATE TABLE IF NOT EXISTS test_tbl (id int(11) NOT NULL auto_increment, RandomTxt TEXT, PRIMARY KEY (id))";
$queryResult = mysqli_query($mysqli , $tableCreate);

for ($i = 1; $i <= 1000; $i++) {
    mysqli_query($mysqli,"INSERT INTO `test_tbl` (RandomTxt) VALUES ('abcdefghijklmnopqrstuvwxyz')" )  ;
}

$result = mysqli_query($mysqli , 'SELECT * FROM test_tbl') ;
$arrayResults = array() ;
while ($row = $result->fetch_assoc()) {
    array_push($arrayResults , $row['RandomTxt']);
}

$dropTable = "DROP TABLE `test_tbl`";
$queryResult = mysqli_query($mysqli , $dropTable);

After each query, I have a PHP timer script. The results of this updated script are:

AWS EC2 -> AWS RDS (fully AWS)
Db connection established at : 0.04184008
Table created at : 0.05363607
Data inserted into the table at : 3.64998102
Data is read from table and inserted into an array at : 3.65190291
Table dropped at : 3.66061902

Remote Server -> AWS RDS (part AWS)
Db connection established at : 0.08496714
Table created at : 0.11568093
Data inserted into the table at : 21.78033495
Data is read from table and inserted into an array at : 21.82050204
Table dropped at : 21.84762096

Remote Server -> Database running on same server (non AWS)
Db connection established at : 0.00079298
Table created at : 0.00664401
Data inserted into the table at : 0.15589213
Data is read from table and inserted into an array at : 0.16134501
Table dropped at : 0.16468596

I just want to reiterate that I am new to using AWS and if these results are what people would expect, then please let me know. I need to know if connecting to RDS from a non AWS server should be fast enough for production use as at the moment, it certainly isn't.

EDIT 2 I also ran time nc -vz xxx.xxx.eu-west-1.rds.amazonaws.com 3306 to check timings. The result from the non AWS host was:

real    0m0.227s
user    0m0.014s
sys     0m0.005s

The result from EC2 to RDS was:

real    0m0.112s
user    0m0.002s
sys     0m0.000s

Answer 1

For the 'remote' test, I'm surprised it was only 5.49s going across the pond. You have 1002 roundtrips between the US to the UK. This implies 5.5ms per ping time; I thought it was more than that. If you put all 1000 inserts into a single statement , all the 'insert' timings will improve dramatically. This change will help even local code, just not as dramatically (only 10-fold instead of nearly 1000-fold).

As I understand the description of the various configurations, the ping time (up to 5.5ms) is the biggest factor in speed.

The second biggest number is committing a transaction. The 'local' 150ms to commit 1000 rows, one at a time, implies a very fast 0.15ms. That must be SSD.

I would prefer timing with microtime(true) before and after each step, then print the difference. It smells like there is extra overhead in the numbers you provided.

Another thing to verify: Was the table ENGINE=InnoDB (not MyISAM) in all tests? It would be better to be explicit on the CREATE TABLE .

So... Batch INSERTs and do other things to diminish roundtrips. There are other tips, but we need to see the real code, not an artificial benchmark. Another tip is to combine multiple statements into a Stored Procedure; then there is one roundtrip to CALL the proc rather than several.

Answer 2

After a long time debugging this by my host, they have concluded that this isn't something that can be solved from any service in their data centre unless AWS add their own hardware in a data centre near them.

There is the option of moving the application to one of their servers in their London data centre which has less latency to AWS.

So basically it wasn't normal and it is possible to use RDS with a non-AWS hosted application but it does matter where the non-AWS host is located.