MySQL Query Is Too Slow Or Times Out

Question

I am having a complete nightmare with my application. I haven't worked with datasets this big before, and my query is either timing out or taking ages to return something. I've got a feeling that my approach is just all wrong.

I have a payments table with a postcode field (among others). It has 40,000 rows roughly (one for each transaction). It has an auto-inc PRIMARY key and an INDEX on the postcode foreign-key.

I also have a postcodes lookup table with 2,500,000 rows. The table is structured like so;

postcode | country | county | localauthority | gor
AB1 1AA    S99999    E2304     X               45
AB1 1AB    S99999    E2304     X               45

The postcode field is PRIMARY and I have INDEXes on all the other fields.

Each field (apart from postcode) has a lookup table. In the case of country it's something like;

code   | description
S99999   Wales

The point of the application is that the user can select areas of interest (such as "England", "London", "South West England" etc) and be shown payments results for those areas.

To do this, when a user selects the areas they are interested, I then created a temp table, with one row, listing ALL postcodes for the areas they selected. Then I LEFT JOIN it on to my payments table.

The problem is that if the user selects a big region (like "England") then I have to create a massive temp table (or about 1 million rows) and then compare it to the 40,000 payments to decide which to display.

UPDATE

Here is my code;

$generated_temp_table = "selected_postcodes_".$timestamp_string."_".$userid;
    $temp_table_data = $temp_table
    ->setTempTable($generated_temp_table)
    ->newQuery()
    ->with(['payment' => function ($query) use ($column_values) {
        $query->select($column_values);
    }])
    ;

Here is my attempt to print out the raw query;

    $sql = str_replace(['%', '?'], ['%%', "'%s'"], $temp_table_data->toSql());
    $fullSql = vsprintf($sql, $temp_table_data->getBindings());
    print_r($fullSql);

This is the result;

select * from `selected_postcodes_1434967426_1`

This doesn't look like the right query, I can't work out what Eloquent is doing here. I don't know why the full query is not printing out.

Answer 1

if you have too many result like 1 million, then use offset limit concept. Then it will save you'r time of the query. Also make sure in you select query you are filtering required fields only.( avoid select * from XXXX. use select A, B from XXX).