I am trying to synchronise a Sage Line 50 Access database with Mysql via ODBC

Question

I am trying to source the structure and data from a sage line 50 database but am having trouble with my update/create script.

Basically, I am writing a tool so that the local intranet site can display data sourced from sage to normal employees without using up a sage login (orders due in/stock levels etc). I am having trouble with this because it seems that the Sage50 database was developed by total morons. There are no Unique keys in this database, or, more accurately, very very few. The structure is really old school you can find the structure on pastebin HERE (bit too big for here). You'll notice that there are some tables with 300+ columns, which I think is a bit stupid. However, I have to work with this and so I need a solution.

There are a few problems syncing that I have encountered. Primarily it's the fact ODBC can't limit statements to 1 row so I can check data type, and secondly, with there being no IDs, I can't check if it's a duplicate when doing the insert. At the moment, this is what I have:

$rConn = odbc_connect("SageLine50", "user", "password");

if ($rConn == 0) {
    die('Unable to connect to the Sage Line 50 V12 ODBC datasource.');
}

$result = odbc_tables($rConn);

$tables = array();

while (odbc_fetch_row($result)){
    if(odbc_result($result,"TABLE_TYPE")=="TABLE") {
        $tables[odbc_result($result,"TABLE_NAME")] = array();
    }
}

This produces the first level of the list you see on pastebin.

A foreach statement is then run to produce the next level with the columns within the table

foreach($tables as $k=> $v) {
    $query = "SELECT * FROM ".$k;
    $rRes = odbc_exec($rConn,$query);
    $rFields = odbc_num_fields ($rRes);
    $i = 1;
    while($i <= $rFields) {
        $tables[$k][] = odbc_field_name($rRes, $i);
        $i++;   
    }
    CreateTableandRows($k,$tables[$k]);
}

At the moment, I then have a bodged together function to create each table (not that I like the way it does it).

Because I can't automatically grab back one row (or a few rows), to check the type of data with get_type() to then automatically set the row type, it means the only way I can figure out to do this is to set the row type as text and then change them retrospectively based upon a Mysql query.

Here is the function that's called for the table creation after the foreach above.

function CreateTableandRows($table,$rows) {
    $db = array(
        "host" => '10.0.0.200',
        "user" => 'user',
        "pass" => 'password',
        "table" => 'ccl_sagedata'
    );

$DB = new Database($db);

    $LocSQL = "CREATE TABLE IF NOT EXISTS `".$table."` (
            `id` int(11) unsigned NOT NULL auto_increment,
            PRIMARY KEY  (`id`),";  

    foreach($rows as $k=>$v) {
            $LocSQL .= "
            ".$v." TEXT NOT NULL default '',";
    }

    $LocSQL = rtrim($LocSQL, ',');

    $LocSQL .= "
            ) ENGINE=MyISAM  DEFAULT CHARSET=utf8";

    echo '<pre>'.$LocSQL.'</pre>';

    $DB->query($LocSQL);

}

I then need/want a function that takes each table at a time and synchronizes the data to the ccl_sagedata database. However, it needs to make sure it's not inserting duplicates, ie this script will be run to sync the sage database at the start or end of each day and without ID numbers INSERT REPLACE won't work. I am obviously implementing auto inc primary ID's for each new table in the ccl_sagedata db. But I needs to be able to reference something static in each table that I can identify through ODBC (I hope that makes sense). In my current function, it has to call the mysql database for each row on the sage database and see if there is a matching row.

function InsertDataFromSage($ODBCTable) {
    $rConn = odbc_connect("SageLine50", "user", "pass");
    $query = "SELECT * FROM ".$ODBCTable;
    $rRes = odbc_exec($rConn,$query);
    $rFields = odbc_num_fields ($rRes);
    while( $row = odbc_fetch_array($rRes) ) {
        $result[] = $row;
    } 
    $DB = new Database($db);
    foreach($result as $k => $v) {
        $CHECKEXISTS = "SELECT * FROM ".$ODBCTable." WHERE";
        $DB->query($CHECKEXISTS);
        // HERE WOULD BE A PART THAT PUTS DATA INTO THE DATABASE IF IT DOESN'T ALREADY EXIST
    }
}

The only other thing I can think to note is that the 'new Database' class is simply just a functionalised standard mysqli database class. It's not something I'm having problems with.

So to re-cap.

1. I am trying to create a synchronization script that creates (if not exists) tables within a mysql database and then imports/syncs the data.
2. ODBC Can't limit the output so I can't figure out the data types in the columns automatically (can't do it manually because it's a massive db with 80+ tables
3. I can't figure out how to stop the script creating duplicates because there are no IDs in the sage source database.
4. For those of you not in the UK, Sage is a useless accounting package that runs on water and coal.
5. The Sage database only provides data, it doesn't allow you to input data outside of csv files in the actual program.

Answer 1

I know this is a bit late but Im already doing the same thing but with MS SQL.

Ive used a DTS package that truncates known copies of the tables (ie AUDIT_JOURNAL ) and then copies everything in daily.

I also hit a bit of a wall trying to handle updates of these tables hence the truncate and re-create. Sync time is seconds so its not a bad option. It may be a bit of a ball ache but I say design your sync tables manually.

As you rightly point out, sage is not very friendly to being poked, so id say don't try to sync it all either.

Presumably you'll need reports to present to users but you don't need that much to do this. I sync COMPANY,AUDIT_JOURNAL, AUDIT_USAGE, CAT_TITLE,CAT_TITLE_CUS, CHART_LIST,CHART_LIST_CUS, BANK,CATEGORY,CATEGORY_CUS,DEPARTMENT, NOMINAL_LEDGER,PURCHASE_LEDGER,SALES_LEDGER .

This allows recreation of all the main reports (balance sheet, trial balance, supplier balances, etc all with drill down). If you need more help this late on let me know. I have a web app called MIS that you could install locally but the sync is a combo of ODBC and the DTS.

Answer 2

OK you do not need to create a synchronisation script you can query ODBC in real time you can even do joins like you do in SQL to retrieve data from multiple tables. The only thing you cannot do is write data back to sage.