Query crashes MS Access

Question

THE TASK: I am in the process of migrating a DB from MS Access to Maximizer. In order to do this I must take 64 tables in MS ACCESS and merge them into one. The output must be in the form of a TAB or CSV file. Which will then be imported into Maximizer.

THE PROBLEM: Access is unable to perform a query that is so complex it seems, as it crashes any time I run the query.

ALTERNATIVES: I have thought about a few alternatives, and would like to do the least time-consuming one, out of these, while also taking advantage of any opportunities to learn something new.

1. Export each table into CSVs and import into SQLight and then make a query with it to do the same as what ACCESS fails to do (merge 64 tables).
2. Export each table into CSVs and write a script to access each one and merge the CSVs into a single CSV.
3. Somehow connect to the MS ACCESS DB (API), and write a script to pull data from each table and merge them into a CSV file.

QUESTION: What do you recommend?

CLARIFICATIONS:

1. I am merging tables, not concatenating. Each table has a different structure and different data. It is a normalized CRM database. Companies->contacts->details = ~ 60 tables of details.
2. As the Access db will be scuttled after the db is migrated, I want to spend as little time in Access as possible.

Answer 1

I agree with FrustratedWithFormsDesigner. #2 seems the simplest method.

Here is some tested code if you decide to go that route (requires pyodbc ):

import csv
import pyodbc

MDB = 'c:/path/to/my.mdb'
DRV = '{Microsoft Access Driver (*.mdb)}'
PWD = 'mypassword'

conn = pyodbc.connect('DRIVER=%s;DBQ=%s;PWD=%s' % (DRV,MDB,PWD))
curs = conn.cursor()

SQL = 'SELECT * FROM mytable;' # insert your query here
curs.execute(SQL)

rows = curs.fetchall()

curs.close()
conn.close()

# you could change the 'w' to 'a' for subsequent queries
csv_writer = csv.writer(open('mytable.csv', 'w'), lineterminator='\n')

for row in rows:
    csv_writer.writerow(row)

Answer 2

Since you want to merge 64 tables, may we assume those tables all have the same structure?

If so, create a new empty table with matching structure, then append the rows from each of those 64 tables into the new merge master table. Then export the merge master table as a single CSV file.

The merge operation should not have to be a single complex query.

INSERT INTO tblMergeMaster(
    some_field,
    another_field,
    yet_another)
SELECT
    some_field,
    another_field,
    yet_another
FROM
    tbl_1_of_64;

You can build the INSERT statement 64 times with VBA code, with a different FROM table each time. And execute each statement with CurrentDb.Execute

Answer 3

I would recommend #2 if the merge is fairly simple and straightforward, and doesn't need the power of an RDBMS. I'd go with #1 if the merge is more complex and you will need to write some actual queries to get the data merged properly.

Answer 4

I'm not even clear on what you're trying to do. I assume your problem is that Jet/ACE can't handle a UNION with that many SELECT statements.

If you have 64 identically-structured tables and you want them in a single CSV, I'd create a temp table in Access, append each table in turn, then export from the temp table to CSV. This is a simple solution and shouldn't be slow, either. The only possible issue might be if there are dupes, but if there are, you can export from a SELECT DISTINCT saved QueryDef.

Tangentially, I'm surprised Maximizer still exists. I had a client who used to use it, and the db structure was terribly unnormalized, just like all the other sales software like ACT.