根据多个条件匹配记录（来自大型数据集）

Question

I have a list of bank accounts in my database. We want to try to group these accounts based on household. We match on three criteria:

• SSN
• Customer number (this is an arbitrary number from the bank)
• "Address string" (it's basically the street address plus zip)

If any one of these three things matches between two accounts, the two accounts should be put in the same group.

This can't be done with SQL joins , as far as I understand. I'm also at a loss for how to do it programmatically. We have millions of accounts in our database and the number grows by many thousands (150K-ish) each month, so it's not like it's practical to go through each and every single record and say, "Okay, do a SELECT * WHERE ssn = (this account's SSN) " because it would take forever.

I know this is kind of a vague and open-ended question, but any suggestions on how to proceed? I don't care what language(s) you use in your answer, if you use any.

Answer 1

In my honest oppinion, your best bet is to implement a one-to-many or many-to-many relationship for household-to-account.

Two ways I can think of doing something like this. The first (and probably not the best solution) is to add a column to the account table to store the household. Personally I would stay away from this if at all possible.

And second, create a "household" table to store the household PK, and a household cross reference table to store the household FK, and the account FK.

Then I would create a process with whatever programming language your using (hopeful its object oriented and you can create an "object" that you can use for the next part, and then in the future as well.

Once the database is setup, I would build a "method" that accepts an account, and does a comparison by ssn, customer number, and address and returns either a list of similar account ids (this could be very useful and might make your initial process go quicker) and/or returns a list of households that similar accounts might belong to

THIS is the part that would worry me... there maybe situations where accounts that are linked by address may or may not belong in the same household that accounts linked by account number may or maynot be in) ie. a "child" (one customer number) whos parents have separated has an account set up by each parent (two accounts with most likely different addresses), as well as their OWN accounts... and so on... i would personally come up with some sort of business logic to limit the returned household to only 1 household...

at this point, by having the list of similar accounts, and returning a single household that atleast one of the similar accounts is a part of, you can then update those specific accounts with that household id.

I would setup logic at that point to loop through every account in the table and run it through the process... yes this is going to be expensive.... but you should only have to do it once.

after that, as accounts are entered, setup a process for the entry to automatically find and place accounts into households.

Depending your front end this may or may not be a simple process.

Regardless I would also develope a process/user interface that would allow the user (prefferable a customer service rep) to remove/move accounts between households.

This is a start, just bouncing ideas off you.

Answer 2

Well, I don't see any way around having each record inspect every other record to see if it's in the same household. The only efficiency I see is that you can skip the inspection if the record already is in a household. In psuedocode

delete all record.household
currentHousehold=1

foreach record
    record.household=currentHousehold
    foreach record
        if record.household is null
            if meet criteria 1
                record.household=currentHousehold
                exit
            else if meet criteria 2
                record.household=currentHousehold
                exit
            else if meet criteria 3
                record.household=currentHousehold
                exit
            end if
        end if
    next record

    currentHousehold++
next record

The assumption is that you add a household column to the table, which you can group on. I indicate that current household values should be cleared - this is in case some of the data changes.

If you can intercept any possible change to your criteria fields then you can find that records' new household then and there. In that case, household values can stay put and the script would only have to find households for new records (or just do that when the record is added if you can). If you have that kind of control, then you should be able to put each record in a household once for existing records, when added or when criteria fields are modified.