在 Python 中使用 FP-Growth 算法确定最频繁模式

Question

I have used FP-Growth algorithm in python using the mlxtend.frequent_patterns fpgrowth library. I have followed the code that was mentioned in their page and I have generated the rules which I feel are recursive. I have formed a dataframe using those rules. Now I am trying to calculate support and lift using loops but it is taking a lot of time, which I am finding inefficient.

The code I have used are as follows:

records = []
for i in range(0, 13748):
    records.append([str(df.values[i,j]) for j in range(0, 12)])

patterns = pyfpgrowth. find_frequent_patterns(records, 10)

rules = pyfpgrowth. generate_association_rules(patterns,0.8)


def support_count(rhs):
    count=0
    rhs=set(rhs)
    for j in data_item['Items']:
        j=set(j)
        if(rhs.issubset(j)):
            count=count+1
    return count


rhs_support=[]
for i in df_r['Consequent']:
    a=support_count(i)
    rhs_support.append(a/len(data_item))

Is there any other easier way to calculate support and lift using FPGrowth?

Answer 1

These calculations require a lot of computation and can be slow on large data sets. One of the best ways to solve for this is to simply run as many of these calculations in parallel as you can. Your local machine may not be sufficient enough to provide the speed you are looking for.

If you have access to cloud computing, I would recommend using pySpark to achieve your goals. The SparkML library has FPGrowth built in and I have used it to build a production recommendation system that processes millions of transactions with about half a million products and the entire process takes about 20 minutes including all of the metrics you are asking for. This is of course using a rather large cluster, with about 200 cores total, so your own performance is going to be proportional to the amount of compute you are willing to pay for.

In any case, if you have never tried before, I'd recommend looking into DataBricks on the Azure platform. You can get a free trial and the code to implement FPGrowth is very simple.

FPGrowth in SparkML

DataBricks