如何在 SparklyR 中从我的 model 中提取 feature_importances？

Question

I would like to extract feature_importances from my model in SparklyR. So far I have the following reproducible code that is working:

library(sparklyr)
library(dplyr)

sc <- spark_connect(method = "databricks")

dtrain <- data_frame(text = c("Chinese Beijing Chinese",
                              "Chinese Chinese Shanghai",
                              "Chinese Macao",
                              "Tokyo Japan Chinese"),
                     doc_id = 1:4,
                     class = c(1, 1, 1, 0))

dtrain_spark <- copy_to(sc, dtrain, overwrite = TRUE)

pipeline <- ml_pipeline(
  ft_tokenizer(sc, input_col = "text", output_col = "tokens"),
  ft_count_vectorizer(sc, input_col = 'tokens', output_col = 'myvocab'),
  ml_decision_tree_classifier(sc, label_col = "class", 
                 features_col = "myvocab", 
                 prediction_col = "pcol",
                 probability_col = "prcol", 
                 raw_prediction_col = "rpcol")
)

model <- ml_fit(pipeline, dtrain_spark)

When I try to run the ml_stage step below, I find that I cannot extract a vector of feature_importances , but rather it is a function. A prior post ( how to extract the feature importances in Sparklyr? ) displays it as a vector which I would like to obtain. What could be my error here? Is there another step I need to take to unwrap the function and get a vector of values here?

ml_stage(model, 3)$feature_importances

Here is what my output to the ml_stage looks like (instead of a vector of values):

function (...) 
{
    tryCatch(.f(...), error = function(e) {
        if (!quiet) 
            message("Error: ", e$message)
        otherwise
    }, interrupt = function(e) {
        stop("Terminated by user", call. = FALSE)
    })
}
<bytecode: 0x559a0d438278>
<environment: 0x559a0ce8e840>

Answer 1

I am not sure if this is what you want, but could combine the vectorizer model and vocaculary to extract the feature_importances of your model which will results in a table with the importances of your text. You could use the following code:

library(sparklyr)
library(dplyr)

sc <- spark_connect(method = "databricks")

dtrain <- data_frame(text = c("Chinese Beijing Chinese",
                              "Chinese Chinese Shanghai",
                              "Chinese Macao",
                              "Tokyo Japan Chinese"),
                     doc_id = 1:4,
                     class = c(1, 1, 1, 0))

dtrain_spark <- copy_to(sc, dtrain, overwrite = TRUE)

pipeline <- ml_pipeline(
  ft_tokenizer(sc, input_col = "text", output_col = "tokens"),
  ft_count_vectorizer(sc, input_col = 'tokens', output_col = 'myvocab'),
  ml_decision_tree_classifier(sc, label_col = "class", 
                              features_col = "myvocab", 
                              prediction_col = "pcol",
                              probability_col = "prcol", 
                              raw_prediction_col = "rpcol")
)

model <- ml_fit(pipeline, dtrain_spark)

tibble(
  token = unlist(ml_stage(model, 'count_vectorizer')$vocabulary),
  importance = ml_stage(model, 'decision_tree_classifier')$feature_importances
)