繁体   English   中英

如何在Apache Spark Java中将数据集的数组类型转换为字符串类型

[英]How to convert array type of dataset into string type in Apache Spark Java

我的数据集中有一个数组类型需要转换为字符串类型。 我以常规方式尝试过。 我觉得我们可以做得更好。 你能指导我吗? 输入数据集1

    +---------------------------+-----------+-------------------------------------------------------------------------------------------------+
    ManufacturerSource         |upcSource  |productDescriptionSource                                                                          |                                                                                                                                                                            |
    +---------------------------+-----------+-------------------------------------------------------------------------------------------------+
    |3M                         |51115665883|[c, gdg, whl, t27, 5, x, 1, 4, x, 7, 8, grindig, flap, wheels, 36, grit, 12, 250, rpm]           |                                                                                                                                                                            |
    |3M                         |51115665937|[c, gdg, whl, t27, q, c, 6, x, 1, 4, x, 5, 8, 11, grinding, flap, wheels, 36, grit, 10, 200, rpm]|                                                                                                                                                                             |
    |3M                         |0          |[3mite, rb, cloth, 3, x, 2, wd]                                                                  |                                                                                                                                                                             |
    |3M                         |0          |[trizact, disc, cloth, 237aaa16x5, hole]                                                         |                                                                                                                                                                             |
    -------------------------------------------------------------------------------------------------------------------------------------------

预期输出数据集

     +---------------------------+-----------+---------------------------------------------------------------------------------------------------|
     |ManufacturerSource         |upcSource  |productDescriptionSource                                                                           |                                                                                                                                                                           |
     +---------------------------+-----------+---------------------------------------------------------------------------------------------------|
     |3M                         |51115665883|c gdg whl t27 5 x 1 4 x 7 8 grinding flap wheels 36 grit 12 250 rpm               |                |                                                                                                                                                         |
     |3M                         |51115665937|c gdg whl t27 q c 6 x 1 4 x 5 8 11 grinding flap wheels 36 grit 10 200 rpm                         |                                                                                                                                                                        |
     |3M                         |0          |3mite  rb  cloth  3  x  2  wd                                                                      |                                                                                                                                                                          |
     |3M                         |0          |trizact  disc  cloth  237aaa16x5  hole                                                             |                                                                                                                                                                          |
     +-------------------------------------------------------------------------------------------------------------------------------------------|

常规方法1

        Dataset<Row> afterstopwordsRemoved = 
         stopwordsRemoved.select("productDescriptionSource");
          stopwordsRemoved.show();

        List<Row> individaulRows= afterstopwordsRemoved.collectAsList();

        System.out.println("After flatmap\n");
        List<String> temp;
        for(Row individaulRow:individaulRows){
         temp=individaulRow.getList(0);
        System.out.println(String.join(" ",temp));
        }

方法2(未产生结果)

异常:无法执行用户定义的函数($ anonfun $ 27 :(数组)=>字符串)

       UDF1 untoken = new UDF1<String,String[]>() {
        public String call(String[] token) throws Exception {
            //return types.replaceAll("[^a-zA-Z0-9\\s+]", "");
             return Arrays.toString(token); 
        }

        @Override
        public String[] call(String t1) throws Exception {
            // TODO Auto-generated method stub
            return null;
        }
    };

    sqlContext.udf().register("unTokenize", untoken, DataTypes.StringType);

    source.createOrReplaceTempView("DataSetOfTokenize");
    Dataset<Row> newDF = sqlContext.sql("select *,unTokenize(productDescriptionSource)FROM DataSetOfTokenize");
    newDF.show(4000,false);

我会用concat_ws

sqlContext.sql("select *, concat_ws(' ', productDescriptionSource) FROM DataSetOfTokenize");

要么:

import static org.apache.spark.sql.functions.*;

df.withColumn("foo" ,concat_ws(" ", col("productDescriptionSource")));

暂无
暂无

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM