使用 Apache spark java 从列内容中提取特定字符串

Question

I have this column from my dataframe:

+--------------------------------------------------------------------+
|          _classname                                                |
+--------------------------------------------------------------------+
|de.abs.org.vertrag.util.test.Elementofthetable                      |
|de.abs.org.vertrag.util.test.classes.Apachesparkproject             |
+--------------------------------------------------------------------+

I want to get as expected dataframe:

+--------------------------------------------------------------------+
|           _classname                                               |
+--------------------------------------------------------------------+
|de.abs.org.vertrag.util.test                                        |
|de.abs.org.vertrag.util.test.classes                                |
+--------------------------------------------------------------------+

I want to get rid of the classname of everypackage which is after the last (.) here is my code:

teamNamesDf = teamNamesDf.select(col("_classname"), substring_index(col("_classname"), ".\\[^A-Z]+", -1).as("testSuiteName"));

This didnt work for me:( does anyone knows how to do that using Apache Spark Java?

Answer 1

You can try this regex, which extracts everything before the final . :

Dataset<Row> result = df.select(regexp_extract(col("_classname"), "^(.*)\\.[^\\.]*$", 1))

Another way is to use substring_index with a variable length that depends on the number of . in the string:

Dataset<Row> result = df.selectExpr("substring_index(_classname, '.', size(split(_classname, '\\\\.')) - 1)")