简体   繁体   English

在另一个数据帧中使用一个数据帧的列

[英]Using the columns of one dataframe in another dataframe

I am trying to read the data from one dataframe and use it in another.我正在尝试从一个数据框中读取数据并在另一个数据框中使用它。 How can I do it gracefully?我怎样才能优雅地做到这一点?

val query = s"select distinct p_id, lower(regexp_replace(p_id,'[^a-zA-Z0-9]+','_')) as p_id_formatted, lower(regexp_extract(f_id,'^([^\\.]+)\\.?',1)) as f_id_formatted, column_name from default.rc_pcoders"
val run_query = sql(query)
val table_name = run_query.select(concat(lit("nepp"), lit("_"),$"p_id_formatted", lit("_") ,$"f_id_formatted ").alias("tablename"),$"column_name")

This gives me below output, which essentially represents a tablename这给了我下面的输出,它基本上代表一个表名

+------------------+-----------+
|tablename         |column_name|
+------------------+-----------+
|nepp_148hl16011_cm|cmtrt      |
|nepp_148hl16011_mh|mhaspe     |
|nepp_148hl16011_ae|aeputt     |
+------------------+-----------+

How can I get the column names from each of these tables?如何从每个表中获取列名? Something like (below query doesn't work though)类似的东西(虽然下面的查询不起作用)

val whole_query = sql("show columns in "table_name.tablename"")

First, collect all the names of the tables to load:首先,收集要加载的所有表的名称:

val tableNames = df.collect().map(row => row.getAs[String]("tablename")).toSeq

Second, get the references to the respective DataFrames, associate them with their column names其次,获取对相应 DataFrame 的引用,将它们与它们的列名相关联

val sqlCtx: SQLContext = // your SQL context ref
val dfToColumns = tableNames.map(table => {
  val columnNames = sqlCtx.table(table).schema.fieldNames.toSeq
  (table, columnNames)
}).toMap

dfToColumns is a Map[String, Seq[String]] with DataFrame names as keys and Seqs of their respective column names as values. dfToColumns是一个Map[String, Seq[String]]以 DataFrame 名称作为键,将它们各自列名称的 Seqs 作为值。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM