![](/img/trans.png)
[英]In a Foundry Code Repository, how do I log debug messages from within a Python transform?
[英]How can I use sqlContext (to execute SQL queries) in the python transform environment in Foundry Code Authoring?
我在创作中做了以下事情
Output(test_dataset_path),
df=Input(og_dataset_path)
)
def compute(ctx, df):
ctx.spark_session.sql(f'''
CREATE TABLE `test_dataset_path` AS
SELECT * FROM `og_dataset_path`
''')
return ctx.spark_session.sql(f'''
SELECT * FROM `og_dataset_path`
''')
它在代码上出错了:
ctx.spark_session.sql(f'''
CREATE TABLE `test_dataset_path` AS
SELECT * FROM `og_dataset_path`
''')
出现错误: pyspar.sql.utils.AnanlysisException: Table or view not found: og_dataset_path
如何解决此错误?
使用 createOrReplaceTempView 应该可以解决这个问题:
from transforms.api import transform_df, Input, Output
@transform_df(
Output("/Users/XXXXX/sqlcsvA2"),
ALL=Input("/datasources/locations/data/cleaned")
)
def my_compute_function(ctx, ALL):
ALL.createOrReplaceTempView('ALL')
return ctx.spark_session.sql('select * from ALL limit 10')
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.