I have done the following in an authoring
Output(test_dataset_path),
df=Input(og_dataset_path)
)
def compute(ctx, df):
ctx.spark_session.sql(f'''
CREATE TABLE `test_dataset_path` AS
SELECT * FROM `og_dataset_path`
''')
return ctx.spark_session.sql(f'''
SELECT * FROM `og_dataset_path`
''')
and it is erroring out on the code:
ctx.spark_session.sql(f'''
CREATE TABLE `test_dataset_path` AS
SELECT * FROM `og_dataset_path`
''')
with the error: pyspar.sql.utils.AnanlysisException: Table or view not found: og_dataset_path
How can I resolve this error?
Using createOrReplaceTempView should resolve this problem:
from transforms.api import transform_df, Input, Output
@transform_df(
Output("/Users/XXXXX/sqlcsvA2"),
ALL=Input("/datasources/locations/data/cleaned")
)
def my_compute_function(ctx, ALL):
ALL.createOrReplaceTempView('ALL')
return ctx.spark_session.sql('select * from ALL limit 10')
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.