使用条件根据分区的镶木地板数据创建表

Question

I am trying to create a table from partitioned data from amazon s3 in databricks cluster . Now the data I have is partitioned on the following

ID , report and date

So I have mounted the data:

%python
ACCESS_KEY = "xxxxxxxxx"
SecretKey = "xxxxxxxxxx"
ENCODED_SECRET_KEY = SecretKey.replace("/", "%2F")
AWS_BUCKET_NAME = "path/parent_directory"
MOUNT_NAME = "parent"
dbutils.fs.mount("s3a://%s:%s@%s" % (ACCESS_KEY, ENCODED_SECRET_KEY, 
AWS_BUCKET_NAME), "/mnt/%s" % MOUNT_NAME)

Now as per the structure of my data's path would be something like this:

/dbfs/parent/id/report/date

Now, I want to create table based on the partition. I want to specify a where condition in the create table where the report_name is specified in condition. There are 5 reports inside the id folder. My query is something like this:

%sql
Create table if not exists abc
(col1 string,
 col2 string,
 col3 bigint)using parquet
OPTIONS (path "/mnt/parent/")
partitioned by (id,report,date) where 
report="report1" ;

I am getting syntax error :

Error in SQL statement: ParseException:mismatched input 'where' expecting <EOF>

I also tried

Create table if not exists report1
(
col1 string,
col2 string,
col3 bigint  )using parquet
OPTIONS (path "/mnt/parent/")
partitioned by (id,report="report1",date)

Can anyone help me with this? Or anyone can help me loading through spark-shell?

Thanks

Answer 1

I think what you really want is an unmanaged table over the data and a view that filters by that partition condition.

create table report
using parquet
options (
  path '/mnt/parent'
);

msck repair table report;

create or replace view report1
as select * from report where report = 'report1';