[英]Table Partitioning in Azure SQL DW based on Cust_Id/Tenant_Id (Multi-tenancy)?
我是MS BI堆棧的新手,嘗試在SQL DW表字段(即cust_id)上創建分區,該分區對於每個客戶(tenant_id)都是唯一的,並且在每個表中都存在,以便每個客戶的數據保留在各自的分區中,而不會影響其他分區。
下面是分區的表結構,數據和輸出:
create table emp
(
cust_id integer not null,
emp_id varchar(5) not null,
emp_name varchar(10) not null
)
with
(
clustered columnstore index
,distribution = round_robin
,partition (cust_id range right for values (100,200,300) )
)
create table dept
(
cust_id integer not null,
dept_id varchar(5) not null,
emp_id varchar(5) not null,
dep_name varchar(10) not null
)
with
(
clustered columnstore index
,distribution = round_robin
,partition (cust_id range right for values (100,200,300) )
)
create statistics emp_stats on dbo.emp(cust_id)
create statistics dept_stats on dbo.dept(cust_id)
emp table:
101 EMP01 XYZ
101 EMP02 ABC
101 EMP03 DEF
201 EE001 JACK
201 EE002 MIKE
dept table:
cust_id dept_id emp_id dep_name
101 D0001 EMP01 IT
101 D0001 EMP02 IT
201 DEP01 EE001 ENG
201 DEP02 EE002 HR
SELECT sch.name AS [schema_name],
tbl.[name] AS [table_name],
ds.type_desc,
prt.[partition_number],
rng.[value] AS [current_partition_range_boundary_value],
prt.[rows] AS [partition_rows]
FROM sys.schemas sch
INNER JOIN sys.tables tbl ON sch.schema_id = tbl.schema_id
INNER JOIN sys.partitions prt ON prt.[object_id] = tbl.[object_id]
INNER JOIN sys.indexes idx ON prt.[object_id] = idx.[object_id] AND prt.[index_id] = idx.[index_id]
INNER JOIN sys.data_spaces ds ON idx.[data_space_id] = ds.[data_space_id]
INNER JOIN sys.partition_schemes ps ON ds.[data_space_id] = ps.[data_space_id]
INNER JOIN sys.partition_functions pf ON ps.[function_id] = pf.[function_id]
LEFT JOIN sys.partition_range_values rng ON pf.[function_id] = rng.[function_id] AND rng.[boundary_id] = prt.[partition_number]
WHERE tbl.name in ('emp','dept')
order by table_name, partition_number
schema_name table_name type_desc partition_number current_partition_range_boundary_value partition_rows
dbo dept PARTITION_SCHEME 1 100 1
dbo dept PARTITION_SCHEME 2 200 1
dbo dept PARTITION_SCHEME 3 300 1
dbo dept PARTITION_SCHEME 4 NULL 1
dbo emp PARTITION_SCHEME 1 100 1
dbo emp PARTITION_SCHEME 2 200 1
dbo emp PARTITION_SCHEME 3 300 1
dbo emp PARTITION_SCHEME 4 NULL 2
問題/澄清:
1) Whether the partition created on cust_id (tenant_id) field along with round_robin distribution method correct? What is the right way to do it? Need to segregate the customer specific data for both performance (load + query) & security reasons.
2) How can we load specific customer data into their respective partition (cust_id) – syntax in SQL DW?
insert into emp (partition = <partition_name_number> ) ?
3) How do I verify that the data is getting loaded into correct respective partition as I am unable to understand the output from above query as to how it is showing 4 partitions and only 1 row for cust_id 101 in emp table when actually there are 3 ? Was expecting that since 101 is between 100 and 200, it should be in partition_number = 1 and 201 which is between 200 and 300 in partition_number = 2? Is this a wrong assumption on how partition range works? Can’t we simply have a List partition created in SQL DW for each cust_id ?
4) As per MS documentation, it by default divides each table into 60 distributed databases and that there should be at least 1M per distribution for a partition. When you don’t know how many customer’s data and their volume you may have in future, how do we approach towards it ?
5) When creating semantic layer (Analysis Services-SSAS) on top of DW, is it helpful to do further do partition on tenant_id or some other field?
感謝您的幫助,意見和建議!
分區在Azure SQL數據倉庫上可用,但是文檔隨附警告,可能會適得其反。 您的示例可能是一個很好的例子。
您應該查看分布而不是分區,以確保您的數據在節點之間達到最佳對齊。
除非有強烈的理由不能使用哈希或復制,否則不要使用round_robin分發。 輪詢將快速加載,但是對該表的后續查詢將很慢。
您是否正在實施尺寸模型? 您有幾個客戶? 一般指導原則是復制維度,除非它們很大(1B +行),在這種情況下,它們可能是哈希分配策略的驅動力。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.