简体   繁体   English

如何使用startrow和stoprow不会导致HBase中的全表扫描?

[英]How does the use of startrow and stoprow not result in a full table scan in HBase?

It is commonly suggested to use a range scan via startrow and stoprow as opposed to a Rowkey Prefix Filter (for example, here ). 通常建议通过startrowstoprow使用范围扫描,而不是使用Rowkey Prefix Filter (例如, 此处 )。 The reasoning for this is because a Rowkey Prefix Filter results in a full table scan of the rowkey, whereas a range scan via startrow and stoprow do not result in a full table scan. 原因是因为Rowkey Prefix Filter导致对rowkey进行全表扫描,而通过startrowstoprow进行的范围扫描不会导致全表扫描。 Why doesn't it? 为什么不呢? Most people say "because the rowkey is stored in lexographical order," which of course, doesn't explain why the Rowkey Prefix Filter cannot leverage this. 大多数人说“因为rowkey按字典顺序存储”,这当然不能解释为什么Rowkey Prefix Filter无法利用它。

At anyrate, how exactly does a range scan via startrow and stoprow not result in a full table scan of the rowkey? 无论如何,通过startrowstoprow进行范围扫描的startrow stoprow不会导致对stoprow进行全表扫描?

Take this small example in python to show why I don't understand how the lexagraphical ordering of the rowkeys means anything when it comes to avoiding a full table scan: 以python中的这个小例子来说明为什么我不理解rowkeys的lexagraphical排序在避免全表扫描方面意味着什么:

rowkeys = ['a1', 'a2', 'a3', 'b1', 'b2', 'b3', 'c1', 'c2', 'c3']

def range_scan(startrow, stoprow):
    is_found = False
    for rowkey in rowkeys:
        if startrow <= rowkey < stoprow:
            is_found = True
            yield rowkey
        else:
            if is_found:
                raise StopIteration()

Clearly, the HBase algorithm differs from this. 显然,HBase算法与此不同。 How does it? 怎么做的?

TLDR: How exactly does HBase avoid a full table scan when doing a range scan with startrow and stoprow? TLDR:在使用startrow和stoprow进行范围扫描时,HBase究竟是如何避免全表扫描的?

In HBase a table is split into regions. 在HBase中,表格被分成区域。 A region is a set of rows that is served by a specific region server. 区域是由特定区域服务器提供服务的一组行。 A region server has (in general) multiple regions from multiple tables for which it handles the requests. 区域服务器(通常)具有来自多个表的多个区域,它们处理请求。

Because the rows are sorted by key it is known in the hbase master which start and stop keys are the boundaries of a each region AND on which region server this region can be queried. 因为行是按键排序的,所以在hbase master中已知启动和停止键是每个区域的边界,以及可以在哪个区域服务器上查询该区域。

So if a scan is done that uses an explicit start and stop row then the query is directed ONLY to the regions/region servers that have keys that may be in the requested range. 因此,如果完成使用显式启动和停止行的扫描,则查询仅定向到具有可能在请求范围内的键的区域/区域服务器。

If the query has a 'small' range of keys then you'll find that in almost all queries only a single region is accessed. 如果查询具有“小”键范围,那么您将发现在几乎所有查询中只访问单个区域。

It is common to have dozens of regions for a HBase table and by using the start and stop row to limit the scan you may find that 99 of the 100 regions of your table do not even know a query on the table has been done. 通常有一个HBase表的几十个区域,并且通过使用开始和停止行限制扫描,您可能会发现表的100个区域中的99个甚至不知道表上的查询已经完成。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM