在python中快速訪問/查詢大分隔文本文件

Question

在搜索了一段時間后，我發現了許多與此問題相關的問題/答案，但沒有真正解決我正在尋找的問題。 基本上，我在python中實現代碼，以便能夠從星型目錄（特別是tycho 2星級目錄）中查詢信息。

該數據存儲在較大的（~0.5千兆字節）文本文件中，其中每行對應於星形的條目。

一些示例行是

0001 00008 1| |  2.31750494|  2.23184345|  -16.3|   -9.0| 68| 73| 1.7| 1.8|1958.89|1951.94| 4|1.0|1.0|0.9|1.0|12.146|0.158|12.146|0.223|999| |         |  2.31754222|  2.23186444|1.67|1.54| 88.0|100.8| |-0.2
0001 00013 1| |  1.12558209|  2.26739400|   27.7|   -0.5|  9| 12| 1.2| 1.2|1990.76|1989.25| 8|1.0|0.8|1.0|0.7|10.488|0.038| 8.670|0.015|999|T|         |  1.12551889|  2.26739556|1.81|1.52|  9.3| 12.7| |-0.2
0001 00016 1| |  1.05686490|  1.89782870|  -25.9|  -44.4| 85| 99| 2.1| 2.4|1959.29|1945.16| 3|0.4|0.5|0.4|0.5|12.921|0.335|12.100|0.243|999| |         |  1.05692417|  1.89793306|1.81|1.54|108.5|150.2| |-0.1
0001 00017 1|P|  0.05059802|  1.77144349|   32.1|  -20.7| 21| 31| 1.6| 1.6|1989.29|1985.38| 5|1.4|0.6|1.4|0.6|11.318|0.070|10.521|0.051| 18|T|         |  0.05086583|  1.77151389|1.78|1.55| 30.0| 45.6|D|-0.2

信息是分隔的和固定的寬度。 每列包含有關恆星的不同信息。 現在，對於我的python實用程序，我希望能夠快速搜索此信息並檢索符合用戶指定的一組條件的星星條目。

例如，我希望能夠找到亮度大於5.5（第18或第19列）的所有恆星，它們在0到30度（第3列）之間有一個右上升，在-45到-35度之間有一個偏角（col 4）有效率。 現在，如果我可以將所有這些信息存儲在內存中，那么很容易將文件讀入一個numpy結構化數組或pandas數據幀，並使用邏輯索引檢索我想要的星星。 不幸的是，我正在處理的機器沒有足夠的內存來執行此操作（在任何給定時間我只有大約0.5千兆字節的內存空閑，而我正在使用的其余程序占用了大量內存）。

我當前的解決方案涉及遍歷文本文件的每一行，解釋數據，並且僅當條目符合指定的條件時才將條目存儲在內存中。 我必須這樣做的方法是

def getallwithcriteria(self, min_vmag=1., max_vmag=17., min_bmag=1., max_bmag=17., min_ra=0., max_ra=360.,
                       min_dc=-90., max_dc=90., min_prox=3, search_center=None, search_radius=None):
    """
    This method returns entire star records for each star that meets the specified criterion.  The defaults for each
    criteria specify the entire range of the catalogue.  Do not call this without changing the defaults as this will
    likely overflow memory and cause your system to drastically slow down or crash!

    Note that all of the keyword argument do not need to be specified.  For instance, we could run

        import tychopy as tp

        tyc = tp.Tycho('/path/to/catalogue')

        star_records = tyc.getallwithcritera(min_vmag=3,max_vmag=4)

    to return all stars that have a visual magnitude between 3 and 4.

    This method returns a numpy structured array where each element contains the complete record for a star that
    matches the criterion specified by the user.  The output array has the following dtype:

            [('starid', 'U12'),
             ('pflag', 'U1'),
             ('starBearing', [('rightAscension', float), ('declination', float)]),
             ('properMotion', [('rightAscension', float), ('declination', float)]),
             ('uncertainty', [('rightAscension', int), ('declination', int), ('pmRA', float), ('pmDc', float)]),
             ('meanEpoch', [('rightAscension', float), ('declination', float)]),
             ('numPos', int),
             ('fitGoodness', [('rightAscension', float), ('declination', float), ('pmRA', float), ('pmDc', float)]),
             ('magnitude', [('BT', [('mag', float), ('err', float)]), ('VT', [('mag', float), ('err', float)])]),
             ('starProximity', int),
             ('tycho1flag', 'U1'),
             ('hipparcosNumber', 'U9'),
             ('observedPos', [('rightAscension', float), ('declination', float)]),
             ('observedEpoch', [('rightAscension', float), ('declination', float)]),
             ('observedError', [('rightAscension', float), ('declination', float)]),
             ('solutionType', 'U1'),
             ('correlation', float)]

    see the readme of the Tycho 2 catalogue for a more formal description of each field.

    If no stars are found that match the specified input then an empty numpy array with the above dtype is returned.

    Note that both a rectangular and a circular area can be specified.  The rectangular search area is specified
    using the min_ra/dc max_ra/dc keyword arguments while the circular search area is specified using the
    search_center and search_radius keyword arguments where the search_center is a tuple, list, numpy array, or
    other array like object which contains the center right ascension in element 0 and the center declination in
    element 1.  It is not recommended to specify both the circular and rectangular search areas.  If the search
    areas do not overlap then no stars will be returned.

    :param min_vmag:  the minimum (brightest) visual magnitude to return
    :param max_vmag:  the maximum (dimmest) visual magnitude to return
    :param min_bmag:  the minimum (brightest) blue magnitude to return
    :param max_bmag:  the maximum (dimmest) blue magnitude to return
    :param min_ra:  the minimum right ascension to return
    :param max_ra:  the maximum right ascension to return
    :param min_dc:  the minimum declination to return
    :param max_dc:  the maximum declination to return
    :param min_prox:  the closest proximity to a star to return
    :param search_center: An array like object containing the center point from which to search radially for stars.
    :param search_radius: A float specifying the radial search distance to use
    :return: A numpy structure array containing the star records for stars that meet the specified criteria
    """

    # form the dtype list that genfromtxt will use to interpret the star records
    dform = [('starid', 'U12'),
             ('pflag', 'U1'),
             ('starBearing', [('rightAscension', float), ('declination', float)]),
             ('properMotion', [('rightAscension', float), ('declination', float)]),
             ('uncertainty', [('rightAscension', int), ('declination', int), ('pmRA', float), ('pmDc', float)]),
             ('meanEpoch', [('rightAscension', float), ('declination', float)]),
             ('numPos', int),
             ('fitGoodness', [('rightAscension', float), ('declination', float), ('pmRA', float), ('pmDc', float)]),
             ('magnitude', [('BT', [('mag', float), ('err', float)]), ('VT', [('mag', float), ('err', float)])]),
             ('starProximity', int),
             ('tycho1flag', 'U1'),
             ('hipparcosNumber', 'U9'),
             ('observedPos', [('rightAscension', float), ('declination', float)]),
             ('observedEpoch', [('rightAscension', float), ('declination', float)]),
             ('observedError', [('rightAscension', float), ('declination', float)]),
             ('solutionType', 'U1'),
             ('correlation', float)]

    # initialize a list which will contain the star record strings for stars that match the input criteria
    records = []

    # loop through each record in the Tycho2 catlogue
    for record in self._catalogueFile:

        # interpret the record as simply as we can
        split_record = record.split(sep="|")

        # check that we are examining a good star, that it falls within the bearing bounds, and that it is far
        # enough away from other stars
        if ("X" not in split_record[1]) and min_ra <= float(split_record[2]) <= max_ra \
                and min_dc <= float(split_record[3]) <= max_dc and int(split_record[21]) >= min_prox:

            # perform the radial search if the user has specified a center and radius
            if search_center is None or pow(pow(float(split_record[2])-search_center[0], 2) +
                                            pow(float(split_record[3])-search_center[1], 2), 1/2.) < search_radius:

                # Check to see if we have values for both blue and visual magnitudes, and check to see if these
                # magnitudes fall within the specified magnitude bounds
                # We need to split this up like this in order to make sure that either the bmag or the vmag exist
                if bool(split_record[17].strip()) and bool(split_record[19].strip()) \
                        and min_bmag <= float(split_record[17]) <= max_bmag \
                        and min_vmag <= float(split_record[19]) <= max_vmag:

                    records.append(record+'\n')

                # if only the visual magnitude exists then check its bounds - also check if the user has specified
                # its bounds
                elif not bool(split_record[17].strip()) and bool(split_record[19].strip()) \
                        and min_vmag <= float(split_record[19]) <= max_vmag and (max_vmag != 17. or min_vmag != 1.):

                    records.append(record+'\n')

                # if only the blue magnitude exists the check its bounds - also check if the user has specified its
                # bounds
                elif not bool(split_record[19].strip()) and bool(split_record[17].strip()) \
                        and min_bmag <= float(split_record[17]) <= max_bmag and (max_bmag != 17. or min_bmag != 1.):

                    records.append(record+'\n')

                # otherwise check to see if the use has changed the defaults.  If they haven't then store the star
                elif max_bmag == 17. and max_vmag == 17. and min_bmag == 1. and min_vmag == 1.:

                    records.append(record+'\n')

    # check to see if any stars met the criteria.  If they didn't then return an empty array.  If they did then use
    # genfromtxt to interpret the string of star records
    if not bool(records):
        nprecords = np.empty((1,), dtype=dform)

        warnings.warn('No stars were found meeting your criteria.  Please try again.')
    else:
        nprecords = np.genfromtxt(BytesIO("".join(records).encode()), dtype=dform, delimiter='|', converters={
            0: lambda s: s.strip(),
            1: lambda s: s.strip(),
            22: lambda s: s.strip(),
            23: lambda s: s.strip(),
            30: lambda s: s.strip()})

        if self._includeProperMotion:
            applypropermotion(nprecords, self.newEpoch, copy=False)

    # reset the catalogue back to the beginning for future searches
    self._catalogueFile.seek(0, os.SEEK_SET)

    return nprecords

這仍然非常慢（雖然比耗盡所有內存並將其他內容推送到交換中更快）。 為了比較，每次我需要檢索星星時需要大約2-3分鍾，我需要在我寫這個程序的過程中從這40次左右（每次都有不同的標准）檢索星星。 該計划的其余部分總共需要5秒鍾。

我現在的問題是，加速這個過程的最佳方法是什么（除了獲得更好內存的更好的計算機之外）。 我願意接受任何建議，只要它們得到很好的解釋，並且不會花費我數月的時間來實施。 我甚至願意編寫一個功能，將原始目錄文件修改為更好的格式（按特定列排序的固定寬度二進制文件），以加快速度。

到目前為止，我已經考慮過memmap'ing文件，但決定反對它，因為我真的認為它不會對我需要做的事情有所幫助。 我還考慮過從數據創建數據庫，然后使用sqlalchemy或類似的東西來查詢數據; 但是，我對數據庫並不是很熟悉，也不知道這是否會提供真正的速度提升。

Answer 1

由於@wflynny已經提到PyTables（HDF5商店） - 與text / CSV /等相比效率更高。 文件。 除此之外，你可以使用.read_hdf(where='<where condition>')從條件讀取PyTables。

您可能想要檢查此比較 。 如果您的計算機是UNIX或Linux，您可能需要檢查羽毛格式 ，這應該非常快。

除此之外，我會檢查是否使用一些RDBMS（MySQL / PostgreSQL / SQLite）加上適當的索引 - 會加快速度。 但是如果你只有0.5 GB的RAM可用並希望同時使用Pandas和RDBMS，則可能會出現問題