堆棧內存溢出
  簡體   English   中英

具有周期性邊界條件的NumPy N維數組

[英]NumPy N-dim Array with periodic boundary conditions

 原文  2019-05-06 12:22:07  9   3    python/ numpy/ vectorization

問題描述

是否可以將具有周期性邊界條件的NumPy N維數組寫入視圖？

例如，假設我具有以下初始數組： 

import numpy as np

arr = np.arange(2 * 3).reshape((2, 3))
# [[0 1 2]
#  [3 4 5]]

就像是： 

periodic_view(array, shape, offset)

導致例如： 

new_arr = periodic_view(arr, (4, 5), (0, 0))
# [[0 1 2 0 1]
#  [3 4 5 3 4]
#  [0 1 2 0 1]
#  [3 4 5 3 4]]

new_arr = periodic_view(arr, (4, 5), (1, 1))
# [[5 3 4 5 3]
#  [2 0 1 2 0]
#  [5 3 4 5 3]
#  [2 0 1 2 0]]

對於symmetric視圖也是如此。

我知道我可以通過慢速直接循環來做到這一點，例如： 

import itertools

def periodic_view(arr, shape, offset):
    result = np.zeros(shape, dtype=arr.dtype)
    for i in itertools.product(*tuple(range(dim) for dim in result.shape)):
        slicing = tuple(
            (j - k) % dim
            for j, k, dim in zip(i, offset, arr.shape))
        result[i] = arr[slicing]
    return result

我想知道是否有辦法通過廣播/跨步機制來做到這一點。

作為獎勵，我將尋找一種可以輕松地適應對稱（而非周期性）邊界條件的解決方案，例如： 

new_arr = symmetric_view(arr, (4, 7), (1, 2))
# [[1 0 0 1 2 2 1]
#  [1 0 0 1 2 2 1]
#  [4 3 3 4 5 5 4]
#  [4 3 3 4 5 5 4]]

編輯

這類似於如何從具有周期性邊界條件的numpy數組中選擇窗口？ 除了在建議的解決方案中使用np.roll()使得此輸出具有形狀大於輸入的形狀而帶來的不便之處，看起來好像是從輸入中復制數據。

編輯2

可以使用np.pad(mode='wrap')和np.pad(mode='symmetric')獲得這些結果，但未將其作為視圖給出。 對於對稱結果，可能沒有使用視圖的簡便方法。 對於循環結果，似乎也沒有。

就np.pad()而言，應該注意的是計時不如其他方法好（請參閱我的回答）。

3 個解決方案

解決方案1
 1  2019-05-06 13:18:33

無法獲得最終所需的輸出作為輸入視圖。 我們可以通過沿兩個軸復制副本然后切片來改進。 偏置輸入應為正值。 解決方案將遵循以下思路： 

def periodic_view_repeat_slicing(arr, out_shp, offset):
    M,N = out_shp
    m,n = arr.shape
    o = (m-offset[0])%m,(n-offset[1])%n

    fwd_offset = (M+m-1)//m,(N+n-1)//n
    reverse_offset = (offset[0]+m-1)//m, (offset[1]+n-1)//n
    p,q = fwd_offset[0]+reverse_offset[0], fwd_offset[1]+reverse_offset[1]

    arrE = np.tile(arr,(p,q))
    out = arrE[o[0]:o[0]+M,o[1]:o[1]+N]
    return out

解決方案2
 1  2019-05-06 16:04:12

這是使用as_strided的解決方案 

import numpy as np
a0=np.arange(2 * 3).reshape((2, 3))

from numpy.lib.stride_tricks import as_strided

def periodic_view(array, shape, offset):
    ox,oy = offset
    stx,sty = array.strides    
    shx,shy = array.shape   
    nshx,nshy = shape
    nx = (nshx+ox-1)//shx +1 #enough room, with offset<shape.
    ny = (nshy+oy-1)//shy +1
    big_view=as_strided(a0,(nx,shx,ny,shy),(0,stx,0,sty)).reshape(nx*shx,ny*shy)
    return big_view[ox:,oy:][:nshx,:nshy]

嘗試： 

a=periodic_view(arr,(4,5),(1,1))

a
Out[211]: 
array([[4, 5, 3, 4, 5],
       [1, 2, 0, 1, 2],
       [4, 5, 3, 4, 5],
       [1, 2, 0, 1, 2]])

a.flags
Out[212]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

但這不是視圖，如果您修改結果，則不會在原始數組上書寫。

解決方案3
 0  2019-05-21 13:07:49

如果確實無法實現內存效率的視圖（可能是這種情況），則NumPy提供np.pad() ，它應盡可能提高內存效率。 盡管此選項允許輸出具有很大的靈活性，並支持許多填充選項，而不僅是循環的-通過mode='wrap' ，對於這種用例，這似乎相對較慢，並且可以使代碼更快多種方式。 在速度和存儲器效率的最佳折衷是通過使用上的結果的圖np.tile()之后的適當np.roll() cyclic_padding_tile_roll() 請注意，可以跳過np.roll()步驟（ cyclic_padding_tile() ），其代價是可能需要更多的內存，這也可能降低整體性能。 可選地，可以通過切片（ cyclic_padding_slicing() ）獲得內存有效且通常快速的實現，一旦基本形狀多次包含在目標形狀中，該方法可能比其他方法慢得多。

這是我測試過的解決方案的代碼。 除非另有說明，否則它們都應適用於任意尺寸。

通過利用以下事實，使用相同的基本代碼來准備offsets ： 

import numpy as np
import functools
import itertools


def prod(items):
    return functools.reduce(lambda x, y: x * y, base_shape)

def reduce_offsets(offsets, shape, direct=True):
    offsets = tuple(
        (offset if direct else (dim - offset)) % dim
        for offset, dim in zip(offsets, shape))
    return offsets

使用索引循環 （原始方法）： 

def cyclic_padding_loops(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape)
    result = np.zeros(shape, dtype=arr.dtype)
    for i in itertools.product(*tuple(range(dim) for dim in result.shape)):
        slicing = tuple(
            (j + k) % dim
            for j, k, dim in zip(i, offsets, arr.shape))
        result[i] = arr[slicing]
    return result

僅使用np.tile() （這與@Divakar使用相同的方法，但適用於任意尺寸）： 

def cyclic_padding_tile(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape)
    tiling = tuple(
        new_dim // dim + (1 if new_dim % dim else 0) + (1 if offset else 0)
        for offset, dim, new_dim in zip(offsets, arr.shape, shape))
    slicing = tuple(
        slice(offset, offset + new_dim)
        for offset, new_dim in zip(offsets, shape))
    result = np.tile(arr, tiling)[slicing]
    return result

使用np.tile()和np.roll() ： 

def cyclic_padding_tile_roll(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape, False)
    tiling = tuple(
        new_dim // dim + (1 if new_dim % dim else 0)
        for offset, dim, new_dim in zip(offsets, arr.shape, shape))
    slicing = tuple(slice(0, new_dim) for new_dim in shape)
    if any(offset != 0 for offset in offsets):
        nonzero_offsets_axes, nonzero_offsets = tuple(zip(
            *((axis, offset) for axis, offset in enumerate(offsets)
            if offset != 0)))
        arr = np.roll(arr, nonzero_offsets, nonzero_offsets_axes)
    result = np.tile(arr, tiling)[slicing]
    return result

僅使用np.pad() ： 

def cyclic_padding_pad(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape, False)
    width = tuple(
        (offset, new_dim - dim - offset)
        for dim, new_dim, offset in zip(arr.shape, offsets))
    result = np.pad(arr, width, mode='wrap')
    return result

使用np.pad()和np.roll() ： 

def cyclic_padding_pad_roll(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape, False)
    width = tuple(
        (0, new_dim - dim)
        for dim, new_dim, offset in zip(arr.shape, shape, offsets))
    if any(offset != 0 for offset in offsets):
        nonzero_offsets_axes, nonzero_offsets = tuple(zip(
            *((axis, offset) for axis, offset in enumerate(offsets)
            if offset != 0)))
        arr = np.roll(arr, nonzero_offsets, nonzero_offsets_axes)
    result = np.pad(arr, width, mode='wrap')
    return result

使用切片循環 ： 

def cyclic_padding_slicing(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape)
    views = tuple(
        tuple(
            slice(max(0, dim * i - offset), dim * (i + 1) - offset)
            for i in range((new_dim + offset) // dim))
        + (slice(dim * ((new_dim + offset) // dim) - offset, new_dim),)
        for offset, dim, new_dim in zip(offsets, arr.shape, shape))
    views = tuple(
        tuple(slice_ for slice_ in view if slice_.start < slice_.stop)
        for view in views)
    result = np.zeros(shape, dtype=arr.dtype)
    for view in itertools.product(*views):
        slicing = tuple(
            slice(None)
            if slice_.stop - slice_.start == dim else (
                slice(offset, offset + (slice_.stop - slice_.start))
                if slice_.start == 0 else
                slice(0, (slice_.stop - slice_.start)))
            for slice_, offset, dim in zip(view, offsets, arr.shape))
        result[view] = arr[slicing]
    return result

使用跨步（這實際上是適用於n-dim輸入的@BM實現）： 

def cyclic_padding_strides(arr, shape, offsets):
    offsets = reduce_offsets(offsets, arr.shape)
    chunks = tuple(
        new_dim // dim + (1 if new_dim % dim else 0) + (1 if offset else 0)
        for dim, new_dim, offset in zip(arr.shape, shape, offsets))
    inner_shape = tuple(
        x for chunk, dim in zip(chunks, arr.shape) for x in (chunk, dim))
    outer_shape = tuple(
        (chunk * dim) for chunk, dim in zip(chunks, arr.shape))
    inner_strides = tuple(x for stride in arr.strides for x in (0, stride))
    # outer_strides = tuple(x for stride in arr.strides for x in (0, stride))
    slicing = tuple(
        slice(offset, offset + new_dim)
        for offset, new_dim in zip(offsets, shape))
    result = np.lib.stride_tricks.as_strided(
        arr, inner_shape, inner_strides, writeable=False).reshape(outer_shape)
    result = result[slicing]
    return result

這是用於測試的代碼： 

def test_cyclic_paddings(base_shape, shape, offsets, cyclic_paddings):
    print('Base Shape: {},  Shape: {},  Offset: {}'.format(base_shape, shape, offsets))
    arr = np.arange(prod(base_shape)).reshape(base_shape) + 1
    ref_result = cyclic_paddings[0](arr, shape, offsets)
    for cyclic_padding in cyclic_paddings:
        test_result = cyclic_padding(arr, shape, offsets)
        result = np.all(ref_result == test_result)
        if not result:
            print(ref_result)
            print(test_result)
        print(': {:24s} {:4s} '.format(cyclic_padding.__name__, 'OK' if result else 'FAIL'), end='')
        timeit_result = %timeit -o cyclic_padding(arr, shape, offsets)

cyclic_nd_paddings = (
    cyclic_padding_tile,
    cyclic_padding_tile_roll,
    cyclic_padding_pad,
    cyclic_padding_pad_roll,
    cyclic_padding_slicing,
    cyclic_padding_loops,
    cyclic_padding_strides,
)

inputs = (
    ((2, 3), (5, 7), (0, 0)),
    ((2, 3), (5, 7), (0, 1)),
    ((2, 3), (5, 7), (1, 1)),
    ((2, 3), (41, 43), (1, 1)),
    ((2, 3, 4, 5), (7, 11, 13, 17), (1, 2, 3, 4)),
    ((2, 3, 4, 5), (23, 31, 41, 53), (1, 2, 3, 4)),
    ((8, 8), (100, 100), (5, 7)),
    ((80, 80), (8000, 8000), (53, 73)),
    ((800, 800), (9000, 9000), (53, 73)),
)


for (base_shape, shape, offsets) in inputs:
    test_cyclic_paddings(base_shape, shape, offsets, cyclic_nd_paddings)
    print()

對於不同的輸入，這些是我得到的結果： 

# Base Shape: (2, 3),  Shape: (5, 7),  Offset: (0, 0)
# : cyclic_padding_tile      OK   6.54 µs ± 70.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_tile_roll OK   6.75 µs ± 29.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_pad       OK   40.6 µs ± 2.44 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   42 µs ± 4.49 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   23 µs ± 693 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_loops     OK   34.7 µs ± 727 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_strides   OK   13.2 µs ± 210 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# Base Shape: (2, 3),  Shape: (5, 7),  Offset: (0, 1)
# : cyclic_padding_tile      OK   6.5 µs ± 223 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_tile_roll OK   19.8 µs ± 394 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad       OK   35.4 µs ± 329 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   58 µs ± 579 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   23.3 µs ± 321 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_loops     OK   33.7 µs ± 280 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_strides   OK   13.2 µs ± 194 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# Base Shape: (2, 3),  Shape: (5, 7),  Offset: (1, 1)
# : cyclic_padding_tile      OK   6.68 µs ± 138 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_tile_roll OK   23.2 µs ± 334 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad       OK   30.7 µs ± 236 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   62.9 µs ± 1 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   23.5 µs ± 266 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_loops     OK   34.6 µs ± 544 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_strides   OK   13.1 µs ± 104 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# Base Shape: (2, 3),  Shape: (41, 43),  Offset: (1, 1)
# : cyclic_padding_tile      OK   8.92 µs ± 63.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_tile_roll OK   25.2 µs ± 185 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad       OK   60.7 µs ± 450 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   82.2 µs ± 656 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   510 µs ± 1.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# : cyclic_padding_loops     OK   1.57 ms ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# : cyclic_padding_strides   OK   18.2 µs ± 639 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# Base Shape: (2, 3, 4, 5),  Shape: (7, 11, 13, 17),  Offset: (1, 2, 3, 4)
# : cyclic_padding_tile      OK   89 µs ± 3.18 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_tile_roll OK   81.3 µs ± 1.24 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad       OK   106 µs ± 2.77 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   148 µs ± 9.02 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   977 µs ± 8.11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# : cyclic_padding_loops     OK   18.8 ms ± 342 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# : cyclic_padding_strides   OK   101 µs ± 1.86 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Base Shape: (2, 3, 4, 5),  Shape: (23, 31, 41, 53),  Offset: (1, 2, 3, 4)
# : cyclic_padding_tile      OK   2.8 ms ± 112 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# : cyclic_padding_tile_roll OK   2.05 ms ± 28.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# : cyclic_padding_pad       OK   6.35 ms ± 237 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# : cyclic_padding_pad_roll  OK   5.81 ms ± 172 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# : cyclic_padding_slicing   OK   40.4 ms ± 838 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# : cyclic_padding_loops     OK   1.71 s ± 44.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_strides   OK   3 ms ± 64.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# Base Shape: (8, 8),  Shape: (100, 100),  Offset: (5, 7)
# : cyclic_padding_tile      OK   16.3 µs ± 901 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
# : cyclic_padding_tile_roll OK   32.6 µs ± 151 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad       OK   65.6 µs ± 229 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_pad_roll  OK   88.9 µs ± 1.05 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# : cyclic_padding_slicing   OK   333 µs ± 1.86 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# : cyclic_padding_loops     OK   8.71 ms ± 58.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# : cyclic_padding_strides   OK   25.1 µs ± 255 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Base Shape: (80, 80),  Shape: (8000, 8000),  Offset: (53, 73)
# : cyclic_padding_tile      OK   148 ms ± 325 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# : cyclic_padding_tile_roll OK   151 ms ± 1.51 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
# : cyclic_padding_pad       OK   443 ms ± 9.42 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_pad_roll  OK   442 ms ± 8.64 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_slicing   OK   182 ms ± 469 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# : cyclic_padding_loops     OK   58.8 s ± 256 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_strides   OK   150 ms ± 534 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

# Base Shape: (800, 800),  Shape: (9000, 9000),  Offset: (53, 73)
# : cyclic_padding_tile      OK   269 ms ± 1.11 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_tile_roll OK   234 ms ± 1.39 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_pad       OK   591 ms ± 3.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_pad_roll  OK   582 ms ± 4.57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_slicing   OK   250 ms ± 4.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_loops     OK   1min 17s ± 855 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# : cyclic_padding_strides   OK   280 ms ± 2.28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 如何從指針創建n-dim numpy數組？ n-dim中n-dim fft的縮放因子 基於參考 n-dim 數組對 n-dim 數組進行操作的最有效方法 在N維稀疏Numpy數組中，向與行/列索引相鄰的所有像元添加數字的快速方法？ 使用周期性邊界條件模糊3D numpy數組 具有周期性邊界條件的 numpy 數組的邊界框（環繞） 將N-dim數組廣播到（N + 1）-dim數組並在除1個dim之外的所有值上求和 Python numpy：向量（n-dim）與數組（nxn-dim）的維度[0] 使用循環（周期性）邊界條件計算 numpy 數組中點之間距離的最快代碼 用Delaunay三角剖分進行插值（n-dim）
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM