[英]How to allocate memory in SLURM?
我是 Slurm 的新手。 下面,我要執行一個 Python 文件,它需要 92.3GiB。 我分配了 120GB,但我的代碼仍然返回 memory 錯誤。
submit_venv.sh
#/bin/bash
#SBATCH --account=melchua
#SBATCH --mem=120GB
#SBATCH --time=2`:00:00
module load python/3.8.2
python3 1.methylation_data_processing.py
使用./submit_venv.sh
運行腳本
追溯:
File "1.methylation_data_processing.py", line 49, in <module>
meth_clin = pd.concat([gene_symbol, meth_clin]) # add gene_symbol to dataframe
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/reshape/concat.py", line 307, in concat
return op.get_result()
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/reshape/concat.py", line 532, in get_result
new_data = concatenate_managers(
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/internals/concat.py", line 222, in concatenate_managers
values = _concatenate_join_units(join_units, concat_axis, copy=copy)
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/internals/concat.py", line 486, in _concatenate_join_units
to_concat = [
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/internals/concat.py", line 487, in <listcomp>
ju.get_reindexed_values(empty_dtype=empty_dtype, upcasted_na=upcasted_na)
File "/scg/apps/software/python/3.8.2/lib/python3.8/site-packages/pandas/core/internals/concat.py", line 441, in get_reindexed_values
missing_arr = np.empty(self.shape, dtype=empty_dtype)
numpy.core._exceptions.MemoryError: Unable to allocate 92.3 GiB for an array with shape (111331, 111332) and data type object
假設您的 slurm.conf 文件正確地將 RAM 列為消耗性資源(例如,SelectTypeParameters=CR_CPU_Memory),問題可能與 Slurm 無關,很可能與您的操作系統不希望將那么多 memory 分配給單個任務。 這里有一個類似的問題: Unable to allocate array with shape and data type 。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.