數據文件和ETS表之間的平均大小比例是多少？

Question

我正在評估使用Erlang ETS來存儲大量的內存數據集。 我的測試數據源是一個CSV文件，僅消耗350 MB的磁盤。

我的解析器逐行讀取並將其拼接成一個列表，然后使用“bag”配置創建一個元組並將其存儲在ETS中 。

在ETS中加載所有數據后，我注意到我的計算機的8GB內存全部消失了，操作系統創建了虛擬內存，占用了16GB或RAM附近。 erlang的Beam進程似乎比磁盤數據的大小消耗大約10倍的內存。

這是測試代碼 ：

-module(load_test_data).
-author("gextra").

%% API
-export([test/0]).

init_ets() ->
  ets:new(memdatabase, [bag, named_table]).

parse(File) ->
  {ok, F} = file:open(File, [read, raw]),
  parse(F, file:read_line(F), []).

parse(F, eof, Done) ->
  file:close(F),
  lists:reverse(Done);

parse(F, Line, Done) ->
  parse(F, file:read_line(F), [ parse_row_commodity_data(Line) | Done ]).

parse_row_commodity_data(Line) ->
  {ok, Data} = Line,
  %%io:fwrite(Data),
  LineList          = re:split(Data,"\,",[{return,list}]),
  ReportingCountry  = lists:nth(1, LineList),
  YearPeriod        = lists:nth(2, LineList),
  Year              = lists:nth(3, LineList),
  Period            = lists:nth(4, LineList),
  TradeFlow         = lists:nth(5, LineList), 
  Commodity         = lists:nth(6, LineList),
  PartnerCountry    = lists:nth(7, LineList),
  NetWeight         = lists:nth(8, LineList),
  Value             = lists:nth(9, LineList),
  IsReported        = lists:nth(10, LineList),
  ets:insert(memdatabase, {YearPeriod ++ ReportingCountry ++ Commodity , { ReportingCountry, Year, Period, TradeFlow, Commodity, PartnerCountry, NetWeight, Value, IsReported } }).


test() ->
  init_ets(),
  parse("/data/000-2010-1.csv").

Answer 1

它強烈依賴於你將它拼接到列表中的意思，然后創建一個元組 。 特別是拼接到列表中會占用大量內存。 如果拆分成列表，則一個字節可占用16B。 容量為5.6GB。

編輯 ：

試試這個：

parse(File) ->
  {ok, F} = file:open(File, [read, raw, binary]),
  ok = parse(F, binary:compile_pattern([<<$,>>, <<$\n>>])),
  ok = file:close(F).

parse(F, CP) ->
  case file:read_line(F) of
    {ok, Line} ->
      parse_row_commodity_data(Line, CP),
      parse(F, CP);
    eof -> ok
  end.

parse_row_commodity_data(Line, CP) ->
  [ ReportingCountry, YearPeriod, Year, Period, TradeFlow, Commodity,
    PartnerCountry, NetWeight, Value, IsReported]
      = binary:split(Line, CP, [global, trim]),
  true = ets:insert(memdatabase, {
         {YearPeriod, ReportingCountry, Commodity},
         { ReportingCountry, Year, Period, TradeFlow, Commodity,
           PartnerCountry, NetWeight, Value, IsReported}
       }).