简体繁体中英

How do I compute Memory required to create igniteRDD

原文 2017-10-23 18:25:16 7 1 apache-spark/ memory-management/ datagrid/ rdd/ ignite

Can anyone please help me understanding, How is the Memory calculation done, to store A String, a long variable, and a int variable in IgniteRDD. I was going to forums and find different answers, and I am totally confused on how to calculate the Memory requirement for my application.

I am trying to calculate 48 Billion records with 2 Strings, 1 int and 2 long variables.

Any help will be greatly appreciated.

Regards, DV Nithin

1 answers

See https://apacheignite.readme.io/docs/capacity-planning

Also, for estimating memory size - I would recommend loading various sized data several times and measuring Memory consumed by it. It will help you to understand approximate needed memory size. These metrics will help you in this: https://apacheignite.readme.io/docs/memory-metrics

IgniteRDD Spark

How do I compute my Foundry 'latest version' dataset faster?

IgniteRDD SQL Table Names

Spark: Is the memory required to create a DataFrame somewhat equal to the size of the input data?

How do I make my many-join / many-union datasets compute faster?

Why do I get a “Hive support is required to CREATE Hive TABLE (AS SELECT)” error when creating a table?

How do I create a Scala sequence of expressions?

How do I force a persist to memory of Spark DataFrames?

where do I create a spark configuration file and set.driver.memory to 2gb?

How can we implement scalable Compute Grid with massive concurrent memory updates supporting low latency and consistency

暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question IgniteRDD Spark How do I compute my Foundry 'latest version' dataset faster? IgniteRDD SQL Table Names Spark: Is the memory required to create a DataFrame somewhat equal to the size of the input data? How do I make my many-join / many-union datasets compute faster? Why do I get a “Hive support is required to CREATE Hive TABLE (AS SELECT)” error when creating a table? How do I create a Scala sequence of expressions? How do I force a persist to memory of Spark DataFrames? where do I create a spark configuration file and set.driver.memory to 2gb? How can we implement scalable Compute Grid with massive concurrent memory updates supporting low latency and consistency

Related Tags

How do I compute Memory required to create igniteRDD

Question

1 answers

solution1 1 2017-10-24 14:28:35

solution1
1 2017-10-24 14:28:35