空间数据和并行计算

Question

I'll be starting my dissertation work in the new year. I will be doing computationally intense analyses with large spatial data (running spatial regression and geo-weighted regression models with US census tract shapefile data). My current computer freezes up when I open up the shapefile. It's a MacBook Pro with 4 cores, 16GB RAM, and 3.4 Ghz processors. But I'm upgrading to a iMac with 128GB of RAM, 3.6Ghz processor with 8 cores.

However, I've been reading about parallel processing and realizing that R only uses one core. So, does that mean that the additional new cores will be useless? If so, then maybe I save some money and don't go for the extra cores? I understand I can use the parallel package (and some others), but I'm not sure that works with the spatial regressions packages.

Any suggestions here would be very much appreciated.

Best, Kasey

Answer 1

R is capable of using multiple cores but not in the same way as other languages like python. When you use the parallel package it pretty much starts an R session per core assigned. Each core loads a copy of the data and does not use shared memory:'(. So you are making use of the multiple cores, your 8 physical cores should be 16 virtual cores with hyperthreading . For example, if you load a list of four dataframes you can analyse them in parallel using the parallel package over four cores, each core starts an R session, loads in the data, and analyses part of the data.

The process of assigning data to separate cores has some overhead so doing a job in serial is most resource efficient. In serial means that each of the four dataframes are analysed in sequence (one after the other) not in parallel.

Since it may take a long time to do what you want in serial (say to looping over thousands of independent dataframes), going parallel can save you time and you can do some scaling tests to determine the number of cores that will be most efficient (eg, using 20 cores may save little more time than using 16 because scaling is not linear in time gain with number of cores, see link 2 below). If your data are huge you may run into ram limitations because each core will require a chunk of ram to load and process the data (eg, maybe you can only use 4 cores because each one needs to load 30 gb of data and store it in RAM, put very very roughly).

I can't speak as to which spatial packages will work in parallel but if the analyses on each core are independent of each other then it shouldn't be an issue (never had problems myself that is.). If you are doing something complicated that requires data to be stitched together among cores then maybe some packages can't handle that.

Additional cores won't be useless but ultimately the best allocation of computer resources is dependent on the data and analyses. I guess I wouldn't base the decision of what computer to purchase on a single project, you may do more scientific computing in the future and often don't necessarily have an accurate idea of the required resources in advance. Also, your university may also have some high performance computing infrastructure for heavy tasks.

This isn't a definitive answer but was too long for a comment (I can remove if inappropriate): Hope it can help :)

See these links for more detail:

• link 1

• link 2

• link 3

PS In the parallel package, make sure you are using the correct function for your OS otherwise it may just run single threaded without you knowing.

PPS do everything you can to increase effeciency in serial by efficient programming (eg, using numeric matrix rather than data frames or being careful subsetting large data as you will be creating copies in ram). Do some profiling to figure out where your bottle necks are and focus on those first. Then worry about going parallel:)