将文本文件另存为RData并将其导入R

Question

SNP filtering for missing and redundant markers

I split a large text file (30 GB in size) to small 40 files in the cluster and saved it as RData. Then I am importing these small RData files to R to filter these for missing and redundant SNP markers. But it is giving an error.

I want to split large files to small files, save them as RData, import to R and filtering for missing and redundant markers.

Answer 1

RData file is a binary file and your splited files are tab separated text file. So, they cannot be load with load() . I will suggest removing all header information and column names before you split the vcf file. And load the files into r with read.table(sep='\\t') . Hope the above can help.

I don't have any idea about how to save text file to RData file outside r environment. But, if you concern the big file size of your vcf file and you want to filter the data before loading, I suggest using read_tsv_chunked which can load every chunk ( chunk_size = ) of data and do some process on each chunk of data, and finally available in r environment. And beware of skip the number of header line before column names. The following are the script.

library(tidyverse)

filterFun <- function(df, pos) {
  df <- unique(df)
  count.nas <- rowSums(is.na(df))
  df[-which(count.nas>(0.05*ncol(df))),]
}

data <- read_tsv_chunked("xxx.vcf", skip = 22, chunk_size = 10000, callback = DataFrameCallback$new(filterFun))