[英]R: How can I read a CSV file with data.table::fread, that has a comma as decimal and point as thousand separator=“.”
我有幾個CSV文件,其中包含本地德語風格的數字,即逗號作為小數分隔符,點數為千分隔符,例如10.380,45。 CSV文件中的值以“;”分隔。 這些文件還包含類字符,日期,日期和時間以及邏輯的列。
read.table函數的問題是,您可以使用dec =“,”指定小數分隔符,但不能指定千位分隔符。 (如果我錯了,請糾正我)
我知道預處理是一種解決方法,但我想以某種方式編寫代碼,其他人可以在沒有我的情況下使用它。
通過設置我自己的類,我找到了一種通過read.csv2以我想要的方式讀取CSV文件的方法,如下例所示。 基於最優雅的方式加載csv點與R中的千位分隔符
# Create test example
df_test_write <- cbind.data.frame(c("a","b","c","d","e","f","g","h","i","j",rep("k",times=200)),
c("5.200,39","250,36","1.000.258,25","3,58","5,55","10.550,00","10.333,00","80,33","20.500.000,00","10,00",rep("3.133,33",times=200)),
c("25.03.2015","28.04.2015","03.05.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016","08.08.2016",rep("08.08.2016",times=200)),
stringsAsFactors=FALSE)
colnames(df_test_write) <- c("col_text","col_num","col_date")
# write test csv
write.csv2(df_test_write,file="Test.csv",quote=FALSE,row.names=FALSE)
#### read with read.csv2 ####
# First, define your own class
#define your own numeric class
setClass('myNum')
#define conversion
setAs("character","myNum", function(from) as.numeric(gsub(",","\\.",gsub("\\.","",from))))
# own date class
library(lubridate)
setClass('myDate')
setAs("character","myDate",function(from) dmy(from))
# Read the csv file, in colClasses the columns class can be defined
df_test_readcsv <- read.csv2(paste0(getwd(),"/Test.csv"),
stringsAsFactors = FALSE,
colClasses = c(
col_text = "character",
col_num = "myNum",
col_date = "myDate"
)
)
我現在的問題是,不同的數據集最多有200列和350000行。 使用上層解決方案,我需要40到60秒才能加載一個CSV文件,我想加快速度。
通過我的研究,我從data.table
包中找到了fread()
,這非常快。 加載CSV文件大約需要3到5秒。
不幸的是,也沒有可能指定千分隔符。 所以我嘗試使用我的解決方案與colClasses,但似乎有問題,你不能使用個人類與fread https://github.com/Rdatatable/data.table/issues/491
另見我的以下測試代碼:
##### read with fread ####
library(data.table)
# Test without colclasses
df_test_readfread1 <- fread(paste0(getwd(),"/Test.csv"),
stringsAsFactors = FALSE,
dec = ",",
sep=";",
verbose=TRUE)
str(df_test_readfread1)
# PROBLEM: In my real dataset it turns the number into an numeric column,
# unforunately it sees the "." as decimal separator, so it turns e.g. 10.550,
# into 10.5
# Here it keeps everything as character
# Test with colclasses
df_test_readfread2 <- fread(paste0(getwd(),"/Test.csv"),
stringsAsFactors = FALSE,
colClasses = c(
col_text = "character",
col_num = "myNum",
col_date = "myDate"
),
sep=";",
verbose=TRUE)
str(df_test_readfread2)
# Keeps everything as character
所以我的問題是:有沒有辦法讀取數字值為10.380,45與fread的CSV文件?
(或者:讀取具有此類數值的CSV的最快方法是什么?)
首先刪除所有逗號。
filepath<-paste0(getwd(),"/Test.csv")
filestring<-readChar(filepath, file.info(filepath)$size)
filestring<-gsub('.','',filestring,fixed=TRUE)
fread(filestring)
我自己從未使用過包裝,但它來自Hadley Wickham,應該是好東西
https://cran.r-project.org/web/packages/readr/readr.pdf
它應該處理語言環境:
locale(date_names = "en", date_format = "%AD", time_format = "%AT", decimal_mark = ".", grouping_mark = ",", tz = "UTC", encoding = "UTF-8", asciify = FALSE)
decimal_mark
和grouping_mark
就是你要找的東西
編輯形式PhiSeu:解決方案
感謝您的建議,這里有兩個解決方案,其中read_csv2()
readr
包中的read_csv2()
。 對於我的350000行CSV文件,大約需要8秒,這比read.csv2解決方案快得多。 (來自hadley和RStudio的另一個有用的軟件包,謝謝)
library(readr)
# solution 1 with specified columns
df_test_readr <- read_csv2(paste0(getwd(),"/Test.csv"),
locale = locale("de"),
col_names = TRUE,
cols(
col_text = col_character(),
col_num = col_number(), # number is automatically regcognized through locale=("de")
col_date2 = col_date(format ="%d.%m.%Y") # Date specification
)
)
# solution 2 with overall definition of date format
df_test_readr <- read_csv2(paste0(getwd(),"/Test.csv"),
locale = locale("de",date_format = "%d.%m.%Y"), # specifies the date format for the whole file
col_names = TRUE
)
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.