[英]R with tcltk/tcltk2: Improve slow performance when displaying big data.frame with TkTable?
Please see two edits below (added later)... 请在下面看到两个修改(稍后添加)...
I have loaded a big data.frame into memory ( 2.7 mio rows and 7 columns - 74 MB of RAM ). 我已经将一个大的data.frame加载到内存中( 2.7 mio行和7列-74 MB RAM )。
If I want to view the data using Tcl/Tk's Tktable widget via the tcltk2 package function tk2edit
如果我想通过tcltk2包函数
tk2edit
使用Tcl / Tk的Tktable小部件查看数据
Example: 例:
library(tcltk2)
my.data.frame <- data.frame(ID=1:2600000,
col1=rep(LETTERS,100000),
col2=rep(letters,1E5),
col3=26E5:1) # about 40 MB of data
tk2edit(my.data.frame)
The basic problem seems to be that each cell of the data.frame must loaded into an tcl array via two nested loops ( see the code in this tktable question ). 基本问题似乎是data.frame的每个单元必须通过两个嵌套循环加载到tcl数组中( 请参阅此tktable问题中的代码 )。
The tcltk2 package's function tk2edit
works the same way, over-simplified: tcltk2软件包的功能
tk2edit
以相同的方式工作,但过于简化:
# my.data.frame contains a lot of rows...
for (i in 0:(dim(my.data.frame)[1])) {
for (j in 0:(dim(my.data.frame)[2]-1)) {
tclarray1[[i,j]] <- my.data.frame[i, j]
}
}
Question: Is there any way to optimize displaying big data.frames with tktable, eg by avoiding the nested loops? 问题:是否有任何方法可以通过tktable优化显示大data.frames,例如通过避免嵌套循环? I just want to view data (no editing required)...
我只想查看数据(无需编辑)...
tktable
has the -variable
option where you can set the tcl array variable that contains ALL the data of the table. tktable
有-variable
选项,您可以设置包含该表的所有数据TCL数组变量。 So we "only" have to find way to create a tcl array from an R data.frame with "one call to tcl from R"... 因此,我们“仅”必须找到一种方法,通过“从R调用tcl”来从R data.frame创建tcl数组...
PS: This is not a problem of the tcltk2
package but seems to be a general problem how to "bulk load" data of a data.frame into Tcl variables... PS:这不是
tcltk2
软件包的问题,但似乎是一个普遍的问题,如何将data.frame的数据“批量加载”到Tcl变量中...
PS2: The good thing is that Tktable
seems to be able to display such a lot of data efficiently (I can scroll and even edit cells without noticing any severe delays). PS2:好处是
Tktable
似乎能够有效地显示大量数据(我可以滚动甚至编辑单元格,而不会注意到任何严重的延迟)。
Tktable
and data in an array Tktable
月1日):将纯Tcl / Tk基准测试结果与Tktable
和数组中的数据Tktable
I have prepared a simple benchmark in Tcl/Tk to measure the execution time and memory consumption of filling a similar Tktable
: 我已经在Tcl / Tk中准备了一个简单的基准测试,以测量执行类似
Tktable
时间和内存消耗:
#!/usr/bin/env wish
package require Tktable
set rows 2700000
set columns 4
for {set row 0} {$row <= $rows} {incr row} {
for {set column 0} {$column < $columns} {incr column} {
if {$row == 0} {
set data($row,$column) Titel$column
} else {
set data($row,$column) R${row}C${column}
}
}
}
ttk::frame .fr
table .fr.table -rows $rows -cols $columns -titlerows 1 -titlecols 0 -height 5 -width 25 -rowheight 1 -colwidth 9 -maxheight 100 -maxwidth 400 -selectmode extended -variable data -xscrollcommand {.fr.xscroll set} -yscrollcommand {.fr.yscroll set}
scrollbar .fr.xscroll -command {.fr.table xview} -orient horizontal
scrollbar .fr.yscroll -command {.fr.table yview}
pack .fr -fill both -expand 1
pack .fr.xscroll -side bottom -fill x
pack .fr.yscroll -side right -fill y
pack .fr.table -side right -fill both -expand 1
Results: 结果:
Conclusion: Tcl/Tk arrays are wasting memory, but the performance is very good (the runtime of 15 minutes when using R with tcltk seem to be caused by R to Tcl/Tk communication overhead. 结论: Tcl / Tk阵列浪费内存,但是性能非常好(将R与tcltk一起使用时15分钟的运行时间似乎是由于R与Tcl / Tk的通信开销引起的。
Test setup: Ubuntu 14.04 64 Bit with 16 GB RAM... 测试设置:Ubuntu 14.04 64 Bit with 16 GB RAM ...
To compare the memory consumption of Tktable
to ttk::treeview
I wrote this code: 为了比较
Tktable
和ttk::treeview
的内存消耗,我编写了以下代码:
#!/usr/bin/env wish
set rows 2700000
set columns 4
set data {}
set colnames {}
for {set i 0} {$i < $columns} {incr i} {
lappend colnames Title$i
}
for {set row 0} {$row <= $rows} {incr row} {
set newrow {}
for {set column 0} {$column < $columns} {incr column} {
lappend newrow R${row}C${column}
}
lappend data $newrow
}
ttk::treeview .tv -columns $colnames -show headings -yscrollcommand {.sbY set} -xscrollcommand {.sbX set}
foreach Element $data {
.tv insert {} end -values $Element
}
foreach column $colnames {
.tv heading $column -text $column
}
ttk::scrollbar .sbY -command {.tv yview}
ttk::scrollbar .sbX -command {.tv xview} -orient horizontal
pack .sbY -side right -fill y
pack .sbX -side bottom -fill x
pack .tv -side left -fill both
Results: 结果:
Conclusion: 结论:
treeview
is more memory efficient than Tktable
since it can use a list instead of an array. treeview
比Tktable
更具内存效率,因为它可以使用列表而不是数组。 I have found one possible solution/workaround using Tktable
in an "unbound" (command) mode . 我发现在“未绑定”(命令)模式下使用
Tktable
一种可能的解决方案/解决方法。
With the command
option of Tktable
you can specify a function that is called each time a cell shall be displayed on the screen. 使用
Tktable
的command
选项,您可以指定每次在屏幕上显示一个单元格时都会调用的函数。 This avoids "loading" all the data from R to Tcl at once improving the "start-up" time and significantly reduces the memory consumption caused by TCL's way of storing arrays and lists. 这避免了一次将所有数据从R加载到Tcl的情况,从而缩短了“启动”时间,并显着减少了TCL存储阵列和列表的方式所导致的内存消耗。
This way every time you scroll a series of function calls are done to ask for the content of the visible cells. 这样,每次滚动一系列函数时,都会要求提供可见单元格的内容。
It works for me even with over 10 mio. 即使超过10毫欧,它也对我有用。 rows!
行!
Drawback: Calling an R function that returns a Tcl variable for each cell is still far from being efficient. 缺点:调用R函数返回每个单元格的Tcl变量仍然远远不够有效。 If you scroll for the first time you can watch the cells being updated.
如果您是第一次滚动,则可以观看正在更新的单元格。 Therefore I am still looking for a bulk data transfer solution between R and Tcl/Tk.
因此,我仍在寻找R和Tcl / Tk之间的批量数据传输解决方案。
Any suggestions to improve the performance are welcome! 欢迎任何改善性能的建议!
I have implemented a small demo (with 1 mio. rows and 21 columns consuming 1.2 GB of RAM) and added some buttons to test different features (like caching). 我实现了一个小型演示(具有1个mio行和21个列,消耗1.2 GB的RAM),并添加了一些按钮来测试不同的功能(例如缓存)。
Note: The long start-up time is caused by creating the underlying test data, NOT by Tktable! 注意:启动时间过长是由创建基础测试数据引起的,而不是由Tktable引起的!
library(tcltk)
library(data.table)
# Tktable example with -command ("unbound" mode) ---------------------------
# Doc: http://tktable.sourceforge.net/tktable/doc/tkTable.html
NUM.ROWS <- 1E6
NUM.COLS <- 20
# generate a big data.frame - this will take a while but is required for the demo
dt.data <- data.table(ID = 1:NUM.ROWS)
for (i in 1:NUM.COLS) {
dt.data[, (paste("Col",i)) := paste0("R", 1:NUM.ROWS, " C", i)]
}
# Fill one cell with a long text containing special control characters to test the Tktable behaviour
dt.data[3,3 := "This is a long text with backslash \\ and \"quotes\"!"]
tclRequire("Tktable")
t <- tktoplevel()
tkwm.protocol(t, "WM_DELETE_WINDOW", function() tkdestroy(t))
# Function to return the current row and column as "calculated" value (without an underlying data "model")
calculated.data <- function(C) {
# Function arguments for Tcl "substitutions":
# See: http://tktable.sourceforge.net/tktable/doc/tkTable.html
# %c the column of the triggered cell.
# %C A convenience substitution for %r,%c.
# %i 0 for a read (get) and 1 for a write (set). Otherwise it is the current cursor position in the cell.
# %r the row of the triggered cell.
return(tclVar(C)) # this does work!
}
# Function to return the content of a data.table for the current row and colum
data.frame.data <- function(r, c) {
if( r == "0")
return(tclVar(names(dt.data)[as.integer(c)+1])) # First row contains the column names
else
return(tclVar(as.character(dt.data[as.integer(r)+1, as.integer(c)+1, with = FALSE]))) # Other rows are data rows
}
frame <- ttklabelframe(t, text = "Data:")
# Add the table to the window environment to ensure killing it when the window is closed (= no more phantom calls to the data command handler)
# Cache = TRUE: This greatly enhances speed performance when used with -command but uses extra memory.
t$env$table <- tkwidget(frame, "table", rows = NUM.ROWS, cols = NUM.COLS, titlerows = 1, selecttype = "cell", selectmode = "extended", command = calculated.data, cache = TRUE, yscrollcommand = function(...) tkset(scroll.y, ...), xscrollcommand = function(...) tkset(scroll.x, ...))
scroll.x <- ttkscrollbar(frame, orient = "horizontal", command=function(...) tkxview(t$env$table,...)) # command that performs the scrolling
scroll.y <- ttkscrollbar(frame, orient = "vertical", command=function(...) tkyview(t$env$table,...)) # command that performs the scrolling
buttons <- ttkframe(t)
btn.read.only <- ttkbutton(buttons, text = "make read only", command = function() tkconfigure(t$env$table, state = "disabled"))
btn.read.write <- ttkbutton(buttons, text = "make writable", command = function() tkconfigure(t$env$table, state = "normal"))
btn.clear.cache <- ttkbutton(buttons, text = "clear cache", command = function() tcl(t$env$table, "clear", "cache"))
btn.bind.data.frame <- ttkbutton(buttons, text = "Fill cells from R data.table",
command = function() {
tkconfigure(t$env$table, command = data.frame.data, rows = nrow(dt.data), cols = ncol(dt.data), titlerows = 1)
tcl(t$env$table, "clear", "cache")
tkwm.title(t,"Cells are filled from an R data.table")
})
btn.bind.calc.value <- ttkbutton(buttons, text = "Fill cells with calculated values",
command = function() {
tkconfigure(t$env$table, command = calculated.data, rows = 1E5, cols = 40)
tcl(t$env$table, "clear", "cache")
tkwm.title(t,"Cells are calculated values (to test the highest performance possible)")
})
tkgrid(btn.read.only, row = 0, column = 1)
tkgrid(btn.read.write, row = 0, column = 2)
tkgrid(btn.clear.cache, row = 0, column = 3)
tkgrid(btn.bind.data.frame, row = 0, column = 5)
tkgrid(btn.bind.calc.value, row = 0, column = 6)
tkpack(frame, fill = "both", expand = TRUE)
tkpack(scroll.x, fill = "x", expand = FALSE, side = "bottom")
tkpack(scroll.y, fill = "y", expand = FALSE, side = "right")
tkpack(t$env$table, fill = "both", expand = TRUE, side = "left")
tkpack(buttons, side = "bottom")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.