简体   繁体   English

使用Golang进行mmap syscall读取

[英]mmap syscall using Golang for reads

I am not an Go unsafe package expert - Neither am I a seasoned C programmer. 我不是Go不安全的软件包专家-我也不是经验丰富的C程序员。 I am trying to read a huge file > 1G using mmap syscall in go. 我正在尝试使用mmap syscall读取大于1G的大文件。 There are a number of reasons I do mmap and munmap as opposed to read, write I/O. 我做mmap和munmap的原因有很多,而不是读写I / O。 That is beside the point - I can write to the file in a test, when I read from the file, I can ascertain that bytes length matches up, but I cannot read contents of this string file :( Can someone suggest some reading? I need to do to go a little further, here's some code I cooked up for sample test: 那不是重点-我可以在测试中写入文件,当我从文件中读取内容时,我可以确定字节长度匹配,但是我无法读取此字符串文件的内容:(有人可以建议读取吗?需要做得更进一步,这是我为示例测试准备的一些代码:

filename := "/tmp/dd_file.db"
f, err := os.OpenFile(filename, os.O_RDWR, 0666)
defer f.Close()
if err != nil {
    fmt.Printf("error opening file: %v", err)
}
stat, _ := f.Stat()
size := stat.Size()
fmt.Printf("[READ-ONLY] : size was : %+v\n", size)
got := make([]byte, size)
if _, err := f.ReadAt(got, 0); err != nil && err != io.EOF {
    panic(err)
}
want, err := ioutil.ReadFile(filename)
if err != nil {
    fmt.Printf("[READ-ONLY] : ioutil.ReadFile: %v", err)
}
// going to change the file size now, punch in a few things
t := unsafe.Sizeof("")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
_, err = f.Seek(int64(t-1), 0)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
_, err = f.Write([]byte(" "))
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
mmap, err := syscall.Mmap(int(f.Fd()), 0, int(t), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
 // not too sure on reading data on string - doesnt work as expected.
map_array := (*[10000]string)(unsafe.Pointer(&mmap[0]))
map_array[0] = "yellow!"
err = syscall.Munmap(mmap)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
newStat, _ := f.Stat()
newSize := newStat.Size()
fmt.Printf("[mmap( ) RW] : size was : %+v\n", newSize)
got = make([]byte, newSize)
if _, err := f.ReadAt(got, 0); err != nil && err != io.EOF {
    panic(err)
}
if len(got) == len(want) {
    fmt.Println("well the lengths are equal atleast??!")
}
if !bytes.Equal(got, want) {
    fmt.Printf("\n [mmap( ) RW] : works! got  %d \n want %d", len(got), len(want))
}

This obviously works as expected - but what if I wanted to read via mmap( ) on an mmapped file, how do I read string out of these bytes (I have a sense there is an encoding package somewhere that I might have to put to use perhaps but then StringHeader on unsafe documentation confused me). 这显然可以按预期工作-但是,如果我想通过mmap()读取mmapped文件,该如何从这些字节中读取字符串(我感觉有些地方可能需要使用编码包)也许但是不安全文档上的StringHeader令我感到困惑)。

Suggestions. 建议。

As @putu noted in a comment, a byte slice can be converted to a string by a simple type conversion: 正如@putu在评论中指出的那样,可以通过简单的类型转换将字节片转换为字符串:

asStr = string(byteSlice) // entire slice as a string
partStr = string(byteSlice[:100]) // first 100 bytes as a string

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM