如何从goroutine的频道连续接收数据

Question

I am a beginner of the Golang. I had made a practice about Go channel. Which I open and read data from a file in the main goroutine, then pass the data to the second goroutine to save to another file with channel. My code is as flows

  func main() {
   f, err := os.OpenFile("test.go", os.O_RDONLY, 0600)
   ch := make(chan []byte)
   buf := make([]byte, 10)
   bytes_len, err := f.Read(buf)
   fmt.Println("ReadLen:", bytes_len)
   if err != nil {
      fmt.Println("Error: ", err)
      return
   }
   go WriteFile(ch)
   for {
      ch<-buf
      bytes_len, err = f.Read(buf)
      if err != nil {
          fmt.Println("error=", err)
          break
      }
      if bytes_len < 10 {
          ch<-buf[:bytes_len]
          fmt.Println("Finished!")
          break
      }
   }
   time.Sleep(1e9)
   f.Close()
 }

  func WriteFile(ch <-chan []byte) {
    fmt.Println("* begin!")
    f, err := os.OpenFile("/home/GoProgram/test/test.file",  os.O_RDWR|os.O_APPEND|os.O_CREATE, 0660)
    if err != nil {
       fmt.Println("* Error:", err)
       return
    }
    /* Method1:  use the "select" will write to target file OK, but it is too slow!!!
    for {
      select {
         case bytes, ok:= <-ch:
            if ok {
              f.Write(bytes)
            } else {
              fmt.Println("* file closed!")
              break
            }
         default:
            fmt.Println("* waiting data!")
      }
    } \*/
    // Method 2: use "for {if}", this will get messed text in target file, not identical with the source file.
    for {
      if bytes, ok := <-ch; ok {
            f.Write(bytes)
            fmt.Println("* buff=", string(bytes))
            bytes = nil
            ok = false
      } else {
        fmt.Println("** End ", string(bytes), "  ", ok)
        break
      }
    }

    /* Method 3: use "for range", this will get messed text like in method2
    for data:= range ch {
         f.Write(data)
       //fmt.Println("* Data:", string(data))
    }
    \*/
    f.Close()
}

My question is why the Method2 and Method3 will get messed text in the target file? How can I fix it?

Answer 1

Method2 and Method3 get messed up text because there's a race on the buffer shared by the reader and writer.

Here's a possible sequence of statement execution for the program above:

 R: bytes_len, err = f.Read(buf)  
 R: ch<-buf[:bytes_len]
 W: bytes, ok := <-ch; ok
 R: bytes_len, err = f.Read(buf)  // this writes over buffer
 W: f.Write(bytes)                // writes data from second read

Run your program with the race dectector . It will flag the issues for you.

One way to fix the problem is to copy the data. For example, create a string from the bytes read and send the string to the channel.

Another option is to connect the goroutines with an io.Pipe . One goroutine reads from the source and writes to the pipe. The other goroutine reads from the pipe and writes to the destination. The pipe takes care of the synchronization issues.

Answer 2

In order to get the code snippets using the for loops in what you have put in comments as Method2 and Method3 , you will need to use a buffered channel.

The reason the text gets messed up in the target file is the loop in func main has no mechanism to synchronize in lock step with the loops listening on the channel in WriteFile .

Sends to a buffered channel, on the other hand, block only when the buffer is full. Receives block when the buffer is empty. So by initializing a channel with a buffer length of one, your can use Method1 and/or Method2 . All that's left is remembering to close the channel when you are done.

func main() {
    f, _ := os.OpenFile("test.txt", os.O_RDONLY, 0600)
    defer f.Close()
    ch := make(chan []byte, 1) // use second argument to make to give buffer length 1
    buf := make([]byte, 10)
    go WriteFile(ch)
    for {
        ch <- buf
        byteLen, err := f.Read(buf)
        if err != nil {
            break
        }
        if byteLen < 10 {
            ch <- buf[:byteLen]
            break
        }
    }
    close(ch) //close the channel when you done
}

func WriteFile(ch <-chan []byte) {
    f, err := os.OpenFile("othertest.txt", os.O_RDWR|os.O_APPEND|os.O_CREATE, 0660)
    defer f.Close()
    if err != nil {
        fmt.Println("* Error:", err)
        return
    }

    //Method 3: use "for range"
    for data := range ch {
        f.Write(data)
    }
}