Upload file with signUrl to google cloud storage using and application

Question

I created a Scala application that as part of its functions upload any type of file (BZ2, csv, txt, etc) to google cloud storage.

If I use the direct upload works fine for small files, but to upload a big file google recommends using “signUrl” and this is not creating the file, or updating a file if I create the file before, or throwing any exception with the error.

Works with small files:

val storage: Storage = StorageOptions.getDefaultInstance.getService
val fileContent = Files.readAllBytes(file.toPath)
val fileId: BlobId = BlobId.of(bucketName, s"$folderName/$fileName")
val fileInfo: BlobInfo = BlobInfo.newBuilder(fileId).build()
storage.create(fileInfo, fileContent)

Don´t work:

val storage: Storage = StorageOptions.getDefaultInstance.getService
val outputPath = s"$folderName/$fileName"
val fileInfo: BlobInfo = BlobInfo.newBuilder(bucketName, outputPath).build()
val optionWrite: SignUrlOption = Storage.SignUrlOption.httpMethod(HttpMethod.PUT)
val signedUrl: URL = storage.signUrl(fileInfo, 30, TimeUnit.MINUTES, optionWrite)
val connection = signedUrl.openConnection
connection.setDoOutput(true)
val out = connection.getOutputStream

val inputStream = Files.newInputStream(file.toPath)
var nextByte = inputStream.read()
while (nextByte != -1) {
 out.write(nextByte)
 nextByte = inputStream.read()
}
out.flush()
inputStream.close()
out.close()

I try reading/writing byte by byte, using and array of bytes, and using a OutputStreamWriter but neither work.

The library that I´m using is:

"com.google.cloud" % "google-cloud-storage" % "1.12.0"

Anyone know why this is not working? Thanks

Answer 1

I found an easier way to write big files in Google storage, without using the signURL.

I will add the code just in case someone found it useful.

storage.create(fileInfo)
var writer:WriteChannel = storage.get(fileInfo.getBlobId).writer()
var inputStream: InputStream = Files.newInputStream(file.toPath)
val packetToRead: Array[Byte] = new Array[Byte](sizeBlockDefault toInt)
while (inputStream.available() > 0){
  val numBytesReaded = inputStream.read(packetToRead,0,sizeBlockDefault toInt)
  writer.write(ByteBuffer.wrap(packetToRead, 0, numBytesReaded))
}
inputStream.close()
writer.close()