適用於Apache Spark的BigQuery連接器-更新分區表

Question

我正在Google DataProc上的Scala中編寫一個Spark作業，該作業每天執行並處理每個標記有交易時間的記錄。 記錄按年份-月份組合進行分組，每個組都寫入GCS中單獨的每月實木復合地板文件（例如2018-07-file.parquet ， 2018-08-file.parquet等）。 請注意，這些文件可以追溯到大約5年后，形成一個非常大的數據集（〜1TB）。

我想將這些文件寫入BigQuery，並讓作業僅更新當前運行中已更改的每月記錄 。 為簡單起見，我想刪除具有更新記錄的任何月份的現有記錄，然后僅從每月實木復合地板文件中加載數據。

我正在嘗試將BigQuery Connector用於DataProc，但是它似乎僅支持更新整個表 ，而不支持例如按日期字段過濾的一批記錄。

做這個的最好方式是什么？ 我嘗試將完整的BigQuery庫JAR包含到我的項目中，並使用數據操作查詢來刪除現有的每月記錄，如下所示：

def writeDataset(sparkContext: SparkContext, monthYear: String, ds: Dataset[TargetOrder]) = {
    val dtMonthYear = FeedWriter.parquetDateFormat.parse(monthYear)
    val bigquery: BigQuery = BigQueryOptions.getDefaultInstance.getService
    val queryConfig: QueryJobConfiguration =
      QueryJobConfiguration.newBuilder("DELETE FROM `" + getBQTableName(monthYear) + "` " +
        "WHERE header.trans_time BETWEEN PARSE_DATETIME('" + FeedWriter.parquetDateFormat.toPattern + "', '" + monthYear + "') " +
        "AND PARSE_DATETIME('" + FeedWriter.parquetDateFormat.toPattern + "', '" + DateUtils.addMonths(dtMonthYear, 1) + "') ")
    .setUseLegacySql(false)
    .build();

    val jobId: JobId = JobId.of(UUID.randomUUID().toString());
    val queryJob: Job = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build()).waitFor()
}

但是我收到以下錯誤消息（我假設不允許在DataProc作業中包含完整的BQ客戶端JAR，或者在BQ連接器中不能很好地發揮作用）：

java.lang.NoSuchMethodError: com.google.api.services.bigquery.model.JobReference.setLocation(Ljava/lang/String;)Lcom/google/api/services/bigquery/model/JobReference;
  at com.google.cloud.bigquery.JobId.toPb(JobId.java:114)
  at com.google.cloud.bigquery.JobInfo.toPb(JobInfo.java:370)
  at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:198)
  at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:187)
  at ca.mycompany.myproject.output.BigQueryWriter$.writeDataset(BigQueryWriter.scala:39)

Answer 1

我發現在DataProc作業中包含完整的客戶端JAR似乎不起作用（因此為什么它們為BQ和其他服務創建了單獨的連接器擴展），所以我最終讓我的Dataproc作業將消息提交到發布/訂閱隊列指示哪個每月實木復合地板文件已更新。 然后，我創建了一個Cloud Function來監視發布/訂閱隊列，並生成BigQuery作業以僅加載已更改的每月文件。

我能夠通過使用表分區 （例如MyTable $ 20180101 ）從BQ表中刪除月記錄，並將所有月記錄分組到同一天（當前，BQ僅支持按DAY而不是按月對表進行分區，所以我例如，必須為設置為2018-01-xx中的所有記錄的2018-01-01的每個記錄創建一個單獨的字段）。

Dataproc中要寫入Pub / Sub隊列的Scala代碼示例：

import java.text.SimpleDateFormat
import java.util.{Date, TimeZone, UUID}

import ca.my.company.config.ConfigOptions
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport
import com.google.api.client.json.jackson2.JacksonFactory
import com.google.api.services.pubsub.Pubsub
import com.google.api.services.pubsub.model.{PublishRequest, PubsubMessage}
import com.google.cloud.hadoop.util.RetryHttpInitializer
import org.apache.spark.streaming.pubsub.SparkGCPCredentials

import scala.collection.mutable

case class MyPubSubMessage (jobId: UUID, processedDate: Date, fileDate: Date,  updatedFilePath: String)

object PubSubWriter {
  private val PUBSUB_APP_NAME: String = "MyPubSubWriter"
  private val messages: mutable.ListBuffer[PubsubMessage] = mutable.ListBuffer()
  private val publishRequest = new PublishRequest()
  private lazy val projectId: String = ConfigOptions().pubsubConfig.projectId
  private lazy val topicId: String = ConfigOptions().pubsubConfig.topicId

  private lazy val client = new Pubsub.Builder(
    GoogleNetHttpTransport.newTrustedTransport(),
    JacksonFactory.getDefaultInstance(),
    new RetryHttpInitializer(
      SparkGCPCredentials.builder.build().provider,
      PUBSUB_APP_NAME
    ))
    .setApplicationName(PUBSUB_APP_NAME)
    .build()

  def queueMessage(message: TlogPubSubMessage) {
    if (message == null) return
    val targetFileDateFormat = new SimpleDateFormat("yyyyMMdd")
    val isoDateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss'Z'")
    isoDateFormat.setTimeZone(TimeZone.getTimeZone("UTC"))

    import scala.collection.JavaConversions._
    val pubSubMessage = new PubsubMessage()
      .setAttributes(Map("msgType" -> "t-log-notification", "jobId" -> message.jobId.toString, "processedDate" -> isoDateFormat.format(message.processedDate), "fileDate" -> targetFileDateFormat.format(message.fileDate)))

    messages.synchronized {
      messages.append(pubSubMessage.encodeData(message.updatedFilePath.getBytes))
    }
  }

  def publishMessages(): Unit = {
    import scala.collection.JavaConversions._
    publishRequest.setMessages(messages)
    client.projects().topics()
      .publish(s"projects/$projectId/topics/$topicId", publishRequest)
      .execute()

    println(s"Update notifications: successfully sent ${messages.length} message(s) for topic '${topicId}' to Pub/Sub")
  }
}

我的Python雲函數示例，可從隊列中使用並生成BQ加載作業：

def update_bigquery(data, context):
    import base64
    from datetime import datetime
    from dateutil import parser
    from google.cloud import bigquery
    from google.cloud.bigquery.table import TimePartitioning
    from google.api_core.exceptions import GoogleAPICallError

    dataset_id = 'mydatasetname'
    table_id_base = 'mytablename'

    # The data field looks like this:
    # {'@type': 'type.googleapis.com/google.pubsub.v1.PubsubMessage', 'attributes': {'fileDate': '20171201',
    # 'jobId': '69f6307e-28a1-40fc-bb6d-572c0bea9346', 'msgType': 't-log-notification',
    # 'processedDate': '2018-09-08T02:51:54Z'}, 'data': 'Z3M6Ly9nY3MtbGRsLWRzLWRhdGE...=='}

    # Retrieve file path (filter out SUCCESS file in the folder path) and build the partition name
    attributes = data['attributes']
    file_path = base64.b64decode(data['data']).decode('utf-8') + "/part*"
    partition_name = attributes['fileDate']
    table_partition = table_id_base + "$" + partition_name

    # Instantiate BQ client
    client = bigquery.Client()

    # Get reference to dataset and table
    dataset_ref = client.dataset(dataset_id)
    table_ref = dataset_ref.table(table_partition)

    try:
        # This only deletes the table partition and not the entire table
        client.delete_table(table_ref)  # API request
        print('Table {}:{} deleted.'.format(dataset_id, table_partition))

    except GoogleAPICallError as e:
        print('Error deleting table ' + table_partition + ": " + str(e))

    # Create BigQuery loading job
    job_config = bigquery.LoadJobConfig()
    job_config.source_format = bigquery.SourceFormat.PARQUET
    job_config.time_partitioning = TimePartitioning(field='bigQueryPartition')

    try :
        load_job = client.load_table_from_uri(
            file_path,
            dataset_ref.table(table_partition),
            job_config=job_config)  # API request

        print('Starting job {}'.format(load_job.job_id))

        # This can be commented-out to allow the job to run purely asynchronously
        # though if it fails, I'm not sure how I could be notified
        # For now, I will set this function to the max timeout (9 mins) and see if the BQ load job can consistently complete in time
        load_job.result()  # Waits for table load to complete.
        print('Job finished.')

    except GoogleAPICallError as e:
        print("Error running BQ load job: " + str(e))
        raise e

    return 'Success'

Answer 2

bigquery4s怎么樣？

它是BQ Java客戶端的Scala包裝器。。我遇到了同樣的問題，它對我有用。