用Pyspark将时间戳记写入Postgres

Question

I'm working on a Spark script on Python (using Pyspark). I have a function that returns a Ro w with some fields, including

timestamp=datetime.strptime(processed_data[1], DATI_REGEX)

processed_data[1] is a valid datetime string.

Edit to show complete code:

DATI_REGEX = "%Y-%m-%dT%H:%M:%S"

class UserActivity(object):
    def __init__(self, user, rows):
        self.user = int(user)
        self.rows = sorted(rows, key=operator.attrgetter('timestamp'))

    def write(self):
        return Row(
            user=self.user,
            timestamp=self.rows[-1].timestamp,
        )

def parse_log_line(logline):
    try:
       entries = logline.split('\\t')
       processed_data = entries[0].split('\t') + entries[1:]

       return Row(
           ip_address=processed_data[9],
           user=int(processed_data[10]),
           timestamp=datetime.strptime(processed_data[1], DATI_REGEX),
       )
     except (IndexError, ValueError):
          return None


logFile = sc.textFile(...)
rows = (log_file.map(parse_log_line).filter(None)
        .filter(lambda x: current_day <= x.timestamp < next_day))
user_rows = rows.map(lambda x: (x.user, x)).groupByKey()
user_dailies = user_rows.map(lambda x: UserActivity(current_day, x[0], x[1]).write())

The problem comes when I try to write that on a PostgreSQL DB, doing the following:

fields = [
    StructField("user_id", IntegerType(), False),
    StructField("timestamp", TimestampType(), False),
]
schema = StructType(fields)
user_dailies_schema = SQLContext(sc).createDataFrame(user_dailies, schema)
user_dailies_schema.write.jdbc(
    "jdbc:postgresql:.......",
    "tablename")

I get the following error:

Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 172, in main
    process()
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 167, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/types.py", line 576, in toInternal
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/types.py", line 576, in <genexpr>
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/types.py", line 436, in toInternal
    return self.dataType.toInternal(obj)
  File "/Users/pau/Downloads/spark-2.0.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/types.py", line 190, in toInternal
    seconds = (calendar.timegm(dt.utctimetuple()) if dt.tzinfo
AttributeError: 'int' object has no attribute 'tzinfo'

    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more

Any idea on how to solve that?

Answer 1

The problem is relatively simple. PySpark Row is a tuple ordered by field names. It means that when you create:

Row(user=self.user, timestamp=self.rows[-1].timestamp)

the output structure is ordered as:

Row(timestamp, user)

StructType from the other hand is ordered as is. As a result you're code is tries to use user id as a timestamp. You should either return a plain tuple :

class UserActivity(object):
    ...
    def write(self):
        return (self.user, timestamp)

or use lexicographically ordered schema:

schema = StructType(sorted(fields, key=operator.attrgetter("name")))

Finally you can use namedtuple to achieve both attribute access and predefined order.

On a side note don't use groupByKey like this. It is a typical case when one would use reduceByKey :

(log_file.map(parse_log_line)
    .map(operator.attrgetter("user", "timestamp"))
    .reduceByKey(max))

with multiple fields:

from functools import partial

(log_file.map(parse_log_line)
    .map(lambda x: (x.user, x))
    .reduceByKey(partial(max, key=operator.itemgetter("timestamp")))
    .values())

or DataFrame aggregations:

from pyspark.sql import functions as f

(sqlContext
    .createDataFrame(
        log_file.map(parse_log_line)
          # Another way to handle ordering is to choose fields
          # before you call createDataFrame
          .map(operator.attrgetter("user", "timestamp")),
        schema)
    .groupBy("user_id")
    .agg(f.max("timestamp").alias("timestamp")))

Also if you want to retrieve SQLContext you should use the factory method:

SQLContext.getOrCreate(sc)

Creating new context like you do can have unexpected side effects.