Optimizing Flink transformation

Question

I have the following method that computes the probability of a value in a DataSet :

/**
   * Compute the probabilities of each value on the given [[DataSet]]
   *
   * @param x single colum [[DataSet]]
   * @return Sequence of probabilites for each value
   */
  private[this] def probs(x: DataSet[Double]): Seq[Double] = {
        val counts = x.groupBy(_.doubleValue)
          .reduceGroup(_.size.toDouble)
          .name("X Probs")
          .collect

        val total = counts.sum

        counts.map(_ / total)
  }

The problem is that when I submit my flink job, that uses this method, its causing flink to kill the job due to a task TimeOut . I am executing this method for each attribute on a DataSet with only 40.000 instances and 9 attributes.

Is there a way I could do this code more efficient?

After a few tries, I made it work with mapPartition , this method is part of a class InformationTheory , which does some computations to calculate Entropy, mutual information etc. So, for example, SymmetricalUncertainty is computed as this:

/**
   * Computes 'symmetrical uncertainty' (SU) - a symmetric mutual information measure.
   *
   * It is defined as SU(X, y) = 2 * (IG(X|Y) / (H(X) + H(Y)))
   *
   * @param xy [[DataSet]] with two features
   * @return SU value
   */
  def symmetricalUncertainty(xy: DataSet[(Double, Double)]): Double = {
    val su = xy.mapPartitionWith {
      case in ⇒
        val x = in map (_._2)
        val y = in map (_._1)

        val mu = mutualInformation(x, y)
        val Hx = entropy(x)
        val Hy = entropy(y)

        Some(2 * mu / (Hx + Hy))
    }

    su.collect.head.head
  }

With this, I can compute efficiently entropy , mutual information etc. The catch is, it only works with a level of parallelism of 1, the problem resides in mapPartition .

Is there a way I could do something similar to what I am doing here with SymmetricalUncertainty , but with whatever level of parallelism?

Answer 1

I finally did it, don't know if its the best solution, but its working with n levels of parallelism:

def symmetricalUncertainty(xy: DataSet[(Double, Double)]): Double = {
    val su = xy.reduceGroup { in ⇒
        val invec = in.toVector
        val x = invec map (_._2)
        val y = invec map (_._1)

        val mu = mutualInformation(x, y)
        val Hx = entropy(x)
        val Hy = entropy(y)

        2 * mu / (Hx + Hy)
    }

    su.collect.head
  } 

You can check the entire code at InformationTheory.scala , and its tests InformationTheorySpec.scala