在 Google App Engine 中運行 Google Cloud Dataflow 管道時出現“ClassNotFoundException: sun.security.provider.Sun”

Question

Dataflow 管道中的DoFn包含一個類型，其中Random字段指向SecureRandom實例，並且該字段在使用DataflowPipelineRunner在 Dataflow 服務中運行時無法反序列化。 （下面的堆棧跟蹤）

我們使用其默認構造函數創建SecureRandom ，它恰好交回了一個使用sun.security.provider.Sun作為其java.security.Provider的實例（請參閱SecureRandom#getProvider ）。 SecureRandom擴展了Random ，它是可序列化的。

Dataflow 服務在嘗試反序列化此類時會阻塞，因為它無法創建sun.security.provider.Sun 。

仔細觀察堆棧跟蹤，我看到反序列化是通過com.google.apphosting.runtime.security.UserClassLoader發生的，現在我的理論是這個類加載器不允許加載sun.*類，或者至少這個特定的sun.*類。

java.lang.IllegalArgumentException: unable to deserialize com.example.Example@13e88d
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:73)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.clone(SerializableUtils.java:88)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Bound.<init>(ParDo.java:683)
    [...]
    Caused by: java.lang.ClassNotFoundException: sun.security.provider.Sun
    at com.google.apphosting.runtime.security.UserClassLoader.loadClass(UserClassLoader.java:442)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:375)
    at java.lang.Class.forName0(Native Method)
    [...]

Answer 1

問題是sun.security.provider.Sun沒有出現在 App Engine JRE 白名單中，所以類加載器不能實例化它的實例：

https://cloud.google.com/appengine/docs/java/jrewhitelist

但幸運的是，您仍然可以在同一環境中使用new SecureRandom() 。

為了解決這個問題，我們向帶有Random字段的類添加了一個自定義的反序列化掛鈎。 簡化示例：

class Example implements Serializable {

  // See comments on {@link #writeObject} for why this is transient.
  // Should be treated as final, but can't be declared as such.
  private transient Random random;

  //
  // [Guts of the class go here...]
  //

  /**
   * Serialization hook to handle the transient Random field.
   */
  private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();
    if (random instanceof SecureRandom) {
      // Write a null to tell readObject() to create a new
      // SecureRandom during deserialization; null is safe to use
      // as a placeholder because the constructor disallows null
      // Randoms.
      //
      // The dataflow cloud environment won't deserialize
      // SecureRandom instances that use sun.security.provider.Sun
      // as their Provider, because it's a system
      // class that's not on the App Engine whitelist:
      // https://cloud.google.com/appengine/docs/java/jrewhitelist
      out.writeObject(null);
    } else {
      out.writeObject(random);
    }
  }

  /**
   * Deserialization hook to initialize the transient Random field.
   */
  private void readObject(ObjectInputStream in)
      throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    Object newRandom = in.readObject();
    if (newRandom == null) {
      // writeObject() will write a null if the original field was
      // SecureRandom; create a new instance to replace it. See
      // comments in writeObject() for background.
      random = new SecureRandom();
      random.nextDouble(); // force seeding
    } else {
      random = (Random) newRandom;
    }
  }
}