使用 F# 创建数据帧

Question

I'm trying to create a simple Spark DataFrame with F# as it is used in Spark.Net test

let schema =
    StructType (
        [|
            StructField("Name", new StringType())
            StructField("Age", new IntegerType())
            StructField("Date", new DateType())
        |] )

let rows: obj[][] =
    [|
        [| "Alice"; 20; new Date(2020, 1, 1) |]
    |]

session.CreateDataFrame(rows |> Array.map GenericRow, schema).Show(10, 0, false)

eventually, this code fails (as exactly the same code using .NET collections)

JVM method execution failed: Nonstatic method 'createDataFrame' failed for class '7' when called with 2 arguments ([Index=1, Type=GenericRow[], Value=Microsoft.Spark.Sql.GenericRow[]], [Index=2, Type=JvmObjectReference, Value=8], )

Answer 1

The above runs well for me

+-----+---+----------+
|Name |Age|Date      |
+-----+---+----------+
|Alice|20 |2020-01-01|
+-----+---+----------+

• Please check you're using Spark < 2.4.5 I did the test with 2.4.1
• Java > 1.8
• Also check you are providing all parameters

$SPARK_HOME/bin/spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin/Debug/netcoreapp3.1/microsoft-spark-2.4.x-0.11.0.jar dotnet bin/Debug/netcoreapp3.1/yourLibrary.dll