[英]Using SqlBulkCopy with F# to export matrix in SQL
I want to transfer a large amount of data from F# to an SQL table.我想将大量数据从 F# 传输到 SQL 表。 Basically my F# code creates a matrix of three columns (
UserID, ProductID and price
) and N lines.基本上,我的 F# 代码创建了一个包含三列(
UserID, ProductID and price
)和 N 行的矩阵。 I would like to "copy/pate it" into a database I tried several options but at the end, the transfer of data from F# is really slow (around one hour for 10000 lines).我想将它“复制/粘贴”到一个数据库中,我尝试了几个选项,但最后,从 F# 传输数据真的很慢(10000 行大约一小时)。
Thanks to answers of a previous question How to include a stored procedure in F# , an interesting way to resolve this problem is to use SqlBulkCopy
.感谢上一个问题如何在 F# 中包含存储过程的答案,解决此问题的一个有趣方法是使用
SqlBulkCopy
。
SqlBulkCopy
requires a database type for its WritetoServer
method but I didn't find any existing code or simple way to convert a matrix into a database. SqlBulkCopy
的WritetoServer
方法需要数据库类型,但我没有找到任何现有代码或将矩阵转换为数据库的简单方法。 Do you have any suggestions or ideas?你有什么建议或想法吗?
This should get you started (and reference the documentation for more information about SqlBulkCopy
):这应该让您开始(并参考文档以获取有关
SqlBulkCopy
更多信息):
//you must reference System.Data and System.Xml
open System.Data
open System.Data.SqlClient
let bulkLoadUserPurchases (conn:SqlConnection) (userPurchases: list<int * int * float>) =
use sbc = new SqlBulkCopy(conn, SqlBulkCopyOptions.TableLock, null, BatchSize=500, BulkCopyTimeout=1200, DestinationTableName="YOUR_TABLE_NAME_HERE")
sbc.WriteToServer(
let dt = new DataTable()
["UserID", typeof<int>
"ProductID", typeof<int>
"Price", typeof<float>]
|> List.iter (dt.Columns.Add>>ignore)
for userPurchase in userPurchases do
let userId, productId, price = userPurchase
let dr = dt.NewRow()
dr.["UserID"] <- userId
dr.["ProductID"] <- productId
dr.["Price"] <- price
dt.Rows.Add(dr)
dt)
If you're okay with third party libraries, fastmember can be really useful here.如果您对第三方库没问题, 那么 fastmember在这里非常有用。
module DB =
open System.Data
open System.Data.SqlClient
open FastMember
// val bulk : conn:SqlConnection -> table:string -> columns:seq<string*string> -> items:seq<a'>
let bulk conn table columns items =
use bcp = new SqlBulkCopy(connection = conn)
bcp.EnableStreaming <- true
bcp.DestinationTableName <- table
for n,v in columns do
bcp.ColumnMappings.Add(new SqlBulkCopyColumnMapping(sourceColumn = n, destinationColumn = v))
|> ignore
bcp.WriteToServer(items |> ObjectReader.Create)
This leaves you with a generic method that you can just funnel any seq<a'>
into.这为您留下了一个通用方法,您可以将任何
seq<a'>
到其中。 Along with some config parameters, which you can of course configure to your needs.连同一些配置参数,您当然可以根据需要进行配置。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.