[英]Deserialize non-standard JSON-like format in C#
I am trying to deserialize a file containing structured data with C# to export to CSV and insert into an MSSQL Database in the end.我正在尝试使用 C# 反序列化包含结构化数据的文件,以导出到 CSV 并最终插入 MSSQL 数据库。
The file looks a bit like JSON but only the individual lines are valid JSON.该文件看起来有点像 JSON,但只有个别行是有效的 JSON。
Example:例子:
{"field1": "value1", "field2": "value2", "field3": "value3"}
{"field1": "value4", "field2": "value5", "field3": "value6"}
{"field1": "value7", "field2": "value8", "field3": "value9"}
I tried to use Newtonsoft.Json looping over the individual lines but the execution takes a very long time this way as the file given to me is very large (multiple million lines).我尝试使用 Newtonsoft.Json 在各个行上循环,但是执行需要很长时间,因为给我的文件非常大(数百万行)。
Alternativley I tried making this entire thing valid json altering the string using a StringBuilder but that results in loading this entire file into RAM at once which does not seem to be a reasonable option at all. Alternativley 我尝试使这整个事情有效 json 使用 StringBuilder 更改字符串,但这会导致将整个文件一次加载到 RAM 中,这似乎根本不是一个合理的选择。
I even thought about splitting the file into chucks which are then processed concurrently but I assume there must be a cleaner option to do this?我什至考虑过将文件分成卡盘,然后同时处理,但我认为必须有一个更干净的选项来做到这一点?
Is this a format that is in some way standardized?这是一种在某种程度上标准化的格式吗? What would be the smartest way to go about importing this?
go 关于导入这个的最聪明的方法是什么? Should I skip the CSV and use C# to directly insert the data into the datebase?
我应该跳过 CSV 并使用 C# 直接将数据插入数据库吗?
Any help appreciated!任何帮助表示赞赏!
Given the size of the file, I'd be tempted to create a custom stream (see Implement custom stream ) that wraps the real file stream such that the first few bytes returned were "{items: [", after that the bytes for the real stream and at the end "]}".鉴于文件的大小,我很想创建一个自定义 stream (请参阅实现自定义 stream )来包装真实文件 stream真正的 stream 和末尾的“]}”。 That would mean you could use your Newtonsoft.JSON library without trying to build the entire corrected JSON in memory.
这意味着您可以使用您的 Newtonsoft.JSON 库,而无需尝试在 memory 中构建整个更正后的 JSON。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.