比较两个大型列表的最佳方法，C＃

Question

This is for one of my ETL project to sync two database, some table is 4G, so ETL job just load updated data to insert or update, that works fine, but the source table will delete some records, and I want to delete from my table too. What I did is:

List<long> SourceIDList; // load all ID from source table
List<long> MyIDList; // load all ID from my table
var NeedRemoveIDList = MyIDList.Except( SourceIDList );
foreach(var ID in NeedRemoveIDList)
   // remove from my table

The code logic work, but load ID from 4G table to List will through "out of memory" exception, is there better way?

Answer 1

感谢所有注释，最终我在数据库中执行了此操作，在临时表中插入了两个列表，并使用SQL进行比较，花了一些时间来插入数据，但是由于这是ETL作业，因此只需几分钟就可以了。