将 csv 文件加入 shapefile 的快速有效方法

Question

I'm trying to join a csv file with millions of rows to a shapefile using JoinField but it's taking forever. And when the join is completed, I get 0 in all rows of the joined field. I also tried using dictionaries with UpdateCursor but the join didn't happen. Is there a better way to do this?

The JoinField code I used is:

arcpy.MakeFeatureLayer_management("mukey.shp", "mapunit")
arcpy.CopyRows_management(kvalues_path, "kvalues")   #to give the table OIDs
arcpy.JoinField_management("mapunit", "mukey", "kvalues", "mukey", "ksat_mday")

"mukey" is the common field between the csv file and the shapefile, while "ksat_mday" is the field that I want to join to the shapefile.

The dictionary with UpdateCursor code I used was for replacing a Join connecting two Feature Classes. May be the code didn't work because I was joining a csv file to a shapefile and not two Feature Classes. The code was taken from https://community.esri.com/t5/python-blog/turbo-charging-data-manipulation-with-python/ba-p/884079 .

Answer 1

Usually, such concatenating and joins are faster if you use Geopandas instead of arcpy, but since you stated that you are working with millions of rows, which implies heavy data frames you are working with here is my suggestion:

1. Geopandas and Pandas are relatively slow libraries due to implementing one core of CPU at a time, but the best new alternative is Pollar which exactly works like Pandas data frame; however, by parallelizing the workflow over all CPU cores, you can see the results in seconds rather than days. All you need to do is:

Install Polars in an environment:

$ pip install polars

or

$ conda install polars

then using this syntax convert your tables to an excel data sheet in the ongoing environment you are working with arcpy

arcpy.conversion.TableToExcel(Input_Table, Output_Excel_File, {Use_field_alias_as_column_header}, {Use_domain_and_subtype_description})

then you can easily read the tables you have and join them by polars which exactly works like Pandas and Geopandas.

in the end, make sure to regenerate your shapefiles.

arcpy.conversion.ExcelToTable(Input_Excel_File, Output_Table, {Sheet}, {field_names_row}, {cell_range})

Please let me know if you encountered any issues regarding this solution