arcpy - Ways to Speed Up Python Scripts Running As ArcGIS Tools

This is a pretty general question. I just wondering what tips and tricks GIS programmers have used to speed up arcpy scripts that you import into the toolbox and run.

I work most everyday writing little scripts to help non-GIS users at my office process GIS data. I have found that ArcGIS 10.0 processing in general is slower than 9.3.1 and sometimes it gets even slower when running a python script.

I'm going to list a particular example of a script that takes over 24 hours to run. It's a loop that tabulates the area of a raster in a buffer for each shape in the buffer. The buffer has about 7000 shapes. I don't believe it should run this long. A

while x <= layerRecords:

    arcpy.SetProgressorLabel("Tabulating Row: " + str(x) + " of " + str(ELClayerRecords))
    arcpy.SelectLayerByAttribute_management(Buff,"NEW_SELECTION", "Recno = " + str(x))                                  # Selecting the record
    TabulateArea(Buff, "Recno", MatGRID, "VALUE", ScratchWS + "/tab" + str(z) +".dbf", nMatGRIDc)                          # Tabulate the area of the single row

    arcpy.AddMessage ("          - Row: " + str(x) + " completed")
    x = x + 1

    z = z + 1

Before anyone says it, I have run tabulate area on the entire buffer, but it produces errors if run on more then 1 record. It's a flawed tool, but I have to use it.

Anyways, if anyone has any ideas on how to optimize, or speed up this script, it would be most appreciated. Otherwise, di you have any speed up tricks for python, when used in ArcGIS?

Answer

A couple potential suggestions to help speed up your process are:

1. 
   

   Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use arcpy.MakeFeatureLayer_management("buff","buff_lyr") above your "while" loop, and then change your references below the while loop to use "buff_lyr".

   
   



2. 
   

   Process as much of your GP operations using the in_memory workspace as possible... Use arcpy.CopyFeatures_management(shapefile, "in_memory\memFeatureClass") to move your source into memory. This only works well if you have enough RAM to read all of the feature class(es) that you need into memory. Beware, however, that there are some GP operations that cannot run using the in_memory workspace (Eg: the Project tool).

   
   

From ArcGIS 9.3 online help article "Intermediate data and the scratch workspace" (note, this language was removed from the 10.0 & 10.1 help):

NOTE: Only tables and feature classes (points, lines, polygons) can be written to the in_memory workspace. The in_memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks and network datasets. Only simple features and tables can be written.

From ArcGIS 10.1 online help article "Using in-memory workspace":

The following considerations must be made in deciding to write output to the in-memory workspace:

• Data written to the in-memory workspace is temporary and will be deleted when the application is closed.


• Tables, feature classes, and rasters can be written to the in-memory workspace.


• The in-memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks, and network datasets.


• Feature datasets or folders cannot be created in the in-memory workspace.