Thursday 4 May 2017

python - How to merge many shapefiles in different folders?


Directory structure is basically as such: county\City\prep\POI\Completed\C1, ...C2, ...etc\


I'm trying to write a script to go through the *\Completed*\POIS.shp and merge them all together, at a country level. So the input would be the country folder as root.


So, to clarify:


county1\City1\prep\POI\Completed\C1\POIS.shp


county1\City1\prep\POI\Completed\C2\POIS.shp


county1\City2\prep\POI\Completed\C1\POIS.shp



etc


I started to write out the walk through the folders, but I figured I should get some guidance first. So now I've got it working so if the merged files are in a specific folder I can easily run the XY coordinates and other selections I need to do. The step just above where I've commented out should be the merge of all POIS.shp files and copied to the directory to run the other tools. Hopefully (but probably not) not confusing.


import arcpy
import os
import sys


# Folder name to run XY script
folder = arcpy.GetParameterAsText(0)
##countryFolder = arcpy.GetParameterAsText(1)

# workspace
arcpy.env.workspace = folder

# List shapefiles
arcpy.AddWarning("Building List of Shapefiles...")
fcList = arcpy.ListFeatureClasses()

## Merge a Country's completed POI files (Country\*\PDA\Completed_Field_Files\*\POIS.shp\)
##for dirs in os.walk(countryFolder)
## arcpy.Merge_management("POIS.shp",


# Add coordinates # code is good and works!
try:
for fc in fcList:
arcpy.MakeFeatureLayer_management(fc, "temp_" + fc)
shapeFile = "temp_" + fc
# Add XY coor to table
arcpy.AddXY_management(shapeFile)
arcpy.AddWarning("Building" + " " + shapeFile + " " + "XY Coordinates Successful")
# Select only records we want with SQL: Action<>D; Position=Yes,NA; POI_ST_NUM<>blank

arcpy.AddWarning("Selecting" + " " + shapeFile + " " + "Records...")
try:
arcpy.SelectLayerByAttribute_management(shapeFile,"NEW_SELECTION","POI_ST_NUM <> ' '")
except:
arcpy.AddWarning("No Blank Street Numbers")
try:
arcpy.SelectLayerByAttribute_management(shapeFile,"REMOVE_FROM_SELECTION","ACTION_COD = 'D' OR ACTION_COD = 'FR' OR ACTION_COD = 'AFR'")
except:
arcpy.AddWarning("No Action Codes to Remove")
try:

arcpy.SelectLayerByAttribute_management(shapeFile,"REMOVE_FROM_SELECTION", "POSITION = 'NO' OR POSITION = ' '")
except:
arcpy.AddWarning("No Position Errors to Remove")
# Copy selected records in feature claseses to new shapefile
arcpy.AddWarning("Creating New Shapefile...")
arcpy.CopyFeatures_management(shapeFile, "RWT_XY_" + fc)
arcpy.AddWarning("***********************")
arcpy.AddWarning("Huzzah! File/s Complete")
arcpy.AddWarning("***********************")
except:

arcpy.AddError("*********************")
arcpy.AddError("DOH! Adding XY Failed")
arcpy.AddError("*********************")

Thanks! By the way, this is my 2nd script ever with no previous training or knowledge, so be easy on me. :-D


EDIT: try to make it more clear


I have dozens POIS.shp files I want to merge, but they are all located in different folders according to the structure I listed above. Always in a C1, C2, C3, ..., C16 folder. The C folders are always in Completed_Field_Files folder. The structure may differ a little bit, so I want the tool to crawl and merge all POIS.shp files within the Completed_Field_Fieles\C# folders. There's other POIS.shp files as well, so I ONLY want the ones within those specific folders. I'm basically just merging a bunch of shapefiles that are located in different but specific folders



Answer



Something like this (untested) should put you on the right track:


import os

import arcpy
import fnmatch

def listCountyPOIs(county_folder):
pois = []
for dirpath, dirnames, filenames in os.walk(county_folder):
for filename in filenames:
fullpath = os.path.join(dirpath, filename)
if fnmatch.fnmatch(fullpath, r"*\Completed_Field_Files\C*\POIS.shp"):
pois.append(fullpath)

return pois

if __name__ == "__main__":
countyFolder = arcpy.GetParameterAsText(0)
outputShapefile = arcpy.GetParameterAsText(1)
countyPOIs = listCountyPOIs(countyFolder)
arcpy.Merge_management(countyPOIs, outputShapefile)

A more generic version of this function is given below. This is a generator function.


def matchFiles(root_dir, pattern):

for dirpath, dirnames, filenames in os.walk(root_dir):
for filename in filenames:
fullpath = os.path.join(dirpath, filename)
if fnmatch.fnmatch(fullpath, pattern):
yield fullpath

Usage (using the OP's example):


pattern = r"*\Completed_Field_Files\C*\POIS.shp"
countyPOIs = list(matchFiles(countyFolder, pattern))

No comments:

Post a Comment

arcpy - Changing output name when exporting data driven pages to JPG?

Is there a way to save the output JPG, changing the output file name to the page name, instead of page number? I mean changing the script fo...