Building Large Network Datasets from OpenStreetMap in ArcGIS?

I'm trying to build a "large" network dataset from OSM data in ArcGIS. Doing it all at once is taking a "long time" and seems like there is a big opportunity for it to fail at some point which would cause me to have to start over. I would like to be able to ingest smaller dataset so that the processing would not take so long.

Is it possible to merge multiple smaller network datasets to a single dataset (that doesn't require rebuilding the entire network) that I can make routing calls to? Or maybe to do routing calls across multiple network datasets?

For reference the dataset I used was an OSM extract for all of California. It is 11GB of xml. It took me 4.5 days to turn that into feature classes. It took another 5.5 days to create the network dataset. The gdb is now 19.6GB. This was running on a m3.medium. I would eventually like to be able to route anywhere in the US.

Answer

As an answer to both Uffe Kousgaard comments about "what the 18GB file contains" compared to a routable shapefile, and a possible answer to this question:

• You don't explicitly state it, but I guess you used the ArcGIS Editor for OpenStreetMap to convert your data. If not, I really recommend to have a look at it, as it contains a dedicated option to create routable networks of the converted data.


• The Editor by default converts all data of the OSM file, meaning you end up not only with all highway line elements as necessary for building your network, but all other stuff like landuse and building polygons as well. The line Feature Class will contain other stuff as well, like waterways. This explains the 11-18GB data size compared to a lean routable highway network.


• You could therefore consider to select features with a highway=x only, and export these to a dedicated Feature Class, preferably in a separate File Geodatabase as well. Please note (I have never created a routable network myself using the tools in the ArcGIS Editor for OpenStreetMap), that you may need to export other features as well, depending on what features are required by the Editor to build a routable network.


• If you are just interested in routing for vehicles, discarding highway=footway, highway=bridleway, highway=cycleway, highway=pedestrian, highway=steps and highway=path should sharply reduce the size of the routable network as well.


• I would strongly recommend to mildly generalize your features before creating a network. This will discard a lot of unnecessary detail / vertices / nodes that don't add anything to the routable network besides ballast. However(!), if you do this, you must use the options for maintaining topology during generalization. The Simplify Line tool of ArcGIS has an option for this. You must set:
  
  
  
  1. BEND_SIMPLIFY if you want to maintain the initial shapes as accurately as possible, POINT_REMOVE for speed.
  
  
  
  2. Set CHECK for the "error_checking_option"
  
  
  3. Set RESOLVE_ERRORS for the "error_resolving_option". Please note that this option can significantly increase generalization processing time, but since it is vital for maintaining the topology of the routable network, you must use it.
  
  
  4. Set an appropriate tolerance. You could start with something like 25 meter as a tolerance. 25 meters may sound like a lot when considering e.g. detailed small junctions or roundabouts, but please note that since you will select RESOLVE_ERRORS, ArcGIS will iteratively reduce this tolerance whenever connecting features would become topologically broken, until the generalized features are topologically correct while still somewhat generalized. So even setting values above 25 meters (e.g. 100 meter), should still retain basic topology of the network using the RESOLVE_ERRORS option. However, even setting a tolerance of 25 meters, is likely to reduce your Feature Class's size by maybe something like 90%.