arcgis desktop - Average of raster elevation based on polygon boundaries

I am attempting to extract and calculate the average height of buildings using LiDAR converted to raster elevation data. I've created a DSM to cover the entire area and I am trying to extract just the buildings out, I need the average height based upon the DSM elevation pixels. How do I calculate this?

I've attempted clipping these images out according to the polygons (which created thousands of individual rasters) but still I am unsure how to get the average within the bounds of the footprint.