Remove sections, defined by a line or polygon, from a raster in QGIS

I have created a raster layer of heights using the TIN interpolate function. I also have another raster layer with more accurate information on the height of particular areas defined by polygons/lines. I would like to remove these sections from the TIN generated raster and replace them with the information in the second raster.

I have tried using the clipper function, but this removes the wrong part (it retains the sections I want removed). Can anyone provide some guidance on how to achieve inserting values from a section of one raster into another?

QGIS 2.6.0 on Windows.