tile map service - GDAL TMS (WMS) mini driver

I've a problem with GDALs TMS mini driver. I've created an XML file, which I can open in QGIS. I am using local tiles on my computer which I pre-rendered with Mapnik, the map covers entire country of Austria and the XML looks like this:

    
        file:///f:/austria/map/${z}/${x}/${y}.png

        EPSG:3857
        image/png
    
    
        -20037508.34
        20037508.34
        20037508.34
        -20037508.34
        17
        1

        1
        top
    
    EPSG:3857
    256
    256
    3
    204,303,400,404,500,501
    true

The problem is: Whenever I zoom/pan into such window extents that is not completely covered with my tiles, the result is a scrambled image like this:

When I zoom/pan to an area covered with my tiles, the result is OK:

What I think is the issue: GDAL cannot merge and display the map correctly if there are missing tiles. If I was loading tiles from an http source, the server would probably return http request codes like 404 or 204. When configuring the TMS service in the XML, you have an option to define what to do when a tile is not returned/can't be found using these 2 tags:

204,303,400,404,500,501
true

The problem is, I don't know what is the response when accessing locally stored tiles and some are not found, or how do you specify what to do in such a case. I was also thinking of limiting the extent of the layer, but there are no options in the XML syntax for doing so.