openlayers - Open Layers 4 fit to extent zoom too much out

Working with OpenLayers 4.2.0, I have the extent [-45930163.095729224, -3380418.677804029, -5390354.729126186, -3011487.2342575104] which is a box around Santa Catarina state, Brazil.

But instead of having the map center in this box, I'm actually being centered in Australia, which would be an outer extend, if that is the right word for it, such as the map center as:

My code is (ignore the extent visibility and the condition for "Infinity"):

        extent = [-45930163.095729224, -3380418.677804029, -5390354.729126186, -3011487.2342575104];

        if (extent[0] != "Infinity") {
            map.getView().fit(extent, {size: map.getSize(), padding: [15, 0, 0, 0]});
        }

I've already tried to use a size different than map.getSize(), but all I get is the map zoomed in Australia.

Since I can't comment, @bartvde: It didn't helped too much. Kind of zoomed in, but the center is still in Australia.