Rasters lose quality when Georeferencing using ArcGIS for Desktop?

I've started to have a problem when georeferencing some rasters in ArcGIS 10.2.2 where the image appears in much lower quality with false colors and higher contrast. The images appearing in ArcGIS do not look like the original rasters at all. These are usually JPG images, although I think I have also had the same problem with TIFFs in the past. The display and pyramid settings are both set to use bilinear interpolation.

Sometimes this problem fixes itself after I update the georeferencing. If it does fix itself, it then sometimes goes back to the poor image quality after I zoom in or out. I haven't been able to find any rhyme or reason as to which rasters have this problem and when.

Does anyone know what is happening with these rasters or how to fix this issue?

The raster is appearing like this:

Instead of this (not shown at the same angle or scale, but you get the idea):

EDIT:

Taking your comments into account, I attempted a few more things, but still hitting a wall with this.

Running raster calculator cleaned it up a bit but didn't fully resolve the problem. This issue is also happening with rasters that are grayscale to begin with. It's also more than a contrast issue because text becomes illegible--it almost seems like it's downsampling somehow. This sometimes happens as soon as I bring the raster into the MXD.

Looking at Layer Properties > Symbology > Show: RGB Composite > Stretch, I set the Type to "minimum-maximum" and unchecked "apply gamma stretch," which seemed to help a fair amount, but it's still not looking entirely like the original image. These are also not setting I've ever turned on or off before, so I don't know how they might be changing automatically.

Did this for both an RBG and a grayscale image with the same results. I'm going to be working with dozens of rasters, so I'm trying to avoid running a ton of processes on each one.

Is pyramid information stored anywhere else besides in the .ovr file?

So, in the following image, I altered those things above in the Layer properties, rebuilt the pyramids, and here is what it looks like. Far more legible, but still not right: