As a former electronic retoucher, I see cyan-tinged blue and a muddy, slightly orange brown... and a terrible photography job overall. There is definitely zero black and zero white in this image!!!
I suspect that some people mentally color-correct the light blue to white, which shifts the muddy orange to gold, while others mentally color correct the muddy orange-brown to black, which then shifts the light blue to a strong blue. While all of us mentally color-correct all the time, this photo is really odd since you can reasonably perform either mental color correction and come up with a workable picture.
Ultimately, it's got to be something like what you said. This video sort-of explains it (takes only a minute if you jump ahead to the 8:00 minute mark):
It attributes the effect to how your brain interprets shadows, and it gives a different example to illustrate.
It's a fun effect. The dress image must be near a a threshhold as to how our brains interprets color in shadows.
That said, if you're only able to see it one way, why would that be? Is it the color calibration on your monitor, or overall brightness level, and/or something else (maybe even the level of background lighting)? When I look at it on my living room HDTV, it only looks blue-black, no matter from where or what angle I look at it, or even if I move it around the screen (Patrick's technique) to block portions of the image. Or perhaps the image size (bigger on the HDTV) is also a factor? On my laptop, I can see it either way. I'm not sure I've yet seen a complete explanation of all the factors that might be involved (or ruled-out) in the dress image.