what does color of goggles matter?

Actually I do the opposite.  I wear dark lenses (cool metalic looking ones) in the pool.  They kinda calm me and look great. 

But I very specifically wear clear ones at races in the lake.  I have found they fog up a bit and it is damn hard to see out of the water so I want all the advantages I can have.  And for anyone who says you end up looking at the sun, you are missing a great technique.  When swimming toward the sun, you can see the sun through the water; keep the sun at the same angle means you are swimming in a straight line.

I too keep a mix, clear, gray, orange/yellow, and blue. 

For some, the blue (especially the dark blue) makes it hard to see the orange bouys often used to mark the swim course.  I've not noticed this, but I haven't done an exhaustive analysis either.  I prefer darker lenses in the pool with the clear water and light bottom/sides.  If it's sunny, I like dark lenses outside, even if the water is murky.  If overcast, I'll use lighter or even clear lenses. 

I purposefully bought smoke-colored goggles for an open water lake swim a few years back, thinking it would keep the sunrise out of my eyes.  Instead, it turned the day-glo orange buoys dark green -- the same color as the lake water!  I had to remove them to sight the entire way, so it was a very long and inefficient swim. 

I would suggest bringing a few pairs with you and testing them to see if you have any issues.  Good luck!

Laurie