Both phenomena occur and overlap. The main factor controlling them is particle size: scattering occurs for particles smaller than the wavelength of light, and reflection for bigger ones, but there is overlap. For reflection (and transmission) the absorptivity (Beer Lambert Bouguer law) is also important. A common error is to assume that Rayleigh scattering precludes Mie Scattering, and that a particle involved in Mie scattering can’t reflect or transmit light, but these names are not hard and fast laws and the phenomena overlap.
Rayleigh scattering occurs when particles are much smaller than the wavelength, for example N2 in the air. Rayleigh scattering is much stronger for blue light (4x) than for red light, and the affected light is scattered backwards as much as forwards, and also to the side. Thus, more blue light than red light is transmitted backwards. This is why the sky is blue: if you look at the sun, you see the light that is unaffected by scattering at all, plus the scattered component that has had a lot of the blue removed. Therefore, the sun appears yellow or orange when viewed through our atmosphere. The scattered blue light ultimately is scattered by more molecules in the air and into your eye.
This also explains why sunsets are red (one reason, anyway). The scattering effect becomes stronger and stronger as more and more molecules are encountered by the setting sun’s rays passing obliquely through the atmosphere. Therefore, the lower the sun, the more reddened it becomes because the blue light has been preferentially scattered away. The effect is most noticeable for humid atmospheres with lots of water vapor.
As particle size increases to approach the wavelength, Mie scattering takes over and dominates. In Mie scattering, the wavelength dependence is much lower, and so is back-scattering and side-scattering. Scattering in the forward direction is dominant. This is why an aerosol of very fine dust or smoke particles may make the sun setting in the desert, with dry air, appear white, or be surrounded by a white halo. In general, pure Mie scattering causes the air to be whitish, but smoke can make the air seem bluish (as in the volcanic haze from Kilauea).
There is considerable confusion about Mie scattering and “Non-selective scattering” in the literature and on the net. Mostly, it is claimed that Mie scattering affects particles larger than the wavelength of light, and this is confused with “non-selective scattering” from clouds of water droplets. True enough, but Mie also ranges down to particles smaller than light. So it covers a wide range. Also, for really big particles, the composition of the particle affects the color of the “scattered” light. Here, what really happens involves reflection from and transmission through big particles. This imbues the light with the color of the particle. And is why dust clouds may appear yellowish.
Back to water clouds: the key observation is that clouds even appear white at wavelengths where water appears dark because of absorption (for example, snow appears dark in the near infrared). So it must be non-selective Mie scattering, and not reflection, that controls the color of clouds.
But Mie scattering is forward scattering. Why do clouds look white when we look away from the sun? Another factor is at work here – the density of the particles in the air. With clouds, there are lots of droplets, so the light forward-scattered soon strikes another droplet. Because the light isn’t exclusively scattered in the exact same direction it was traveling before it hit the cloud, eventually – after many collisions – some will end up going to the side, and some back the way it came. Thus, light from a cloud is diffusely “reflected” and is also bright in the NIR, even though absorption of NIR light is strong. Also, most of the light from these small droplets is mainly reflected from surfaces, and never gets into the droplet where it can be absorbed. Light reflected from the surface (think: mirror) is not colored.
The above paragraph shows why there is a lot of confusion about non-selective scattering in the literature and on the web. It is a comnplicated subject in a cross-over size range and involves several processes.
Again, for larger aerosols, the material properties cause light to be differentially absorbed, such that dust clouds appear yellow.
How light is scattered at different wavelengths as a function of particle size
OK, again about red sunsets:
First, Rayleigh scattering makes the sun redder on the horizon, and Mie scattering gives it a whitish appearance. Humid air increases Rayleigh scattering, and dust will make it reddish
There is another phenomenon, refraction. Light rays are bend during passage through the air, blue more than red. So as the sun sets, blue rays are reflected down but red rays are reflected less so. They will reflect of clouds to make the sky reddish at sunset. They will also scatter from smaller particles n on-selectively to cast scattered light onto the viewer. (Forward scatter of the reddened sunlight will make the sun itself appear red, too.
The processes are simple, their manifestations complex.
Relative amount of scattering (Log)
Mie: d<l to d>l
Envelopes are for Rayleigh scattering
Probability that light will go in a given direction is given by the distance from the particle to the envelope
Not all light is scattered – some passes through the atmosphere unaffected