Data-Backed Methods for Local Weather Prediction

We all like to think we're pretty good at building intuition about the world around us. After all, that's something we as humans do best; we're great learners by nature.

But it's also healthy to challenge our assumptions from time to time. Let's not forget, for example, that humans used to think the Earth was both flat and the center of the universe. With a little data and an openness to new ideas, our forebears eventually saw beyond the most convenient way to consider the cosmos and accepted a different reality.

Mining Data For Useful Generalizations

Today, we have access to massive amounts of data to help us challenge our assumptions and keep each other honest. It's not just about Googling facts when an argument comes up at dinner; we can also use data to refine the day-to-day generalizations that help us make sense of the world and dictate how we spend our time.

We've realized that we make a lot of daily assumptions about the weather in particular. Sometimes those assumptions are informed (like checking a weather app when you wake up to decide what to wear or leveraging knowledge from the Farmer's Almanac for gardening), but more often, our assumptions are based on either folklore (like Groundhog Day) or what data scientists call heuristics: broader first-hand observations and patterns we recognize over time.

There are plenty of old wives' tales about how to predict the weather via local observation (like leaves turning over before it rains or hairs curling up when it's humid), but we can go beyond that to crunch some actual numbers, since there is detailed weather information stretching back decades for much of the world (from sources like the Dark Sky API). How can we leverage those numbers to act on what we think we already know?

Hack #1 - Clouds Before Rain

We can safely assume that it's cloudy while it rains. So first things first, let's double-check what we think we know and see what the data show us.

The graph above shows data from 10 cities across the US with varying climates, elevations, and geographic influences. With 5 years of data recorded hourly and 4x interpolation, that gives us over 1.75 Million data points to work with (each of which contains 12-15 sensor values). Specifically, the graph shows the cross correlation between cloud cover and precipitation intensity as time is shifted +/-24 hours.

Clearly there's a peak in the center, meaning that it is in fact cloudiest when it's raining. Success!

But clouds come and go all the time, so the more interesting insight is how quickly the clouds roll in before it rains. Let's consolidate all the information down to the average for all locations and look at the weighted cloud cover before it rains.

Now we're getting somewhere. Not only does it get cloudy when it rains, but the rate at which it gets cloudy (cloud acceleration) increases as the storm approaches. This is especially true in the 4-6 hours before rainfall.

Pro Tip: When the cloud cover seems to accelerate, expect rain within 4-6 hours.

Hack #2 - Wind Before Rain

We can also look at wind speed before a storm, like we did with cloud cover. Since the magnitude and direction of wind is highly variable between storm fronts and geographic locations, we expect the data to be a bit more bumpy and scattered.

If you squint, you can see that while each location has its own characteristics, there are some significant similarities in the overall shapes. Specifically, notice the small spike in wind speed shortly before it rains (at time = 0 on the x axis).

By averaging all of the locations, the emergent shape becomes much clearer. The mean wind speed picks up in the 4-8 hours ahead of a storm front, then quickly falls back down in the hour before it actually starts to precipitate.

We can also look at wind bearing (direction the wind is blowing) before and after it rains.

The direction the wind blows depends heavily on location, but in the US most wind blows west to east (180 degrees on this graph). Since all locations' dominant wind direction falls within +/-90 degrees of each other, we can average them to look for trends without introducing significant artifacts.

The average wind direction before and after it rains rounds out our intuition about stormy air movement. Not only does the wind speed pick up prior to wet weather, but the wind direction also changes by roughly 30 degrees before and after a storm (usually toward the north). And it doesn't take anything fancy to detect wind direction; just pin a straw to the top of a pencil or lick your finger and hold it up in the air.

Pro Tip: When it's been windy, then calms down and changes direction, expect rain within an hour.

Hack #3 - Temperature Before Rain

We also imagined that the temperature decreases when rain is on the way. But like curious cats (who, by the way, also have fine temperature sensitivity), we wondered: how much of that is feeling cold because of darker sky and increased wind, versus actual temperature change?

The graph above shows the temperature change for 5 years of data across 10 locations in the 24 hours before and after it rains. Each location has different temperature norms, so we expect them to be at different levels, but once again we can combine them and focus on their shape to look for patterns.

The drop in temperature is significant, with steady cooling of around 4 degrees Fahrenheit in 4 hours (with roughly 2 degrees falling in the final hour or so). That's enough to feel a chill in the air, even without the wind and overcast skies. Also notice that there's typically a minor upswing in temperature in the 4-6 hours before the drop, amplifying the feeling of rapid temperature change.

We can also see what the dew point is doing while the temperature fluctuates to get a sense of how humid or muggy it feels. Notice that the dew point rises up like a ramp before the rain and then steadily falls away. When the temperature and dew point come closer together, the humidity will be higher, as the air is increasingly saturated with water.

Pro Tip: When the temperature quickly drops and the humidity rises, expect rain in the next couple of hours.

Hack #4 - Predicting Sunshine

While it seems most helpful to use weather data to better predict precipitation, we can use the same approach to predict clear, sunny skies. To do this, we created an anti-precipitation metric called clear sky intensity that factors in both a lack of rain and cloud cover.

We looked at two major factors for sunny skies: temperature and wind speed.

Naturally, the temperature typically climbs once skies are clear, but there is also an interesting clue beforehand - the shallow dip. We expect the temperature to rise and fall more or less in sync with days and nights, but the data suggest that an increased low temperature value (the minimum for those 24 hours) can be used as a predictor of clear skies on the way.

We were also able to see a marked change in wind speed before the skies cleared up. While the average effect is small (only about 0.3 MPH difference), the pattern is compelling, particularly when used alongside other information. As wind speed decreases over a 12-hour window, clouds are likely to subside with it.

Pro Tip: When low temps increase and winds die down, expect clear skies in the next 3-6 hours.

Hack #5 - Leveraging Pressure

While most of the major weather data recorded can be easily seen or felt, there is one that stands out as more elusive and abstract - barometric pressure.

Barometers have been used for centuries because of their apparent ability to "know" more about the upcoming weather for a single location than anything else. That's because weather patterns move around in high and low pressure systems that often take days to pass over an area.

Being able to sense those tiny fluctuations in pressure means being able to tell when a change in weather is on the way.

Notice how a dip in pressure is correlated with the rain intensity above, though the magnitude and exact timing of the pressure change is dependent on location.

By taking the average across all locations and looking at the relationship between rain and the preceding pressure, we can get a much better picture of what's going on. Generally, there is a significant decrease in pressure 3-6 hours before the rain starts coming down.

The opposite is also true; a significant increase in pressure is associated with warmer, clearer weather. The graph above illustrates this by showing the average pressure before clear weather.

So for the old-school weather aficionado, keeping tabs on a barometer's reading over time may be the best way to predict what's on the way, especially if the power goes out (or if you're entertaining sea captains!).

Pro Tip: When the barometric pressure dips quickly, expect rain in 3-6 hours. When it climbs steadily, look for sun in 6-24 hours.

Posted on   2016-09-28
Filed under  
Life Hack
Big Data

All hacks posted by Makefast Workshop are open source and shared without any strings attached for your amusement, use, and continued experimentation.

Happy hacking!