Weather as a service: How the UK’s Met Office is creating big data clouds
Illustration: Stuart Daley
Share on LinkedIn
Share on Xing

Weather as a service: How the UK’s Met Office is creating big data clouds

Jim Mortleman – May 2016

With a growing commercial focus, the UK government’s official weather forecaster is seeking to evolve its big data services into a set of cloud-based platforms, highlights Met Office CIO Charles Ewen.

Despite its public face as the UK’s official weather forecaster, the Met Office likes to set its agenda in a much wider context. As CIO Charles Ewen puts it, the agency’s modelling of weather and climate is actually about “protecting life and prosperity, enhancing wellbeing and supporting economic growth.”

To attain such lofty goals — and create the 3,000 tailored forecasts and briefings it delivers each day — the Met Office needs to process a huge amount of data — half a billion discrete observations a day, with more than 200 million alone used to predict the weather, says Ewen. And according to the CIO its global coverage puts the Met Office in the running as the world’s largest monitor of weather.

In keeping with Moore’s Law, the data it deals with is doubling every few years, and the problem Ewen faces is that traditional IT models simply break down when dealing with this kind of volume. So providing valuable insight to clients — from the UK military engaged in foreign operations to consumer goods companies keen to know likely ice-cream sales and energy companies planning their electricity supply — is increasingly challenging.

“Think of it like gravity. When data grows this big, it starts to attract more data because it’s far easier to move smaller things to the bigger data than to move the big data elsewhere.”

“We’re facing issues today that won’t hit the bulk of organizations for another five or 10 years,” says Ewen. “Techniques that work with smaller numbers – even if it’s still ‘big data’ – start to get seriously challenging. The old file-based integration, store-and-forward model – taking a data set and distributing it to many places – just doesn’t work anymore,” he says. “Think of it like gravity. When data grows this big, it starts to attract more data because it’s far easier to move smaller things to the bigger data than to move the big data elsewhere.”

As a result, Ewen says the Met Office is focusing on strategies that will stop it “drowning in data, yet thirsting for information. We want the techniques that maximise the information content and minimise the data content.”

To do this the organization considers its use of IT in terms of three information planes: What happened? Why did it happen? What will happen? “The ‘what happened?’ plane is fuelled by the observation network of land, sea and satellite stations across the world.

The ‘why did it happen?’ plane covers the Met Office’s huge scientific research program employing around 1,000, which is further magnified by international collaboration.
API family
For those using the Met Office’s services, the real interest lies in ‘what will happen?’ Ewen highlights how the Met Office is increasingly making use of cloud platforms and APIs (application programming interfaces) to open up its information and provide its clients with decision-making tools and services.

As such, Ewen says the Met Office is in the process of recasting many of its services as cloud platforms and has recently started using API management software from CA Technologies to create a family of interfaces that will securely open up key data sources to partners, developers, mobile apps and other cloud services. “Increasingly, we’ll be pitching ourselves as a platform as a service, big data as a service organization,” he says.

The organization is also looking to a next generation of APIs that will allow it to create increasingly intelligent data services. “This allows you to take a problem and bring it back to the data by using technologies such as blockchain, secure containers and Docker. Essentially, these technologies can take IP, lock it in a box, move it and operate it against a data set and securely send the output for use elsewhere,” he explains. He cites the example of a connected home asking the Met Office when to switch on its heating services.
Crowdsourced forecasts

Internet of Things applications of all kinds are set to have a big impact on Met Office data collection. As the use of sensors proliferates, there will be a growing wealth of third-party observational data with the potential to enrich and expand the organization’s range of services. It also sees huge potential in crowdsourced data.

It has big plans for its existing Weather Observations Website (WOW); an open cloud-based database fed by weather observations from the public and interested third parties. “We realized the significance of crowdsourced observations for our organization five years ago and created the WOW site, which now contains around five billion observations,” Ewen says. “We’re in the processes of re-engineering the platform, which will feature a full set of rich APIs. And because it’s an open platform, anyone else can combine WOW data with their data to produce a whole range of innovative services and applications.”

In particular, he encourages organisations involved in smart city projects to connect. As well as exploiting their data in closed networks they should also be thinking of “pumping smart city data into this open network because for us that data adds huge equity to the service and will benefit the whole of the UK.”

• Charles Ewen was speaking at IoT Tech Expo 2016 in London.

First published May 2016
Share on LinkedIn
Share on Xing

    Your choice regarding cookies on this site

    Our website uses cookies for analytical purposes and to give you the best possible experience.

    Click on Accept to agree or Preferences to view and choose your cookie settings.

    This site uses cookies to store information on your computer.

    Some cookies are necessary in order to deliver the best user experience while others provide analytics or allow retargeting in order to display advertisements that are relevant to you.

    For a full list of our cookies and how we use them, please visit our Cookie Policy

    Essential Cookies

    These cookies enable the website to function to the best of its ability and provide the best user experience for you. They can still be disabled via your browser settings.

    Analytical Cookies

    We use analytical cookies such as those used by Google Analytics to give us information about the way our users interact with - this helps us to make improvements to the site to enhance your experience.

    For a full list of analytical cookies and how we use them, visit our Cookie Policy

    Social Media Cookies

    We use cookies that track visits from social media platforms such as Facebook and LinkedIn - these cookies allow us to re-target users with relevant advertisements from

    For a full list of social media cookies and how we use them, visit our Cookie Policy