Using Cloud Services to Improve Weather Forecasting Based on Weather Big Data Scraped From Web Sources

Abstract

Big Data (BD) scraping systems are among the recommended approaches for large-scale web data extraction. However, these systems for collecting large amounts of data face many challenges, including processing, storage, and data extraction reliability. Due to its potentials, cloud computing is becoming a viable solution to support BD scraping systems. This paper tenders a cloud based-web scraping framework for weather BD extraction and analysis. The aim is to extract weather BD from web sources, analyze this data and use it for visualization and forecasting purposes, and this by enabling elastic and on-demand resources. The framework is implemented using Selenium and Amazon Web Services and tested with Morocco weather data. Analysis of the performance and scalability of the proposed cloud-based scrapper shows that it offers greater efficiency in terms of data collection and analysis and in terms of forecast quality, thanks to the ability to take advantage of cloud resources.