Deep reinforcement learning for vehicle-to-grid control of long-term parked EVs : a case study of Oslo airport Gardermoen

Abstract

There is a growing demand for electricity that leads to costly expansions of the grid capacity having to be made. Additionally this increase in demand also leads to the introduction of unregulated energy resources that causes greater variations in the electricity prices. Solutions needs to be developed to reduce the stress on the energy grid and the variations in the prices. One such solution is to use the batteries in electric vehicles (EVs) as batteries for the grid, also known as vehicle-to-grid (V2G). This case-study reviewed the potential use of the batteries of the long-term parked EVs at the P10 parking garage at Oslo airport Gardermoen for such V2G. To be able to realise the full potential of the EVs for V2G algorithms have to be developed that can optimize the charging and discharging of the EVs. This thesis looked at a type of reinforcement learning (RL) algorithms called double deep Q-learning (DDQL), Genetic algorithms (GA) and a simple algorithm as potential algorithms for optimizing these charging and discharging decisions. The DDQL was divided into two separate versions, one called meanDDQL, that used the mean of the price data, and one called STD-DDQL that used the shape of the price data for optimization. The algorithms were tested for the Nissan Leaf EVs, that are a part of a very limited number of EVs that can work as grid batteries. The Nissan Leafs were optimized using the electricity price data from Nordpool for the years 2019, 2020 and 2021. The tests showed that the GA was the best performing algorithm, followed by the mean-DDQL, the STD-DDQL and finally the simple algorithm. It was observed that the GA had long runtimes, but that this was not a problem for optimizing for the next 24 hours. Still for real-time optimization, or for optimization that needs to happen within seconds, then it was too slow and the DDQL algorithms were to be preferred. It was further observed that if either of the DDQL algorithms were to be used then the mean-DDQL was a better choice than the STD-DDQL as it was more robust towards no-profit periods. Using the optimal algorithm, GA, it was observed that under realistic conditions using the EVs for grid batteries would lead to an average earning of bellow 200 kr for the car owner for 2019 and 2020. For the year 2021 then the average profit was around 1500 NOK when using the GA for optimization. This showed that for a regular year then V2G used purely to earn a profit based on the daily price difference was low, and so the EVs should probably rather be optimized for power tariffs.