Due to increasing numbers of intermittent and distributed generators in power systems, there is an increasing need for demand responses to maintain the balance between electricity generation and use at all times. For example, the electrification of transportation significantly adds to the amount of flexible electricity demand. Several methods have been developed to schedule such flexible energy consumption. However, an objective way of comparing these methods is lacking, especially when decisions are made based on incomplete information which is repeatedly updated. This paper presents a new benchmarking framework designed to bridge this gap. Surveys that classify flexibility planning algorithms were an input to define this benchmarking standard. The benchmarking framework can be used for different objectives and under diverse conditions faced by electricity production stakeholders interested in flexibility scheduling algorithms. Our contribution was implemented in a software toolbox providing a simulation environment that captures the evolution of look-ahead information, which enables comparing online planning and scheduling algorithms. This toolbox includes seven planning algorithms. This paper includes two case studies measuring the performances of these algorithms under uncertain market conditions. These case studies illustrate the importance of online decision making, the influence of data quality on the performance of the algorithms, the benefit of using robust and stochastic programming approaches, and the necessity of trustworthy benchmarking.