This paper aims to improve the forecasting of electricity market prices by incorporating the characteristics of electricity market prices that are discretely affected by the fuel cost per unit, the unit generation cost of the large-scale generators, and the demand. In this paper, two new techniques are introduced. The first technique applies feature generation to the label and forecasts the transformed new variables, which are then post-processed by inverse transformation, considering the characteristic of the fuel types of marginal generators or prices through two variables: fuel cost per unit by the representative fuel type and argument of the maximum of Probability Density Function (PDF) calculated by Kernel Density Estimation (KDE) from the previous price. The second technique applies decomposition to the demand, followed by a feature selection process to apply the major decomposed feature. It is verified using gain or SHapley Additive exPlanations (SHAP) value in the feature selection process. In the case study, both showed improvement in all indicators. In the Korean Electricity Market, the unit generation cost for each generator is calculated monthly, resulting in a step-wise change in the electricity market price depending on the monthly fuel cost. Feature generation using the fuel cost per unit improved the forecasting by eliminating monthly volatility caused by the fuel costs and reducing the error that occurs at the beginning of the month. It improved the Mean Squared Percentage Error (MAPE) of 3.83[%]. Using the argument of the maximum PDF calculated by KDE improved the forecasting during the test period, where discrete monthly variations were not included. The resulting MAPE was 3.82[%]. Combining these two techniques resulted in the most accurate performance compared to the other techniques used, which had a MAPE of 3.49[%]. The MAPE of the forecasting with the decomposed data of the original price was 4.41[%].