Extreme learning machine (ELM) has aroused a lot of concern and discussion for its fast training speed and good generalization performance, and it has been used diffusely in both regression and classification problems. However, on account of the randomness of input parameters, it requires more hidden nodes to obtain the desired accuracy. In this paper, we come up with a firefly-based adaptive flower pollination algorithm (FA-FPA) to optimize the input weights and thresholds of the ELM algorithm. Nonlinear function fitting, iris classification and personal credit rating experiments show that the ELM with FA-FPA (FA-FPA-ELM) can obtain significantly better generalization performance (such as root mean square error, classification accuracy) than traditional ELM, ELM with firefly algorithm (FA-ELM), ELM with flower pollination algorithm (FPA-ELM), ELM with genetic algorithm (GA-ELM) and ELM with particle swarm optimization (PSO-ELM) algorithms.