The Fourier transform (FT) is a mathematical tool widely used in signal processing applications; however, it presents limitations when dealing with non-stationary time series. By considering the fractional powers of the FT operator, a generalized version is obtained known as the fractional FT. This transformation allows free rotations of the time–frequency plane that can be exploited for processing frequency modulated signals. This work addresses the problem of characterizing noisy, multicomponent, and non-linear frequency modulated signals through a proper order of the fractional FT, whose kernel consists of a chirp with linear frequency modulation. The estimation of the optimal fractional FT order obeys a strategy that includes the quantification of the compactness of fractional Fourier domains and the search for the order that leads to the most compact domain. For this purpose, five compactness measures are assessed in combination with four different optimization algorithms. Numerical experiments are performed on synthetic signals, generated under distinct frequency modulation conditions, and on real acoustic signals. The results reveal that the spectral second moment and the spectral entropy provide robust and reliable measures of the compactness of the fractional Fourier domain. These metrics enable an effective computation of the optimal fractional order that describes the frequency modulation content of the underlying signal. The optimization algorithms assessed in this study yield similar estimations of the optimal fractional order, yet the coarse-to-fine algorithm is more efficient in terms of computation time, followed by the particle swarm optimization algorithm. Moreover, it is verified that the strategy can be adopted for extracting dynamical information of the frequency modulation content from synthetic signals with multiple linear and non-linear components and from real acoustic data, e.g., bat and bird recordings. The extensive assessment of the signal processing strategy based on the fractional FT outlined in this work provides relevant information for exploring further applications with time series captured when studying complex and non-stationary processes, such as biological, medical, or economic systems. © 2023 Elsevier Ltd