The population of a small town increased by $50\%$ in 2020, decreased by $40\%$ in 2021, increased by $20\%$ in 2022, and decreased by $x\%$ in 2023. If the population at the end of 2023 was the same as at the beginning of 2020, what is $x$ to the nearest integer?