At noon on a certain day, Town X is $N$ degrees warmer than Town Y. At $3:00$ PM the temperature in Town X has fallen by $4$ degrees while the temperature in Town Y has risen by $6$ degrees, at which time the temperatures in the two towns differ by $1$ degree. What is the sum of all possible values of $N$?