At noon on a certain day, City A is $N$ degrees colder than City B. At $5:00$ PM the temperature in City A has risen by $7$ degrees while the temperature in City B has fallen by $2$ degrees, at which time the temperatures in the two cities differ by $4$ degrees. What is the product of all possible values of $N$?