The product of two positive integers is $80$. If the first integer is $1$ more than three times the second integer, what is the smaller of the two integers?