A grid of size $7\times 7$ is filled with the numbers $1$ through $49$ in some order. The product of the numbers in each row and column is calculated. What is the least number of rows and columns that could have a product divisible by $3$?