In a math competition, Student P contributed $\frac{1}{3}$ of the team's total score, and Student Q contributed $\frac{3}{8}$ of the total score. Student R scored $20$ points. The remaining $4$ team members each scored no more than $1$ point. What was the total number of points scored by the remaining $4$ team members?