On Thanksgiving, Sam ate $\frac{1}{4}$ of his pies and then gave $3$ pies to his grandmother. The next day he ate $\frac{1}{2}$ of his remaining pies and then gave $5$ pies to his grandfather. On the third day he ate his final $10$ pies. How many pies did Sam have at the beginning?