Plant A and Plant B were once the same height. Since then Plant A has grown $25\%$ while Plant B has grown $\frac{1}{4}$ as many inches as Plant A. Plant A is now $40$ inches tall. How tall, in inches, is Plant B now?