Plant A and Plant B were once the same height. Since then Plant A has grown $20\%$ while Plant B has grown $\frac{1}{3}$ as many inches as Plant A. Plant A is now $36$ inches tall. How tall, in inches, is Plant B now?