A set $\mathcal{S}$ of points in the $xy$-plane is symmetric about the origin, both coordinate axes, and the line $y=x$. If $(4,0)$ is in $\mathcal{S}$, what is the smallest number of points in $\mathcal{S}$?