I suspect the two known families you "do not want to rule out" are SIDH schemes and LWE schemes with no ring structure, like Frodo. At present SIDH is too slow and LWE keys are too big, but both could improve dramatically over the next several years.
Jeff