Multilevel modulation formats for robust long-haul high capacity transmission door