Peggy Post says:
“White can be worn 365 days a year. The old rule about wearing white only between Memorial Day and Labor Day is a thing of the past.”
Peggy Post in Emily Post’s Etiquette, page 56
So enjoy your Labour Day folks, and don’t let those snooty friends of yours bully you into putting away your whites.