EXHIBIT 23 INDEPENDENT AUDITORS' CONSENT The Board of Directors Casual Male Corp. We consent to the incorporation by reference in Registration Statements of Casual Male Corp. on Form S-8 No. 33-10385, No. 33-20302, No. 33-39425, No. 33- 59786, No. 33-59788, No. 33-59790 and No. 33-60605, and on Form S-3 No. 33- 51645, No. 333-2797 and No. 333-35923 of our report dated March 21, 2001, except for Note 12, as to which the date is December 10, 2001 appearing in the Annual Report on Form 10-K of Casual Male Corp. for the year ended February 3, 2001. Our report contains an explanatory paragraph that states that the Company has filed a petition for reorganization under Chapter 11 of the Bankruptcy Code and has a net capital deficiency, which raise substantial doubt about its ability to continue as a going concern. The consolidated financial statements do not include any adjustments that might result from the outcome of that uncertainty. /s/ KPMG Peat Marwick LLP -------------------------------- KPMG Peat Marwick LLP Boston, Massachusetts December 14, 2001