EXHIBIT 23.1


                         CONSENT OF INDEPENDENT AUDITORS


We  consent to the  reference  to our firm under the  caption  "Experts"  in the
Amendment No. 1 to the Registration Statement on Form S-3 and related Prospectus
of Casual Male Retail Group, Inc. (formerly, Designs, Inc.) for the registration
of 24,809,338  shares of its common stock and to the  incorporation by reference
therein of our report  dated March 11, 2002,  with  respect to the  consolidated
financial statements and schedule of Designs, Inc. included in its Annual Report
(Form 10-K) for the year ended  February 2, 2002,  filed with the Securities and
Exchange Commission.


                                                           /s/ ERNST & YOUNG LLP

Boston, Massachusetts
August 20, 2002