Destination XL (DXLG) S-8 Registration of securities for employees

Exhibit 23.2

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statement (Form S-8) pertaining to the Casual Male Retail Group, Inc. 2006 Incentive Compensation Plan to register 750,000 shares of common stock, of our reports dated March 18, 2009, with respect to the consolidated financial statements of Casual Male Retail Group, Inc. included in its Annual Report (Form 10-K) for the year ended January 31, 2009, and the effectiveness of internal control over financial reporting of Casual Male Retail Group, Inc. filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

Boston, Massachusetts

November 20, 2009