Exhibit 23.1 CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM We consent to the incorporation by reference in Registration Statements No. 33-86312 and No. 333-53750 on Forms S-3/S-8 of our report dated March 24, 2005 relating to the financial statements and financial statement schedule of Hampshire Group, Limited, appearing in this Annual Report on Form 10-K of Hampshire Group, Limited for the year ended December 31, 2004. /s/ Deloitte & Touche LLP - ----------------------------- Deloitte & Touche Charlotte, North Carolina March 30, 2005