EXHIBIT 23.1 CONSENT OF ERNST & YOUNG LLP, INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM The Board of Directors Delta Apparel, Inc. We consent to the incorporation by reference in the Registration Statement (Form S-8 No. 333-61190) pertaining to the Delta Apparel, Inc. 2001 Stock Option Plan and Delta Apparel, Inc. 2001 Incentive Stock Award Plan of our report dated August 12, 2004, with respect to the consolidated financial statements and schedule of Delta Apparel, Inc. included in the Annual Report on Form 10-K for the year ended July 3, 2004. /s/ Ernst & Young LLP Atlanta, Georgia September 8, 2004