EXHIBIT (23) Consent of Ernst & Young LLP, Independent Auditors We consent to the incorporation by reference in the Registration Statements of Russell Corporation listed below of our report dated February 10, 2004, with respect to the consolidated financial statements and schedule of Russell Corporation included in this Annual Report (Form 10-K) for the year ended January 3, 2004. Form S-3 Registration Statement No. 33-47906 Form S-3 Registration Statement No. 33-54361 Form S-8 Registration Statement No. 33-69679 Form S-8 Registration Statement No. 333-89765 Form S-8 Registration Statement No. 333-30236 Form S-8 Registration Statement No. 333-30238 Form S-8 Registration Statement No. 333-55338 Form S-8 Registration Statement No. 333-55340 Form S-8 Registration Statement No. 333-97129 /s/ Ernst & Young LLP Birmingham, Alabama March 12, 2004