EXHIBIT 23 CONSENT OF INDEPENDENT AUDITORS We consent to the incorporation by reference in the Registration Statement (Form S-8 No. 333-75477) pertaining to the Retirement Savings Plan, the Registration Statement (Form S-8 No. 333-21857 and S-8 No. 333-91360) pertaining to the Equity Incentive Plan, the Registration Statement (Form S-8 No. 333-22673) pertaining to the 1995 Director Stock Option Plan, and the Registration Statement (Form S-8 No. 333-109520) pertaining to the Directors Deferred Compensation Plan, of our report dated January 28, 2004, with respect to the consolidated financial statements and schedules of American Medical Security Group, Inc. included in the Annual Report (Form 10-K) for the year ended December 31, 2003. /s/ ERNST & YOUNG LLP Milwaukee, Wisconsin March 11, 2004