Exhibit 23 CONSENT OF ERNST & YOUNG LLP, INDEPENDENT AUDITORS We consent to the incorporation by reference in the Registration Statement (Form S-8 No. 333-106861) pertaining to the 2003 Stock Incentive Plan of Cavco Industries, Inc. of our report dated April 21, 2004, with respect to the consolidated financial statements of Cavco Industries, Inc. included in the Annual Report (Form 10-K) for the year-ended March 31, 2004. /s/ Ernst & Young LLP Phoenix, Arizona May 17, 2004