EXHIBIT 23.3 Consent of Independent Registered Public Accounting Firm -------------------------------------------------------- The Board of Directors Provident Bancorp, Inc.: We consent to the incorporation by reference in the registration statement on Form S-4 to be filed by Provident Bancorp, Inc. of our report dated November 26, 2003, with respect to the consolidated statements of financial condition of Provident Bancorp, Inc. as of September 30, 2003 and 2002, and the related consolidated statements of earnings, changes in stockholders' equity, cash flows, and comprehensive income for each of the years in the three-year period ended September 30, 2003, and all related financial statement schedules, which report appears in the September 30, 2003, annual report on Form 10-K of Provident Bancorp, Inc. /s/ KPMG LLP New York, New York August 13, 2004