1 EXHIBIT 23.1 SOUTHERN FINANCIAL BANCORP Consent Letter May 6, 2004 2 INDEPENDENT ACCOUNTANTS' CONSENT We consent to the incorporation by reference in the Current Report on Form 8-K of Provident Bankshares Corporation of our report dated February 27, 2004, relating to the consolidated balance sheets of Southern Financial Bancorp, Inc. and subsidiaries as of December 31, 2003 and 2002, and the related consolidated statements of income, comprehensive income, changes in stockholders' equity, and cash flows for each of the years in the three-year period ended December 31, 2003, which report appears in the Southern Financial Bancorp, Inc. Form 10-K. /s/ KPMG LLP Richmond, Virginia May 6, 2004