As provided for by the CARES Act, the Company has offered payment modifications to borrowers. At June 30, 2021, $42.8 million, or 3.3% of total loans remained in some form of a modification, as compared to $86.7 million, or 6.7% of total loans at December 31, 2020. These loan modifications at June 30, 2021 include $11.4 million on interest only, $26.4 million on full deferral, and $5.0 million with extended amortization.
Asset Quality
Non-performing loans totaled $33.8 million at June 30, 2021, a decrease of $0.4 million from $34.2 million at the end of the linked quarter. Non-performing loans to total loans was 2.61% at June 30, 2021, and 2.68% and 0.70% at the end of the linked quarter and prior year quarter, respectively. The increase in non-performing loans as compared to the prior year quarter was primarily due to placing on non-accrual in the fourth quarter 2020, several significant loans previously modified in accordance with the CARES Act passed by Congress in 2020.
At June 30, 2021, $5.4 million of the Company’s allowance for loan losses was allocated to impaired loans totaling $36.1 million, compared to $5.1 million of the Company's allowance for loan losses allocated to impaired loans totaling $36.6 million at the end of the linked quarter. At June 30, 2021 management determined that $12.4 million, or 34%, of total impaired loans required no reserve allocation compared to $12.6 million, or 34% of total impaired loans at the end of the linked quarter, primarily due to adequate collateral values.
In the second quarter 2021, the Company had net loan charge-offs of $26,000 compared to net loan recovery of $248,000 in the linked quarter, and $29,000 of net loan recovery in the prior year quarter. The Company recorded provision expense of $0.4 million for loan losses for the second quarter 2021 driven principally by growth in the portfolio, compared to no provision for the linked quarter and $0.9 million for the prior year quarter.
The allowance for loan losses at June 30, 2021 was $18.7 million, or 1.45% of outstanding loans, and 55.5% of non-performing loans. At March 31, 2021, the allowance for loan losses was $18.4 million, or 1.44% of outstanding loans, and 53.6% of non-performing loans. At June 30, 2020, the allowance for loan losses was $16.6 million, or 1.30% of outstanding loans, and 186.6% of non-performing loans. The allowance for loan losses at June 30, 2021 represents management’s best estimate of probable losses inherent in the loan portfolio and is commensurate with risks in the loan portfolio as of that date.
Deposits
Total deposits at June 30, 2021 were $1.4 billion, a decrease of $13.0 million, or 0.9%, from March 31, 2021, and an increase of $53.4 million, or 4.0%, from the end of the prior year quarter. The decrease in total deposits in the current quarter as compared to the linked quarter is due to the maturity of $18 million of time deposits in the current quarter related to government programs.
Core deposits were $1.3 billion at June 30, 2021, an increase of $116.4 million, or 10.0%, from June 30, 2020. Growth in year-over-year core deposits is indicative of the higher savings rate customers have chosen in response to pandemic conditions.