Loan Portfolio and Composition
During the second quarter of 2021, gross loans decreased to $2.27 billion as of June 30, 2021, a decrease of 6.5% from $2.43 billion as of March 31, 2021, and a decrease of 6.4% from $2.43 billion as of June 30, 2020. We continue to process a significant volume of PPP loan forgiveness applications from our customers. Excluding the effect of PPP loan forgiveness, the loan portfolio as of June 30, 2021 increased by $17.4 million, or 3.4% annualized from March 31, 2021. We currently see many borrowers maintaining and often accelerating their anticipated need for credit in 2021 and 2022, which has allowed our current loan pipeline to remain at elevated levels. Historically, the third and fourth quarters of our fiscal year are periods of higher loan growth for us and we anticipate this trend will continue in 2021.
Asset Quality
Asset quality continues to improve from levels experienced at the height of the COVID-19 pandemic with non-performing loans declining by $2.4 million or 23.9% during the second quarter of 2021 from the first quarter of 2021. Many of our borrowers have enjoyed a healthy increase in business activity due primarily to heightened demand for goods and services. The provision for loan losses recorded for the second quarter of 2021 was $1.3 million, which increased the allowance to $16.5 million, or 0.73% of the $2.27 billion in gross loans outstanding as of June 30, 2021. Provision expense for the second quarter of 2021 related primarily to the provisioning of loans moving from acquired loan portfolios to the organic loan portfolio upon their renewal.
Nonperforming loans to loans held for investment ratio continues to remain low at 0.33% as of June 30, 2021, which decreased from 0.41% as of March 31, 2021, and increased from 0.31% as of June 30, 2020. Annualized net charge-offs were 20 basis points for the second quarter of 2021 compared to 14 basis points for the first quarter of 2021.
Substantially all loans in active deferment related to the COVID-19 pandemic at March 31, 2021 have resumed regularly scheduled payments during the second quarter of 2021, and only $2.6 million of loans remain on deferral as of June 30, 2021.
Deposits and Borrowings
Deposits totaled $2.57 billion as of June 30, 2021, a decrease of 1.0% from $2.60 billion as of March 31, 2021, and an increase of 6.5% from $2.41 billion as of June 30, 2020. Noninterest-bearing demand deposits decreased $28.2 million, or 3.5%, from March 31, 2021, and increased $26.4 million, or 3.5%, from June 30, 2020. The decrease in noninterest-bearing deposits is primarily due to seasonal factors and deposit accounts related to PPP loans funded during the first quarter of 2021. Noninterest-bearing demand deposits represented 30.0% of total deposits as of June 30, 2021, down from 30.8% of total deposits as of March 31, 2021, and down from 30.9% of total deposits as of June 30, 2020. Interest-bearing demand deposits increased $43.6 million, or 9.0%, from March 31, 2021 primarily due to the seasonality of public funds. The average cost of deposits was 0.32% for the second quarter of 2021, representing a 5 basis point decrease from the first quarter of 2021 and a 34 basis point decrease from the second quarter of 2020. The decrease in average cost of deposits was due primarily to the continued repricing of certificates of deposit that matured during the second quarter of 2021.
Borrowings decreased by $72.6 million during the second quarter of 2021 to $119.1 million due primarily to repayment of advances under the Paycheck Protection Program Liquidity Facility from the Board of Governors of the Federal Reserve System. Borrowings totaled 3.9% of total assets at June 30, 2021, compared to 6.1% at March 31, 2021 and 6.5% at June 30, 2020.
2