Territorial Bancorp (TBNK) 10-K 2013 FY Annual report

Exhibit 23

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Territorial Bancorp Inc.:

We consent to the incorporation by reference in the registration statements No. 333-168839 and No. 333-170579 on Form S-8 of Territorial Bancorp Inc. of our reports dated April 23, 2014, with respect to the consolidated balance sheets of Territorial Bancorp Inc. as of December 31, 2013 and 2012, and the related consolidated statements of income, comprehensive income, stockholders’ equity and comprehensive income, and cash flows for each of the years in the three-year period ended December 31, 2013, and the effectiveness of internal control over financial reporting as of December 31, 2013, which reports appear in the December 31, 2013 annual report on Form 10-K of Territorial Bancorp Inc.

Our report dated April 23, 2014, on the effectiveness of internal control over financial reporting as of December 31, 2013, expresses our opinion that Territorial Bancorp Inc. did not maintain effective internal control over financial reporting as of December 31, 2013 because of the effect of material weaknesses on the achievement of the objectives of the control criteria and contains an explanatory paragraph that states material weaknesses related to the Company’s (1) risk assessment processes for effective identification of financial reporting risks and controls, (2) investment securities policies and processes, and (3) income tax policies and procedures, have been identified and included in Management’s Report on Internal Control Over Financial Reporting.

/s/ KPMG LLP

Honolulu, Hawaii

April 23, 2014