Parent Company Only Financial Statements Condensed Parent Company Only Cash Flows (Details) - USD ($)
$ in Thousands
 3 Months Ended 12 Months Ended
Dec. 31, 2015
Sep. 30, 2015
Jun. 30, 2015
Mar. 31, 2015
Dec. 31, 2014
Sep. 30, 2014
Jun. 30, 2014
Mar. 31, 2014
Dec. 31, 2013
Sep. 30, 2013
Jun. 30, 2013
Mar. 31, 2013
Dec. 31, 2015
Dec. 31, 2014
Dec. 31, 2013
Condensed Financial Statements, Captions [Line Items]                              
Net income $ 2,117 $ 2,446 $ 2,368 $ 2,229 $ 2,940 $ 2,735 $ 2,367 $ 2,645 $ 3,934 $ 2,612 $ 16,363 $ 1,556 $ 9,160 $ 10,687 $ 24,465
ESOP, stock options, and restricted stock compensation                         440 384 1,416
Change in deferred tax assets, net                         4,170 5,602 (13,742)
Change in other assets                         270 (567) 1,558
Changes in other liabilities                         2,295 484 (685)
Net cash provided by operating activities                         16,715 18,585 15,205
Net cash provided in investing activities                         (45,314) 28,220 (2,611)
Proceeds from stock options exercises                         935 3,611 3,023
Repurchase and retirement of common stock                         (18,717) (17,566) (28,090)
Dividends paid                         (3,237) (2,938) (1,895)
Net cash provided by (used by) financing activities                         30,261 1,669 (44,760)
Net increase (decrease) in cash                         1,662 48,474 (32,166)
Cash and cash equivalents at beginning of year       104,049       55,575       87,741 104,049 55,575 87,741
Cash and cash equivalents at end of year 105,711       104,049       55,575       105,711 104,049 55,575
Parent Company [Member]                              
Condensed Financial Statements, Captions [Line Items]                              
Net income                         9,160 10,687 24,465
Equity in undistributed earnings of subsidiaries                         (9,854) (11,555) (25,563)
Dividends received from subsidiary                         6,785 72,300 14,491
ESOP, stock options, and restricted stock compensation                         0 13 48
Change in deferred tax assets, net                         1,101 (540) (584)
Change in receivables from subsidiaries                         (1,608) 8 (12)
Change in payables to subsidiaries                         (32) 50 66
Change in other assets                         (55) (16) 119
Changes in other liabilities                         (8) (130) (38)
Net cash provided by operating activities                         5,489 70,817 12,992
Investments in subsidiaries                         0 0 71
ESOP loan repayment                         1,115 1,054 1,011
Net cash provided in investing activities                         1,115 1,054 1,082
Proceeds from stock options exercises                         935 3,611  
Proceeds for vested awards                         282 282 1,508
Repurchase and retirement of common stock                         (18,717) (17,566) (28,090)
Dividends paid                         (3,237) (2,938) (1,895)
Net cash provided by (used by) financing activities                         (20,737) (16,611) (25,454)
Net increase (decrease) in cash                         (14,133) 55,260 (11,380)
Cash and cash equivalents at beginning of year       $ 64,471       $ 9,211       $ 20,591 64,471 9,211 20,591
Cash and cash equivalents at end of year $ 50,338       $ 64,471       $ 9,211       $ 50,338 $ 64,471 $ 9,211