Parent Company Only Financial Information (Details 2) (USD $)
In Thousands, unless otherwise specified
 3 Months Ended 12 Months Ended
Dec. 31, 2014
Sep. 30, 2014
Jun. 30, 2014
Mar. 31, 2014
Dec. 31, 2013
Sep. 30, 2013
Jun. 30, 2013
Mar. 31, 2013
Dec. 31, 2014
Dec. 31, 2013
Dec. 31, 2012
Dividends from subsidiaries:                      
Other income $ 14,739 $ 14,204 $ 15,012 $ 14,986 $ 14,529 $ 15,630 $ 15,783 $ 16,641 $ 58,941 $ 62,583 $ 66,852
Expense:                      
Interest expense: 1,570 1,600 1,635 1,694 1,881 2,003 2,215 2,532 6,499 8,631 14,770
Income tax benefit (4,763) (4,708) (4,025) (4,038) (3,528) (3,572) (3,787) (3,224) (17,534) (14,111) (10,497)
Distributions less than net income of subsidiaries:                      
Net income 7,593 9,109 8,185 7,887 6,920 7,933 7,440 6,433 32,774 28,726 22,357
Parent                      
Dividends from subsidiaries:                      
Non-bank                 2,000 2,000 2,500
Interest and dividend income                 20 46 49
Other income                 3,475 4,669 6,231
Total operating income                 5,495 6,715 8,780
Expense:                      
Salaries and employee benefits                 6,778 5,139 4,718
Interest expense:                 1,183 1,220 1,351
Operating expense                 2,563 2,163 2,469
Total expense                 10,524 8,522 8,538
(Loss) income before income tax benefit and distributions less than net income of subsidiaries                 (5,029) (1,807) 242
Income tax benefit                 3,131 1,875 1,155
(Loss) income before distributions less than net income of subsidiaries                 (1,898) 68 1,397
Distributions less than net income of subsidiaries:                      
Bank                 31,991 26,416 20,097
Non-bank                 2,681 2,242 863
Net income                 $ 32,774 $ 28,726 $ 22,357