SEGMENT REPORTING - Narrative (Details)
$ in Thousands
 3 Months Ended 12 Months Ended
Jun. 30, 2016
USD ($)
Dec. 31, 2016
USD ($)
Sep. 30, 2016
USD ($)
Jun. 30, 2016
USD ($)
Mar. 31, 2016
USD ($)
Dec. 31, 2015
USD ($)
Sep. 30, 2015
USD ($)
Jun. 30, 2015
USD ($)
Mar. 31, 2015
USD ($)
Dec. 31, 2016
USD ($)
segment
Dec. 31, 2015
USD ($)
Dec. 31, 2014
USD ($)
Segment Reporting Information [Line Items]                        
Number of reportable segments | segment                   3    
Net income (loss)   $ 2,307 $ 2,310 $ 6,499 $ 1,796 $ 1,402 $ 1,406 $ 1,905 $ 2,103 $ 12,912 $ 6,816 $ 2,079
Increase (decrease) in provision expense                   3,632 2,493 2,582
Net Income (loss) from continuing operations                   8,977 6,603 2,655
Commercial & Retail Banking                        
Segment Reporting Information [Line Items]                        
Increase (decrease) in net interest income                   8,200    
Increase (decrease) in average loan balances                   179,000    
Increase (decrease) in noninterest income                   1,600    
Increase (decrease) in performance of interest rate cap                   818    
Increase (decrease) in gain on sale of securities                   882    
Increase (decrease) in other operating income                   133    
Increase (decrease) in debit card income and interchange income                   199    
Increase (decrease) in volume of portfolio loans sold                   (371)    
Increase (decrease) in mortgage fee income                   (259)    
Increase (decrease) in noninterest expense                   2,400    
Increase (decrease) in salaries expense                   543    
Increase (decrease) in occupancy and equipment expense                   494    
Increase (decrease) in data processing expense                   832    
Increase (decrease) in professional fees                   (776)    
Increase (decrease) in provision expense                   1,100    
Mortgage Banking                        
Segment Reporting Information [Line Items]                        
Increase (decrease) in net interest income                   (32)    
Increase (decrease) in noninterest income                   6,400    
Increase (decrease) in mortgage fee income                   6,400    
Increase (decrease) in noninterest expense                   7,600    
Increase (decrease) in salaries expense                   6,900    
Increase (decrease) in occupancy and equipment expense                   117    
Increase (decrease) in data processing expense                   $ 98    
Increase (decrease) in origination volume                   26.40%    
Increase (decrease) in earn out payments                   $ 1,800    
Increase (decrease) in mortgage processing expense                   197    
Increase (decrease) in travel, entertainment, dues, and subscriptions expense                   133    
Increase (decrease) in other operating expense                   134    
Increase (decrease) in loan expenses                   55    
Increase (decrease) in marketing expense                   (115)    
Financial Holding Company                        
Segment Reporting Information [Line Items]                        
Increase (decrease) in net interest income                   22    
Increase (decrease) in noninterest income                   916    
Increase (decrease) in noninterest expense                   2,300    
Increase (decrease) in salaries expense                   1,700    
Increase (decrease) in occupancy and equipment expense                   259    
Increase (decrease) in other operating expense                   66    
Increase (decrease) in income tax expense                   475    
Increase (decrease) in professional fees                   220    
Operating Segments | Commercial & Retail Banking                        
Segment Reporting Information [Line Items]                        
Net income (loss)                   9,900 5,964 4,831
Increase (decrease) in provision expense                   3,632 2,493 2,582
Net Income (loss) from continuing operations                   9,900 5,964 4,831
Operating Segments | Mortgage Banking                        
Segment Reporting Information [Line Items]                        
Net income (loss)                   3,062 3,829 (149)
Increase (decrease) in provision expense                   0 0 0
Net Income (loss) from continuing operations                   3,062 3,829 (149)
Operating Segments | Financial Holding Company                        
Segment Reporting Information [Line Items]                        
Net income (loss)                   312 (3,058) (1,979)
Increase (decrease) in provision expense                   0 0 0
Net Income (loss) from continuing operations                   (3,985) (3,058) (1,979)
Operating Segments | Insurance                        
Segment Reporting Information [Line Items]                        
Net income (loss)                   (362) 81 (624)
Increase (decrease) in provision expense                   0 0 0
Net Income (loss) from continuing operations                   0 0 0
USI Insurance Services | Discontinued Operations, Disposed of by Sale                        
Segment Reporting Information [Line Items]                        
Gain on sale of subsidiary $ 6,900                 $ 6,926 $ 0 $ 0