Loans - Loan Categories Narrative (Details) - USD ($)
$ in Thousands
12 Months Ended
Dec. 31, 2021
Dec. 31, 2020
Dec. 31, 2019
Gross loans      
Loan origination fees $ 7,500 $ 13,400  
Accrued interest receivable on loans 10,200 13,800  
Outstanding loan balances to related parties 37,700 48,700  
Unadvanced lines of credit available to related parties 14,500 23,200  
New loans and net increases to loan balances to related parties during period 2,100 16,100  
Principal paydowns on related party loans 21,800 11,400  
Financing receivable, average loan size 2,920,684 3,073,860  
Commercial      
Gross loans      
Participation loans amount 62,600 77,100  
Financing receivable, average loan size 2,574,807 2,726,710  
Participations loans sold that are still serviced amount 66,700 65,300  
Tax exempt interest income on qualified commercial loans 1,800 2,000 $ 2,100
Average tax exempt loan balances 54,500 58,000  
SBA Paycheck Protection Program ("PPP")      
Gross loans      
Loan origination fees $ 2,400 10,000  
Interest rate 1.00%    
Finance receivable term, minimum 2 years    
Finance receivable term, maximum 5 years    
Gross deferred Income $ 26,200    
Amortization of deferred loan origination fees, net 16,600 7,200  
Provision for credit losses 2,400    
Financing receivable, average loan size 71,502 443,070  
SBA Paycheck Protection Program ("PPP") | Quarterly Average      
Gross loans      
Financing receivable, average loan size 187    
Residential mortgages      
Gross loans      
Financing receivable, average loan size 256,940 252,995  
Amount of loans serviced for others $ 10,400 $ 13,700