Investment Securities (Summary of Available-for-sale and Held-to-maturity Securities) (Details) - USD ($)
Mar. 31, 2020
Dec. 31, 2019
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost $ 739,900,000 $ 496,165,000
Gross Unrealized Gains 21,951,000 4,598,000
Gross Unrealized Losses 312,000 1,768,000
Estimated Fair Value 761,539,000 498,995,000
Investment securities held-to-maturity    
Amortized Cost 0 86,780,000
Gross Unrealized Gains   935,000
Gross Unrealized Losses   154,000
Estimated Fair Value 0 87,561,000
Equity securities held-for-trading    
Estimated Fair Value 2,558,000 3,581,000
GSE debentures    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 231,779,000 182,922,000
Gross Unrealized Gains 6,176,000 2,939,000
Gross Unrealized Losses 138,000 58,000
Estimated Fair Value 237,817,000 185,803,000
Investment securities held-to-maturity    
Amortized Cost   31,228,000
Gross Unrealized Gains   113,000
Gross Unrealized Losses   51,000
Estimated Fair Value   31,290,000
GSE CMOs    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 62,391,000 87,001,000
Gross Unrealized Gains 1,151,000 22,000
Gross Unrealized Losses 15,000 1,091,000
Estimated Fair Value 63,527,000 85,932,000
GSE MBSs    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 319,088,000 153,049,000
Gross Unrealized Gains 9,551,000 797,000
Gross Unrealized Losses 141,000 503,000
Estimated Fair Value 328,498,000 153,343,000
Investment securities held-to-maturity    
Amortized Cost   9,360,000
Gross Unrealized Gains   0
Gross Unrealized Losses   81,000
Estimated Fair Value   9,279,000
SBA commercial loan asset-backed securities    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 1,000 34,000
Gross Unrealized Gains 0 0
Gross Unrealized Losses 0 0
Estimated Fair Value 1,000 34,000.0
Corporate debt obligations    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 25,389,000 28,484,000
Gross Unrealized Gains 570,000 502,000
Gross Unrealized Losses 0 0
Estimated Fair Value 25,959,000 28,986,000
U.S. Treasury bonds    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 55,776,000 44,675,000
Gross Unrealized Gains 3,858,000 338,000
Gross Unrealized Losses 0 116,000
Estimated Fair Value 59,634,000 44,897,000
Municipal obligations    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 44,976,000  
Gross Unrealized Gains 645,000  
Gross Unrealized Losses 3,000  
Estimated Fair Value 45,618,000  
Investment securities held-to-maturity    
Amortized Cost   45,692,000
Gross Unrealized Gains   822,000
Gross Unrealized Losses   0
Estimated Fair Value   46,514,000
Foreign government obligations    
Debt Securities, Available-for-sale [Line Items]    
Amortized Cost 500,000  
Gross Unrealized Gains 0  
Gross Unrealized Losses 15,000  
Estimated Fair Value $ 485,000  
Investment securities held-to-maturity    
Amortized Cost   500,000
Gross Unrealized Gains   0
Gross Unrealized Losses   22,000
Estimated Fair Value   $ 478,000