Schedule of Mortgage-Backed Securities (Details) - USD ($)
$ in Thousands
12 Months Ended
Dec. 31, 2015
Dec. 31, 2014
Mortgage-Backed Securities [Line Items]    
Fair Value $ 1,242,266 $ 1,393,303
Residential Mortgage Backed Securities [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal 1,177,078 1,306,824
Unamortized Premium (Discount) 53,646 53,147
Amortized Cost 1,239,215 1,370,751
Gross Unrealized Gain 11,428 26,195
Gross Unrealized Losses (8,377) (3,643)
Fair Value $ 1,242,266 $ 1,393,303
Weighted Average Coupon 3.97% 3.99%
Weighted Average Yield 3.39% 3.41%
Weighted Average Life (Years) [1] 7 years 26 days 7 years 164 days
Non-Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 48,408 $ 50,668
Unamortized Premium (Discount) (18,013) (20,377)
Amortized Cost 30,395 30,291
Gross Unrealized Gain 2,264 2,896
Gross Unrealized Losses (1,258) (686)
Fair Value $ 31,401 $ 32,501
Weighted Average Coupon 2.48% 2.29%
Weighted Average Yield 20.97% 10.76%
Weighted Average Life (Years) [1] 4 years 296 days 4 years 354 days
15-year fixed rate mortgages [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 162,546 $ 130,720
Unamortized Premium (Discount) 7,839 6,304
Amortized Cost 170,385 137,024
Gross Unrealized Gain 531 1,113
Gross Unrealized Losses (655) (109)
Fair Value $ 170,261 $ 138,028
Weighted Average Coupon 3.38% 3.40%
Weighted Average Yield 2.31% 2.46%
Weighted Average Life (Years) [1] 4 years 361 days 5 years 157 days
20-year fixed rate mortgages [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 18,477 $ 9,764
Unamortized Premium (Discount) 1,277 577
Amortized Cost 19,754 10,341
Gross Unrealized Gain 153 227
Gross Unrealized Losses (77) 0
Fair Value $ 19,830 $ 10,568
Weighted Average Coupon 4.00% 4.00%
Weighted Average Yield 2.75% 3.04%
Weighted Average Life (Years) [1] 6 years 183 days 6 years 336 days
30-year fixed rate mortgages [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 842,524 $ 1,042,550
Unamortized Premium (Discount) 53,832 61,089
Amortized Cost 896,356 1,103,639
Gross Unrealized Gain 8,117 20,379
Gross Unrealized Losses (3,679) (1,764)
Fair Value $ 900,794 $ 1,122,254
Weighted Average Coupon 4.12% 4.09%
Weighted Average Yield 3.11% 3.29%
Weighted Average Life (Years) [1] 8 years 106 days 8 years 175 days
Adjustable Rate Mortgages | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 36,433 $ 41,710
Unamortized Premium (Discount) 2,196 2,813
Amortized Cost 38,629 44,523
Gross Unrealized Gain 81 90
Gross Unrealized Losses (180) (330)
Fair Value $ 38,530 $ 44,283
Weighted Average Coupon 4.05% 4.60%
Weighted Average Yield 2.68% 3.08%
Weighted Average Life (Years) [1] 5 years 161 days 5 years 354 days
Reverse mortgages [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 68,690 $ 31,412
Unamortized Premium (Discount) 6,515 2,741
Amortized Cost 75,205 34,153
Gross Unrealized Gain 34 300
Gross Unrealized Losses (1,547) (28)
Fair Value $ 73,692 $ 34,425
Weighted Average Coupon 4.63% 4.91%
Weighted Average Yield 2.54% 2.56%
Weighted Average Life (Years) [1] 5 years 234 days 4 years 270 days
Interest-Only [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Amortized Cost $ 8,491 $ 10,780
Gross Unrealized Gain 248 1,190
Gross Unrealized Losses (981) (726)
Fair Value $ 7,758 $ 11,244
Weighted Average Coupon 3.82% 4.04%
Weighted Average Yield 3.30% 11.82%
Weighted Average Life (Years) [1] 2 years 131 days 2 years 248 days
Agency Securities [Member] | Agency RMBS [Member]    
Mortgage-Backed Securities [Line Items]    
Current Principal $ 1,128,670 $ 1,256,156
Unamortized Premium (Discount) 71,659 73,524
Amortized Cost 1,208,820 1,340,460
Gross Unrealized Gain 9,164 23,299
Gross Unrealized Losses (7,119) (2,957)
Fair Value $ 1,210,865 $ 1,360,802
Weighted Average Coupon 4.03% 4.05%
Weighted Average Yield 2.94% 3.24%
Weighted Average Life (Years) [1] 7 years 58 days 7 years 197 days
[1] Average lives of RMBS are generally shorter than stated contractual maturities. Average lives are affected by the contractual maturities of the underlying mortgages, scheduled periodic payments of principal, and unscheduled prepayments of principal.