Fair Value Measurements - Financial Instruments Measured at Fair Value on a Recurring Basis (Details) - USD ($)
$ in Thousands
Dec. 31, 2021
Dec. 31, 2020
Financial assets:    
Loans, at fair value $ 29,697 $ 45,000
Derivative assets, at fair value 250 258
Financial liabilities:    
Derivative liabilities, at fair value 228 221
Fair Value, Measurements, Recurring    
Financial assets:    
MSR financing receivables 125,018 9,346
Loans, at fair value 29,697 45,000
Derivative assets, at fair value 250 258
Other assets 12,655 17,690
Financial liabilities:    
Derivative liabilities, at fair value 228 221
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 1    
Financial assets:    
MSR financing receivables 0 0
Loans, at fair value 0 0
Derivative assets, at fair value 20 0
Other assets 5,267 10,821
Financial liabilities:    
Derivative liabilities, at fair value 0 0
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 2    
Financial assets:    
MSR financing receivables 0 0
Loans, at fair value 0 0
Derivative assets, at fair value 230 258
Other assets 0 0
Financial liabilities:    
Derivative liabilities, at fair value 228 221
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 3    
Financial assets:    
MSR financing receivables 125,018 9,346
Loans, at fair value 29,697 45,000
Derivative assets, at fair value 0 0
Other assets 7,388 6,869
Financial liabilities:    
Derivative liabilities, at fair value 0 0
Fair Value, Measurements, Recurring | Agency MBS    
Financial assets:    
Trading securities 483,927 970,880
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 1    
Financial assets:    
Trading securities 0 0
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 2    
Financial assets:    
Trading securities 483,927 970,880
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 3    
Financial assets:    
Trading securities 0 0
Fair Value, Measurements, Recurring | Credit Securities    
Financial assets:    
Trading securities 26,222 26,660
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 1    
Financial assets:    
Trading securities 0 0
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 2    
Financial assets:    
Trading securities 0 14,730
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 3    
Financial assets:    
Trading securities 26,222 11,930
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary    
Financial assets:    
Loans, at fair value 7,442 93,283
Financial liabilities:    
Secured debt of consolidated VIE, at fair value 508 93,627
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 1    
Financial assets:    
Loans, at fair value 0 0
Financial liabilities:    
Secured debt of consolidated VIE, at fair value 0 0
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 2    
Financial assets:    
Loans, at fair value 0 0
Financial liabilities:    
Secured debt of consolidated VIE, at fair value 0 93,051
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 3    
Financial assets:    
Loans, at fair value 7,442 93,283
Financial liabilities:    
Secured debt of consolidated VIE, at fair value $ 508 $ 576