Related Party Transactions - Additional Information (Details) - USD ($)
1 Months Ended 3 Months Ended 9 Months Ended
Mar. 31, 2017
Sep. 30, 2020
Jun. 30, 2020
Sep. 30, 2019
Sep. 30, 2020
Sep. 30, 2019
Dec. 31, 2019
Related Party Transaction [Line Items]              
Management fees   $ 7,393,000   $ 7,418,000 $ 22,234,000 $ 22,162,000  
ACM              
Related Party Transaction [Line Items]              
Management agreement, annual management fee         $ 1    
Period of written notice of termination (in days)         180 days    
Management fee reduction (as a percent)     40.00%        
ACM | Restricted Stock Units (RSUs)              
Related Party Transaction [Line Items]              
Award vesting period         5 years    
Corporate Joint Venture | BUCKLER Securities, LLC              
Related Party Transaction [Line Items]              
Payments to acquire equity method investments $ 352,000            
Percentage of voting interests acquired 10.00% 10.00%     10.00%    
Equity method investments   $ 633,000     $ 633,000   $ 381,000
Proceeds from divestiture of interest in joint venture   117,000     1,382,000    
Required annual reduction in share of gross financing profit   306,000     $ 306,000    
Armour Management Agreement | ACM              
Related Party Transaction [Line Items]              
Automatic renewal period of management agreement (in years)         5 years    
Javelin Management Agreement | ACM              
Related Party Transaction [Line Items]              
Automatic renewal period of management agreement (in years)         1 year    
Related party transaction, management agreement, term         1 year    
Other Expense | ACM              
Related Party Transaction [Line Items]              
Management fees   2,000   12,000 $ 150,000 78,000  
Stock Based Compensation Expense | ACM              
Related Party Transaction [Line Items]              
Management fees   118,000   82,000 402,000 266,000  
Required Regulatory Capital Requirement of Related Party | Corporate Joint Venture | BUCKLER Securities, LLC              
Related Party Transaction [Line Items]              
Loans receivable, related parties   105,000,000.0     $ 105,000,000.0    
Related party transaction, rate         0.00%    
Loans and leases receivable, related parties, proceeds   $ 25,000   $ 479,000 $ 312,000 $ 1,562,000