CONSOLIDATED STATEMENTS OF CASH FLOWS - USD ($)
 12 Months Ended
Dec. 31, 2015
Dec. 31, 2014
Dec. 31, 2013
CASH FLOWS FROM OPERATING ACTIVITIES:      
Net income (loss) $ 17,183,000 $ 62,168,000 $ 46,453,000
Adjustments to reconcile net income to net cash (used in) provided by operating activities:      
Provision for (recovery of) loan losses 49,889,000 1,804,000 3,020,000
Depreciation, amortization, and accretion 19,700,000 3,414,000 14,362,000
Amortization of stock-based compensation 3,145,000 6,566,000 10,472,000
Non-cash incentive compensation to the Manager 0 0 484,000
Deferred income tax (benefit) expense (390,000) (11,536,000) (6,710,000)
Sale (origination) of residential mortgage loans held for sale, net (17,346,000) 96,536,000 146,000
Sale (purchase) of and principal payments on securities, trading, net (5,486,000) (16,515,000) 12,961,000
Net realized and unrealized loss (gain) on investment securities, trading 547,000 2,818,000 324,000
Net realized (gain) loss on sales of investment securities available-for-sale and loans (35,703,000) (15,283,000) (10,986,000)
Loss (gain) on the reissuance / (extinguishment) of debt 1,403,000 4,442,000 0
Loss (gain) on sale of real estate (206,000) (6,127,000) (16,616,000)
Settlement of derivative instruments 909,000 (23,000) 0
Net impairment losses recognized in earnings 372,000 0 855,000
Unrealized gain (loss) and net interest income on linked transactions, net (235,000) (7,850,000) 3,841,000
Linked Transactions, Fair Value Adjustments 235,000 5,615,000 (6,018,000)
Equity in net (earnings) losses of unconsolidated subsidiaries (2,388,000) (4,767,000) (949,000)
Changes in operating assets and liabilities, net of acquisitions      
Decrease (increase) in restricted cash 1,704,000 5,204,000 8,445,000
(Increase) decrease in interest receivable, net of purchased interest 1,456,000 (7,295,000) (1,108,000)
Increase (decrease) in management fee payable 0 171,000 (6,357,000)
Increase (decrease) in security deposits 0 4,696,000 (337,000)
(Decrease) increase in accounts payable and accrued liabilities 2,290,000 (3,363,000) (16,327,000)
Increase (decrease) in accrued interest expense 3,406,000 430,000 (1,445,000)
(Increase) decrease in other assets (4,947,000) 9,873,000 7,259,000
Net cash (used in) provided by operating activities 69,995,000 (65,474,000) 49,672,000
CASH FLOWS FROM INVESTING ACTIVITIES:      
(Increase) decrease in restricted cash 79,799,000 (23,568,000) 22,248,000
Purchase of securities available-for-sale (40,375,000) (180,990,000) (136,282,000)
Principal payments on securities available-for-sale 75,960,000 56,053,000 52,812,000
Proceeds from sale of securities available-for-sale 65,787,000 147,171,000 11,893,000
Return of investment (investment in) in unconsolidated entity 2,715,000 9,557,000 (28,034,000)
Improvement of real estate held-for-sale 0 0 (404,000)
Settlement of derivative instruments 5,553,000 (4,119,000) 0
Proceeds from sale of real estate held-for-sale 301,000 65,753,000 37,001,000
Origination and purchase of loans (940,583,000) (1,019,721,000) (725,657,000)
Principal payments received on loans and leases 560,182,000 376,219,000 590,663,000
Proceeds from sale of loans 130,612,000 209,707,000 674,977,000
Distributions from investments in real estate 0 0 1,094,000
Improvements in investments in real estate 0 (221,000) (365,000)
Purchase of furniture and fixtures 0 (69,000) (133,000)
Acquisition of property and equipment (14,000) (865,000) (373,000)
Investment in loans - related parties 0 (1,572,000) (1,241,000)
Principal payments received on loans – related parties 558,000 3,848,000 1,685,000
Net cash (used in) provided by investing activities (59,505,000) (393,250,000) 492,271,000
CASH FLOWS FROM FINANCING ACTIVITIES:      
Net proceeds from issuances of common stock, dividend reinvestment and stock purchase plan (net of offering costs of $100, $0, and $3,837) 228,000 30,297,000 133,665,000
Net proceeds from dividend reinvestment and stock purchase plan (net of offering costs of $0, $0 and $0) 0 0 0
Repurchase of common stock (25,929,000) (6,832,000) 0
Proceeds from borrowings:      
Repurchase agreements, net of repayments (13,541,000) 277,875,000 15,226,000
Securitizations 505,862,000 235,344,000 260,840,000
Convertible Senior Notes 99,000,000 0 115,000,000
Senior Secured Revolving Credit Facility 138,500,000 113,500,000 0
Reissuance of debt 16,597,000 52,663,000 0
Payments on borrowings:      
Collateralized debt obligations (327,537,000) (451,991,000) (797,573,000)
Securitizations (205,125,000) (34,000,000) 0
Senior Secured Revolving Credit Facility (62,000,000) 0 0
Settlement of derivative instruments 0 3,052,000 0
Mortgage payable 0 0 (13,600,000)
Payment of debt issuance costs (13,799,000) (8,939,000) (9,786,000)
Distributions to non-controlling interest and subordinated note holders (12,433,000) (2,323,000) (30,709,000)
Proceeds received from non-controlling interests 0 14,213,000 5,531,000
Distributions paid on preferred stock (24,390,000) (15,008,000) (6,413,000)
Distributions paid on common stock (90,100,000) (104,225,000) (93,458,000)
Net cash provided by (used in) financing activities (11,639,000) 276,359,000 (364,951,000)
NET (DECREASE) INCREASE IN CASH AND CASH EQUIVALENTS (1,149,000) (182,365,000) 176,992,000
CASH AND CASH EQUIVALENTS AT BEGINNING OF PERIOD 79,905,000 [1] 262,270,000 85,278,000
CASH AND CASH EQUIVALENTS AT END OF PERIOD 78,756,000 [1] 79,905,000 [1] 262,270,000
SUPPLEMENTAL DISCLOSURE:      
Interest expense paid in cash 48,089,000 35,690,000 41,453,000
Income taxes paid in cash 11,710,000 3,305,000 10,710,000
Redeemable Preferred Stock Series A [Member]      
CASH FLOWS FROM FINANCING ACTIVITIES:      
Proceeds from issuance of redeemable preferred shares, net of offering costs 0 8,984,000 112,000
Redeemable Preferred Stock Series B [Member]      
CASH FLOWS FROM FINANCING ACTIVITIES:      
Proceeds from issuance of redeemable preferred shares, net of offering costs 3,028,000 47,481,000 56,214,000
Redeemable Preferred Stock Series C [Member]      
CASH FLOWS FROM FINANCING ACTIVITIES:      
Proceeds from issuance of redeemable preferred shares, net of offering costs 0 116,268,000 0
Moselle CLO [Member]      
CASH FLOWS FROM INVESTING ACTIVITIES:      
Acquisition of controlling interest in Moselle CLO S.A. 0 (30,433,000) 0
Primary Capital Advisors LLC [Member]      
CASH FLOWS FROM INVESTING ACTIVITIES:      
Acquisition of controlling interest in Moselle CLO S.A. $ 0 $ 0 $ (7,613,000)
[1] December 31, 2015December 31, 2014Assets of consolidated Variable Interest Entities ("VIEs") included in the total assets above: Cash and cash equivalents $95 $25 Restricted cash$39,061 $121,247 Investments securities available-for-sale, pledged as collateral, at fair value66,137 119,203 Loans held for sale1,475 282Loans, pledged as collateral and net of allowances of $3.3 million and $8.8 million1,416,441 1,261,137 Interest receivable6,592 8,941 Prepaid expenses238 221 Principal paydown receivable17,800 25,767 Other assets833 (12) Total assets of consolidated VIEs$1,548,672 $1,536,811