NEW HAVEN, Conn.—Federal prosecutors say a New York-based investment bank and broker-dealer has agreed to pay $25 million in penalties for misleading customers trading in residential mortgage-backed securities.

The U.S. attorney's office in New Haven announced Wednesday that as part of the non-prosecution agreement Jefferies LLC will pay up to $11 million in restitution to victims and up to $4.2 million to the Securities and Exchange Commission.

U.S. Attorney Deirdre Daly said employees in Jefferies' fixed income division repeatedly misled their own customers to boost their profits.

Daly says management not only tolerated illegal practices, it encouraged them.

Jefferies has cooperated and already implemented changes.

A former Jefferies executive was convicted earlier of defrauding investment funds established as part of the government's response to the 2008 financial crisis.