"(Mbarushimana) was arrested on Monday afternoon at Frankfurt airport when he was trying to leave the country," said a spokesman at the prosecutors' office in Frankfurt.
He said the Rwandan was on his way to Russia.
Interpol had issued a "red notice", or wanted persons notice, for Mbarushimana for genocide, murder, extermination and the creation of a criminal organisation at the request of its National Central Bureau in Kigali, it said on its website.
Mbarushimana, a Hutu once employed by the U.N. Development Program (UNDP) in Rwanda, has repeatedly denied allegations that he was involved in the killing of the U.N. workers, but he is wanted by Rwandan authorities.
The spokesman at the prosecutor's office said German authorities had acted in response to an extradition request for Mbarushimana from Rwanda.
"It could take four to eight weeks until a decision is taken on extradition," he said, noting the procedure was complicated as Germany and Rwanda had no extradition agreement.
The International Criminal Tribunal for Rwanda (ICTR), a United Nations court that has been mainly trying the Hutu organisers of the Rwanda genocide, declined to prosecute Mbarushimana in 2002 due to lack of evidence.
France granted him refugee status in 2003.
Since then, he has been Executive Secretary of the Democratic Forces for the Liberation of Rwanda (FDLR), which includes former Rwandan military and Interahamwe militia responsible for carrying out the 100-day genocide which killed an estimated 800,000 Rwandan Tutsis and moderate Hutus.
Last month, as part of Interpol's Rwandan Genocide Fugitives Project, the General Secretariat headquarters in Lyon, France, circulated a list of the fugitives wanted by the national central bureau Kigali in connection with the Rwandan genocide to all 186 Interpol member countries.