I'm a libertarian (socialist) as well and I think we're having a misunderstanding about what a refugee is.
The ones coming from africa are no refugees. They're not fleeing from any political persecution, have no right for asylum and may only ask for normal immigration. If that's declined, they have no reason to stay here.
The ones fleeing from war do have a right to asylum and although they technically don't have that in germany, it would be the right course of action to let them in for now and send them home as soon as their war/political persecution is over, unless they're granted the german citizenship of course.
That's simply because the countries who would have to take in the refugees are unable or unwilling to do so.
Syrians should get asylum, they're welcome for now, africans can try normal immigration or fuck of, there's no collective political persecution they're all fleeing from. Poor economy is no reason for asylum.
The process to give the ones who deserve citizenship the ability to work and to decide upon granting asylum or immigration simply takes too long.
If we were to speed up that process, the people who are allowed to stay could actually improve our economy.