The weird taste is caused by lactic acid.
The lactic acid is produced in muscles at all times (even at rest) and then converted to lactate by blood.
You burn the lactic acid as an energy source (not the only source) for muscles.
Lactic acid > lactate > glucose >>glycogen replacement - Efficiency of this process is limited by the oxygen uptake, that is why you should run slowly most of the time to condition your breathing.
When you start feeling this weird taste in your mouth, it means that you are working too hard and your muscles produce more lactic acid than body can use. Body pH declines to acidic spectrum, less than 7, and hence the weird taste.
You may have association between lactic acid and muscle soreness, that is correct but this effect shows only AFTER anaerobic workout.
I think that the point here should be that this taste is a good indicator of working too hard. You can try to push through it, or hold the tempo in which you are experiencing it, but if you are not specifically trying to get to it and see how much you can handle, you should slow down. Overtime your threshold where you experience it will move to faster paces and longer times.
Important thing is that you don't focus on it too much now, know about it, but don't stress about it. Try to push yourself to see how it feels but the important thing in the beginnings is to build a base by slow running for increasing periods of time.