It is, of itself, not harmful but not effective at being an antibacterial. I allow my batches to settle out and if any appears, decant it off. As others have said, vigorous stirring and/or production with heat normally eliminates it.
If producing "cold" then I always allow the batch to rest for a day or so, to see if any fallout of silver oxide occurs. If I am not producing for others, I just drinki it. Never had any issues, the amount is too small.