hi
there are some mismatched terms in your question.

כניסות/יציאות מאוזנות = Balanced I/O

does it happen with different speakers / headphones?
different cables?
when the level is set lower on the speakers and you boost it from the mixer, does it still happen?
if you connect the speakers directly to the soundcard (mind your levels..!)?

to shorten the guessing process, list the connections to / from the mixer in detail, including converter plugs from minijack to jack etc, and the levels in soft-and-hardware (including speaker amp).

which headphone output, the mixers?

that's really asking for it.
I wouldn't keel over and die with surprise if that spagehtti of connectors is to blame.

to check the mixer, take a working RCA without converters and hook up a cd or something to the 2tr in and check on the mixer headphones out,
(works = mixer in+headphone out are good).
then check main outputs in similar way.

or just buy some decent cables.
it's possible that something got messed up or dirty in the mixer connections / pots,
but you anyway need the cables.

in that case, connect like usual and then switch between left and right (the cables that connect the sound card to the mixer).
did the problem chang sides with you?
if so one problem is one or both of those cables or any converters between the mixer and speakers.

the cut n dry is you need to get decent cables without any converter plugs.
this will save you time, trouble and money over time.

in the meantime you can connect the speakers to the sound card.

the shorter and simpler the audio path, the cleaner it is (like you heard).