I was able to solve the problem myself - so this is for anyone who might stumble upon the same problem in the future.
The problem was the inertia (momentum) of the filament spool. The filament spool is held by the famous "Universal filament spool holder" that uses two ball bearings at each end and is extremely easy to move. So when the extruder has faster moves - it pulls the spool and it's inertia pushes the filament a bit more through the sensor than it is actually going into the extruder since the filament can bend and straighten - hence it all makes sense - that sometimes there is more, sometimes less filament going through the sensor - but the average is always exactly the amount that went into the extruder.
The solution is to constrain the filament path before and after the sensor - either by a tighter PTFE tube on both sides or maybe by a sponge filament cleaner at the entrance of the sensor.
If you are using the "Universal filament spool holder" - you can also put two PTFE washers on the sides of the ball bearings and squeeze them a bit with a locking hex nut - that would prevent the spool from rolling more than requested, but would still roll nicely.
Now my readings are normal.
Hope it helps someone having the same problem with the laser filament sensor.