This is dependent on how hash rate is timed on the mining software. The transit time between submitting to a proxy / pool and getting the next batch will result in a zig zag pattern for total hashrate. Hashrate is never flat. Xmrig for example will report on a 5sec interval, if you happen to take 1 second to round trip to your proxy / pool your hashrate will be 0 for that second - resulting in a lower 5sec average.Latency doesn't have any bearing on hashrate. It does have a relationship with the pool's estimate of your hashrate, but that's calculated from difficulty and share submission interval, and is never going to be accurate. Based on the description of hashrate going down a mere 4H/s it's not the pool variance here, that's a local reading from the software.
Your workers ability to do work is completely based on being able to talk over stratum. Hence why it's better to tailor your own difficulty so you spend more time hashing, and not round tripping to your upstream. A 10KH/s machine hashing at 1000 difficulty will have a lower hash rate due to latency and frequency, than a difficulty of 30000.