I followed hints I saw in that discussion online: [SOLVED] - Strange FPS drops after few hours of playing | Tom's Hardware Forum
And I think I found more clues. I used HWMonitor free tool to see various metrics for my hardware right after starting the game, and after an hour of running it, when I see the drop in fps counter. And there are huge differences in voltages and frequencies for GPU in those cases:
As you can see, the core metrics show much more utilization of GPU initially, with it going less performant and hot later on. Then I remembered I also saw this thread: INVESTIGATION: Poor Performance & Power Draw Issues Impacting AMD Radeon 6000 - 9000 Series GPUs - #116 by Evono
So it seems like this is exactly the case? For some reason game just start under-utilize my GPU after some time, drawing less power?