You should not use any timing constructs (# delays) in your testbench code. UVM is designed so that the entire testbench (outside of the BFM functionality of the drivers) is transaction based and completely untimed. By adding these constructs, you are breaking this core tenet.
If you are waiting on your design to achieve a specific state based on an internal register value, I would recommend doing front door register accesses to poll until that value is reached. Doing the bus accesses won’t have any impact on simulation performance.
Also, scoreboards should be used as a component to compare transactions and verify that they are correct at the point they are monitored. Again, this is untimed and eliminates any need to check internal states of your design. If you want to check the internal functionality, bind in additional monitors or use the register back door function to check the values when desired, but don’t ‘wait’ on these register values in the scoreboard.