How to override instance of class extended from uvm_transaction using "set_inst_override_by_type"?

A class object usually can’t be overriden by set_inst_override(as uvm_object’s are not part of the TB component hierarchy), unless some special care is taken during the creation of the object that needs to be overridden.

Add either get_full_name() or some text pattern as the third argument for the create() call during the object creation and use the same as argument for overriding. Example given below:

b1 = B::type_id::create("b1", ,"some_pattern_abc");
factory.set_inst_override_by_type(B::get_type(), B_override::get_type(),  some_pattern_abc.b1);

(You may need to do double check for syntax errors)
Check this UVM cookbook link sequence overrides for complete details.