The PCIe test do not completed sometimes, phase stucks

UVM
1

Hello everyone!

Now I work with my former colleague’s PCIe with PHY VIP environment. I rewrote a sequence with asserting and deasserting interrupts. Some tests (with some seeds) are completed successfully

PCIE INFO--(uvm_phase.svh: 1345      ) [PH/TRC/STRT              ] { 237770.0ns} Phase 'uvm.uvm_sched.post_shutdown' (id=842) Starting phase
PCIE INFO--(uvm_phase.svh: 1439      ) [PH/TRC/SKIP              ] { 237770.0ns} Phase 'uvm.uvm_sched.post_shutdown' (id=842) No objections raised, skipping phase
PCIE INFO--(uvm_phase.svh: 1620      ) [PH/TRC/DONE              ] { 237770.0ns} Phase 'common.run' (id=718) Completed phase
PCIE INFO--(uvm_phase.svh: 1655      ) [PH/TRC/SCHEDULED         ] { 237770.0ns} Phase 'common.extract' (id=727) Scheduled from phase common.run
....
PCIE INFO--(uvm_phase.svh: 1345      ) [PH/TRC/STRT              ] { 237770.0ns} Phase 'common.common_end' (id=704) Starting phase
PCIE INFO--(uvm_phase.svh: 1620      ) [PH/TRC/DONE              ] { 237770.0ns} Phase 'common.common_end' (id=704) Completed phase

but others stuck and failed at uvm_timeout time:

PCIE INFO--(uvm_phase.svh: 1655      ) [PH/TRC/SCHEDULED         ] { 232862.0ns} Phase 'uvm.uvm_sched.shutdown' (id=833) Scheduled from phase uvm.uvm_sched.pre_shutdown
PCIE INFO--(uvm_phase.svh: 1345      ) [PH/TRC/STRT              ] { 232862.0ns} Phase 'uvm.uvm_sched.shutdown' (id=833) Starting phase
PCIE INFO--(uvm_phase.svh: 1439      ) [PH/TRC/SKIP              ] { 232862.0ns} Phase 'uvm.uvm_sched.shutdown' (id=833) No objections raised, skipping phase
PCIE INFO--(uvm_phase.svh: 1620      ) [PH/TRC/DONE              ] { 232862.0ns} Phase 'uvm.uvm_sched.shutdown' (id=833) Completed phase
PCIE INFO--(uvm_phase.svh: 1655      ) [PH/TRC/SCHEDULED         ] { 232862.0ns} Phase 'uvm.uvm_sched.post_shutdown' (id=842) Scheduled from phase uvm.uvm_sched.shutdown
PCIE INFO--(uvm_phase.svh: 1345      ) [PH/TRC/STRT              ] { 232862.0ns} Phase 'uvm.uvm_sched.post_shutdown' (id=842) Starting phase
PCIE INFO--(uvm_phase.svh: 1439      ) [PH/TRC/SKIP              ] { 232862.0ns} Phase 'uvm.uvm_sched.post_shutdown' (id=842) No objections raised, skipping phase

I need an advice what part i need to debug, because i don’t find any uncompleted tasks right now

2

In reply to Levard:

Looks like you are usinng the sub-phases of the run_phase.
Are you sure you need them?.
Switch simply to the run_phase and it will run.

Could you please show some pieces of code wrt to sequence and the UVM components.

3

In reply to chr_sue:

so what I have

pe3rc_cfg_test - has pe3rc_cfg_seq where in body task regs are updated and then we wait for 1 bit to start other activity
pe3rc_cfg_test also extended from pe3rc_base_test where i have some randomization and setting components to uvm_config_db#

class pe3rc_int_test extends pe3rc_cfg_test;
...
        uvm_config_db#(uvm_object_wrapper)::set(this, "env.pcie_agt[*].driver_transaction_seqr[0].main_phase", "default_sequence", pcie_int_seq::type_id::get()); // pcie_int_seq - generate interrupts
...
endclass : pe3rc_int_test

the main sequence here, i randomly generate asserts and deasserts, memorize it in tmp variable, then after main loop deasserting what haven’t been deasserted
all things with svt_* is a from VIP environment from vendor


...
    task body();
        repeat (64) begin
//             if(int_slot != 4'hf) begin
                do_rand_again = 0;
                do begin
                    req = svt_pcie_driver_app_transaction::type_id::create("req");
                    req.cfg = cfg;
                    req.reasonable_pkt_delay_ns.constraint_mode(1'b0);
                    req.randomize() with {
                        transaction_type == MSG;
                        routing_type == LOCAL;
                        message_code inside {ASSERT_INTA, ASSERT_INTB, ASSERT_INTC, ASSERT_INTD, DEASSERT_INTA, DEASSERT_INTB, DEASSERT_INTC, DEASSERT_INTD};
                        ep == 1'b0;
                        pkt_delay_ns inside {[600:2000]};
                    };
                
                    case(req.message_code)
                        svt_pcie_driver_app_transaction::ASSERT_INTA: begin 
                            do_rand_again = int_slot[0] ? 1 : 0;
                            if(!do_rand_again) int_slot[0] = 1'b1;                            
                        end
                        ....
                    endcase
                    `uvm_info(get_name(), $sformatf("try current code=%s when int_slot=b'%0b", req.message_code, int_slot), UVM_LOW)
                end while (do_rand_again);
//             end
            
            `uvm_info(get_name(), req.sprint(uvm_default_line_printer), UVM_LOW)
            start_item(req);
            finish_item(req);
            get_response(req);

            #(req.pkt_delay_ns * 1ns);
        end// repeat (64) begin
        do_rand_again = 0;
        while(int_slot != 0) begin
            `uvm_info(get_name(), $sformatf("After test deasserting, current int_slot=b'%0b", int_slot), UVM_LOW)
            do begin
                req = svt_pcie_driver_app_transaction::type_id::create("req");
                req.cfg = cfg;                
                req.randomize() with {
                    transaction_type == MSG;
                    routing_type == LOCAL;
                    message_code inside {DEASSERT_INTA, DEASSERT_INTB, DEASSERT_INTC, DEASSERT_INTD};
                    ep == 1'b0;                    
                };                    
                ....                   
            end while (do_rand_again);
            
            `uvm_info(get_name(), req.sprint(uvm_default_line_printer), UVM_LOW)
            start_item(req);
            finish_item(req);
            get_response(req);

            #(req.pkt_delay_ns * 1ns);
        end
    endtask : body

endclass : pcie_int_seq

UVM topology is a very long list, I don’t know if I put in here would it be helpful or not

4

In reply to Levard:

Thanks for posting this code. This is not what I’m looking for.
Your error message points to shutdown/post_shutdown. Do you use these sub-phases or not and where are you doing this?

5

In reply to chr_sue:

In my environment I don’t use shutdown/post_shutdown, what about vendor’s classes, i don’t know
If same test with different seeds stucks sometimes and with other seeds passed, maybe to focus more on configuration of passed tests and stucked ones.

6

In reply to Levard:

This should not happen, that the seed has influence on pass/fail of a sequence.
What kind of fail dou see? Is it what you are describing above? Or is it differently.
To debug the issue I’d include some more diagnostic messages in your code.

But finally I do not see any UVM_ERROR in your past.

7

In reply to chr_sue:

I think maybe there is somewhere some task, that uses run_phase and turning on with some special random configuration that i have not find yet

There is no failure, i checked, only stucking after uvm.uvm_sched.post_shutdown and before common.run’ Completed phase

I use +UVM_OBJECTION_TRACE +UVM_PHASE_TRACE

8

In reply to Levard:

Do you see the UVM Summary Report at the end of the simulation?
This is en indication the simulation has a regular end.

9

In reply to chr_sue:

Now i have two logs: 1) stucked test stoped by timeout (no UVM report) 2) test passed

for 2 case

--- UVM Report catcher Summary ---


Number of demoted UVM_FATAL reports  :    0
Number of demoted UVM_ERROR reports  :    0
Number of demoted UVM_WARNING reports:    0
Number of caught UVM_FATAL reports   :    0
Number of caught UVM_ERROR reports   :    0
Number of caught UVM_WARNING reports :    0

PCIE INFO--(vm_report_server.svh: 847) [UVM/REPORT/SERVER        ] { 237770.0ns} 
--- UVM Report Summary ---

** Report counts by severity
UVM_INFO : 1690
UVM_WARNING :    1
UVM_ERROR :    0
UVM_FATAL :    0
** Report counts by id
[OBJTN_TRC]   635
[PH/TRC/DONE]    27
[PH/TRC/SCHEDULED]    26
[PH/TRC/SKIP]    10
[PH/TRC/STRT]    27
[RNTST]     1
[RegModel]    20
[TIMOUTSET]     1
[UVM/REPORT/CATCHER]     1
[UVMTOP]     1
[WAT]    16
[display_checked_out_features]     1
[mon]    50
[new]     4
[open_transaction_log_file]     1
[pcie_int_seq]   207
[pe3rc_int_test]     2
[pe3rc_scbd[0]]    69
[reconfigure_pciesvc_dl]     1
[report_message]   591

$finish called from file "/auto/vgr/uvm/uvm-1.2/src/base/uvm_root.svh", line 517.
$finish at simulation time 237770.0ns
           V C S   S i m u l a t i o n   R e p o r t 
Time: 237770049100 fs
10

In reply to Levard:

If you see this report anything is fine. You should look for the UVM_WARNING where it comes from and what is it saying.

11

In reply to chr_sue:

yes, it is good for passed test, for the stucked one I don not have this report
all of the warnings about RTL or vendors uvm modules

12

In reply to Levard:

Then you have to include diagnostic messages. You can do this even in the VIP model implementing callbacks, i.e. overriding mid_do or post_do callback.
You have to fond out where and why your execution stops.