HOME   IP   TECHNICAL BULLETIN   ARTICLE ARCHIVES   ARTICLE

DesignWare Technical Bulletin Article

Search Tools

Spotlight

PCI Express Switch Enumeration Using VMM-Based DesignWare Verification IP

Keith Young, Corporate Applications Engineer

PCI Express (PCIe) utilizes a point to point interconnect and uses switches to fan out and expand the number of PCIe connections in a system. Upon system boot up a critical task is the discovery or enumeration process of all the devices in the PCIe tree so they can be allocated by the system software. During the enumeration process the system software discovers all of the switch and endpoint devices that are connected to the system, determines the memory requirements and then configures the PCIe devices. The PCIe switch devices represent a special case in this process as their configuration is unique and separate from that of PCIe endpoints. In the simulation testbench environment however, only their configuration is required; the discovery process is not strictly necessary as the number of PCIe devices are known ahead of time. This paper will elucidate the process of switch configuration using Synopsys' PCI Express VMM simulation class libraries for SystemVerilog.

First, while the discovery process is not needed within the testbench environment, the testbench must still select the bus numbers and memory address of all the devices. These bus numbers are shown in Figure 1 as Bus#2, Bus#3, Bus#4, and Bus#5. Figure 1 shows the testbench that is used in the remainder of the article.


Figure 1. Example Testbench With Bus Numbers Assigned

This testbench in Figure 1 is testing a 3 port switch with one upstream port and two downstream ports. The switch in this example has no internal memory so as to simplify the configuration process. The enumeration process selects the bus numbers for each device that has a connection. For this example, the upstream port of the switch will be assigned bus #2. In the switch device, the internal bus between the upstream port and the two downstream ports will be assigned bus #3. The PCIe End Point (EP) devices that are downstream from the switch are Bus #4 and Bus # 5. (see Figure 1). The device numbers for both EP's will be 0.

The next step in the enumeration process is to determine the memory size of each device. Again, this step is skipped because the testbench environment sets the memory size of each device. To simplify this example, the switch will not have any internal memory, and each EP device will have 2M of memory. The EP device on bus 4 will start at address 2M. The EP on bus 5 will start at address 4M. The address range that the Switch will support is from 2M through 6M. Each EP device will only support 32-bit addresses.

Now that the bus numbers, device numbers and address range are known, the Switch can be configured. Each port of the Switch has its own configuration space and the format of the configuration space is type CONFIG 1. Refer to Table 1.

Byte OffsetByte 3Byte 2Byte 1Byte 0
0x00Device IDVendor ID
0x04Status RegisterCommand Register
0x08Class CodeRevision ID
0x0CBIST(0x00)Header TypeLatency TimerCache Line Size
0x10Base Address Register 0
0x14Base Address Register 1
0x18Secondary
Latency Timer
Subordinate
Bus Number
Secondary
Bus Number
Primary
Bus Number
0x1CSecondary StatusI/O LimitI/O Base
0x20Memory LimitMemory Base
0x24Prefetchable Memory LimitPrefetchable Memory Base
0x28Prefetchable Base Upper 32 Bits
0x2CPrefetchable Limit 32 Upper Bits
0x30I/O Limit Upper 16 BitsI/O Base Upper 16 Bits
0x34ReservedCapPtr
0x38Expansion ROM Base Address
0x3CBridge ControlInterrupt PinInterrupt Line
Table 1. PCI Configuration Space Header - Type 1

There are a minimum of 6 registers that need to be programmed for the Switch to support configuration and memory accesses to an EP device. They are:

  1.  Command : Enable memory access
  2.  Primary Bus : Bus number of upstream bus
  3.  Secondary Bus : Bus number of bus directly attached device
  4.  Subordinate Bus : Bus number of farthest downstream device
  5.  Memory Limit : Maximum memory address
  6.  Memory Base : Starting memory address

Each port of the switch has its own CONFIG-1 configuration space. Each of these CONFIG-1 spaces must be programmed. The upstream port needs to be programmed first. The following table shows the configuration parameters and their values that need to be programmed.

RegisterAddressRegisterValue
Command[15:0]0x040116'h0004
Primary Bus[7:0]0x1806 8'b00000010
Secondary Bus[7:0]0x1806 8'b00000011
Subordinate Bus[7:0]0x18068'b00000110
Memory Base[15:0]0x200816'h0020
Memory Limit[15:0]0x200816'h0060
Table 2. Upstream Port Configuration Values

The upstream port configuration space is programmed using CONFIG-0 packets. CONFIG-0 packets use the bus number, device number, function number, and register number to access the register to be programmed. The following VMM task will program the upstream Switch port with the previously listed values.

//  Program Upstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Upstream_switch_config();
  begin
      bit test;
      bit status;
  
      dw_vip_pcie_tlp_transaction  cfg0_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_0);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg0_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };

      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    

      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg0_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00050302;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg0_wr.randomize() with { 
          m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h02; m_bvDevNum == 5'h00;
	   m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00600020;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   
  end
endtask

The downstream port (Device 1 ) has the following values as shown in Table 3.

RegisterAddressRegisterValue
Command[15:0]0x040116'h0004
Primary Bus[7:0]0x1806 8'b00000011
Secondary Bus[7:0]0x1806 8'b00000100
Subordinate Bus[7:0]0x18068'b00000100
Memory Base[15:0]0x200816'h0020
Memory Limit[15:0]0x200816'h0040
Table 3. Downstream Port Configuration Values

The downstream ports configuration space is programmed using CONFIG-1 packets. CONFIG-1 packets also use bus number, device number, function number, and register number to access the register to be programmed. The following VMM task will program downstream device 1 of the downstream port.

//  Program Device 1 Downstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Downstream_device_1_switch_config();
  begin
      bit test;
  
      dw_vip_pcie_tlp_transaction  cfg1_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_1);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg1_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00040403;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00400020;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


  end
endtask

The downstream port (Device 2 ) has the following configuration values as shown in Table 4.

RegisterAddressRegisterValue
Command[15:0]0x040116'h0004
Primary Bus[7:0]0x1806 8'b00000011
Secondary Bus[7:0]0x1806 8'b00000101
Subordinate Bus[7:0]0x18068'b00000101
Memory Base[15:0]0x200816'h0040
Memory Limit[15:0]0x200816'h0060
Table 4. Downstream Port Configuration Values

The downstream ports configuration space is programmed using CONFIG-1 packets. CONFIG-1 packets also use bus number, device number, function number, and register number to access the register to be programmed. The following VMM task will program downstream device 2 of the downstream port.

//  Program Device 2 Downstream Port
//    1)  Write Command register
//    2)  Write Subordinate, Secondary and Primary Bus Numbers
//    3)  Write Memory Limit and Memory Base
task  pcie_rvm_env::Downstream_device_2_switch_config();
  begin
      bit test;
  
      dw_vip_pcie_tlp_transaction  cfg1_wr = 
                                   new ( , dw_vip_pcie_tlp_transaction::CFG_WR_1);
      dw_vip_pcie_tlp_transaction  cfg_tlp;

      //  Write COMMAND REGISTER
      //  Enable memory cycles
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h02; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h001;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00000004;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg1_wr.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   


      //  Write Subordinate Bus, Secondary Bus and Primary Bus Numbers:  
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h02; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h006;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00050503;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

     //  Write Memory Limit and Memory Base
      test = cfg1_wr.randomize() with { 
           m_bvRequesterId == `TBD_REQ_ID;
           m_bvBusNum == 8'h03; m_bvDevNum == 5'h01; 
           m_bvFuncNum == 3'h0; m_bvRegNum == 10'h008;
           m_bvFirstDWBE == 4'hF;  m_bTD == 1'b0; 
           m_bvvPayload[0] == 32'h00600040;
      };
      if (!test) begin
          `vmm_error(log, "TBD Configuration request packet failed to randomize.");
      end
   
      $cast(cfg_tlp, cfg_wr1.copy());
      tbd_gasket.m_oTlpTxInputChan.put(cfg_tlp);    
      cfg_tlp.notify.wait_for(vmm_data::ENDED);   

  end
endtask

The Switch is now ready to accept configuration and memory packets that are for the EP devices connected to the downstream ports of the switch.

In summary, PCIe designs must go through the process of Switch enumeration to discover available switches, and then to configure them. This paper has shown how you can use VMM based PCIe Verification IP and SystemVerilog to perform the task of configuration during the process of Switch enumeration. The examples use the rich set of class, protocol, and packet capabilities of the VMM models to perform the configuration task.