Thursday, July 31, 2014

Capturing Ethernet header, adjusting packetizer speed and moving between OS

 I was asked about what is the inter-packet gap (IPG) and can this be varied. I checked and it turns out that IPG is inserted by PHY and not MAC. So MAC has no control over it. Since we are using Marvell 88E1111 PHY, the PHY inserts preamble, start of frame delimiter, CRC and inter-packet gap. The following wikipedia post is really useful

http://en.wikipedia.org/wiki/Ethernet_frame

Then  i was after capturing Ethernet Header ( preamble, start of frame delimiter, CRC and inter-packet gap) but this is not possible given the hardware (NIC + Ubuntu) that i have.

This stackexchnage  post is useful.

http://serverfault.com/questions/521443/can-wireshark-capture-an-entire-ethernet-frame-including-preamble-crc-and-inter

I came across the following raw Ethernet Capture tool.

https://gist.github.com/austinmarton/2862515

I Compiled its C source code to capture raw packets but for some reason, NIC + Ubuntu are stripping this information (preamble, start of frame delimiter, crc and ipg) and packets start from MAC header  (dst mac address, src MAC address ...)

http://pastebin.com/gMPnzy0g

Then i did calculation based on the transmitter packetizer to see how fast its throughput is. It turns out that it is actually 3x faster than 1 Gbps. As mentioned earlier, either the transmitter should slow down or the receiver should catch up. I am sticking to the first approach of slowing down the transmitter. Below is how and by how much?

I calculated the time to send one packet given 30 fps and then found out the delay to insert between packets (3000 clock cycles, where one cycle = 1/125 Mhz). I shall post these calculations tomorrow.


After configuring 3000 cycles delay between packets, i sent 102 packets (0 to 101 ) (see the last to last blog) from FPGA to PC (Ubuntu 13.10 machine). It turned out that Gstreamer got stuck at packet # 97 and never got to receive packet 98 till 101. Wireshark behaved differently from Gstreamer. It showed a couple of missing packets but those packets were successfully captured by Gstreamer AND Wireshark showed couple of packets that Gstreamer could not capture (e.g., packet # 98, 99, 100 and 101).

 It seemed that the transmitter needed to slow down further.So i increased the inter-packet delay  to 5000 clock cycles but still Gstreamer would get stuck at packet # 97. I knew that if i increase the delay to 20,000 clock cycles, it would work but that is not ideal as it would mean < 10 fps.

So i moved to my laptop that has 1 Gbps NIC but different Operating System (Windows 7) and used a newe Cat 6 cable. I configured the FPGA to send 102 packets to the laptop where the inter-packet delay is configured to be 3000 clock cycles. I was running Wireshark in Windows 7. It turned out that ALL the packets got captured and Wireshark did n't drop even a single packet.  I could not get Gstreamer to work on Windows 7 for some reason. Gstreamer runs but does not produce any output.

Here is the successful 102 packets captured by Wireshark in Windows 7 with inter-packet delay of 3000 clock cycles.

http://pastebin.com/Rw7FXKQd

You may want to look at the older blog post to decipher the above packets.


The point is that successful capture depends upon the NIC, cable and OS. I shall see if can get a recent NIC to reproduce the same behavior with Ubuntu 13.10 machine. Ethtool provides the following information about NIC

http://pastebin.com/VRLUJwLL

The fact that there is no mention of GMII is intriguing. What do you think?


I am on un-official vacation from tomorrow till next Friday. I shall work on this as i get time. The blogs might not be updated regularly. The remaining goal is further testing and documentation.




5 comments:

  1. Tariq,

    Can you provide some more information why you believe the PHY is handling the preamble, start of frame, and CRC?

    I can show you where the GEMAC is actually inserting each of the above. You would not see these in your simulations if they were inserted by the PHY - and each of these fields exist in the HDL simulation.

    Regards,
    Chris

    ReplyDelete
    Replies
    1. This has me a little concerned, the previous post indicated all the sequence number were transferred. But I still don't know where the sequence numbers were probed - if they were probed at the source (probed with chipscope) all we know is that all packets were generated. If the sequence number was extracted from the GMII interface (GMII was probed) then we know it made it out the GEMAC. In addition, if the GMII was probed you would of seen the preamble etc.

      Delete
  2. Yes, it makes sense that the NIC (the MAC in the PC) strips these low-level fields. These do not need to be passed to the app levels. You would need an ethernet analyzer or as Tim suggested a scope with ethernet decoding (but you probably would have a limited buffer). It might be difficult to probe and view the bits of the cable directly.

    If you had a second FPGA and a Gbps hub you could build your own :) Not realistic in the near term.

    Regards,
    Chris

    ReplyDelete
  3. """
    Then i did calculation based on the transmitter packetizer to see how fast its throughput is. It turns out that it is actually 3x faster than 1 Gbps. As mentioned earlier, either the transmitter should slow down or the receiver should catch up. I am sticking to the first approach of slowing down the transmitter. Below is how and by how much?
    """

    I don't understand the above statement - if you are generating packets at 3x the 1Gbps you know what your issue is. The GEMAC (simple_gemac_wrapper) will only transfer at max 1Gbps if you push anything faster without a significant gap (as I pointed out you are using very small buffer sizes) things will be dropped.

    But previously you indicated that the flow control is being handled correctly - if this is true
    true, then I don't understand where the 3x calculation comes from. Or are you saying
    the packetizer *can* generate 3x but it will slow down based on the GEMAC 'src_rdy' flow signal?

    Either way the above is an incomplete and some clarification would be useful.

    Regards,
    Chris

    ReplyDelete
  4. I am so confused by this post. Your original PC receiver did not have a GigE NIC (1 Gbps) card?

    """
    It seemed that the transmitter needed to slow down further.So i increased the inter-packet delay to 5000 clock cycles but still Gstreamer would get stuck at packet # 97. I knew that if i increase the delay to 20,000 clock cycles, it would work but that is not ideal as it would mean < 10 fps.
    """

    Why did you know that it would work with a 20k delay in the packetizer? And yes this would break the requirement but this is very important for debugging. Get if functionally working first and then move to performance.

    """
    So i moved to my laptop that has 1 Gbps NIC but different Operating System (Windows 7) and used a newe Cat 6 cable. I configured the FPGA to send 102 packets to the laptop where the inter-packet delay is configured to be 3000 clock cycles. ...

    Here is the successful 102 packets captured by Wireshark in Windows 7 with inter-packet delay of 3000 clock cycles.
    """

    This seems to make sense, when using a 1Gbps Ethernet interface on the PC it works. I am missing something? Did we just spend a lot of time trying to figure out why 1Gbps rate did not work on a non 1Gbps interface (please tell me I am wrong!).

    Going back to a post a couple days ago, I really would of like to see you slow down the packets to a crawl and then slowly increase them and indicate when they failed. I think a more formal test/debug approach would have saved a bunch of time and effort (trying to compile C packet capturing software tool, etc.). It seems there was more of a fundamental misunderstanding of major components in the system than a specific bug that could of been causing heart-ache.

    Regards,
    Chris

    ReplyDelete