Capturing Ethernet header, adjusting packetizer speed and moving between OS

 I was asked about what is the inter-packet gap (IPG) and can this be varied. I checked and it turns out that IPG is inserted by PHY and not MAC. So MAC has no control over it. Since we are using Marvell 88E1111 PHY, the PHY inserts preamble, start of frame delimiter, CRC and inter-packet gap. The following wikipedia post is really useful

http://en.wikipedia.org/wiki/Ethernet_frame

Then  i was after capturing Ethernet Header ( preamble, start of frame delimiter, CRC and inter-packet gap) but this is not possible given the hardware (NIC + Ubuntu) that i have.

This stackexchnage  post is useful.

http://serverfault.com/questions/521443/can-wireshark-capture-an-entire-ethernet-frame-including-preamble-crc-and-inter

I came across the following raw Ethernet Capture tool.

https://gist.github.com/austinmarton/2862515

I Compiled its C source code to capture raw packets but for some reason, NIC + Ubuntu are stripping this information (preamble, start of frame delimiter, crc and ipg) and packets start from MAC header  (dst mac address, src MAC address ...)

http://pastebin.com/gMPnzy0g

Then i did calculation based on the transmitter packetizer to see how fast its throughput is. It turns out that it is actually 3x faster than 1 Gbps. As mentioned earlier, either the transmitter should slow down or the receiver should catch up. I am sticking to the first approach of slowing down the transmitter. Below is how and by how much?

I calculated the time to send one packet given 30 fps and then found out the delay to insert between packets (3000 clock cycles, where one cycle = 1/125 Mhz). I shall post these calculations tomorrow.


After configuring 3000 cycles delay between packets, i sent 102 packets (0 to 101 ) (see the last to last blog) from FPGA to PC (Ubuntu 13.10 machine). It turned out that Gstreamer got stuck at packet # 97 and never got to receive packet 98 till 101. Wireshark behaved differently from Gstreamer. It showed a couple of missing packets but those packets were successfully captured by Gstreamer AND Wireshark showed couple of packets that Gstreamer could not capture (e.g., packet # 98, 99, 100 and 101).

 It seemed that the transmitter needed to slow down further.So i increased the inter-packet delay  to 5000 clock cycles but still Gstreamer would get stuck at packet # 97. I knew that if i increase the delay to 20,000 clock cycles, it would work but that is not ideal as it would mean < 10 fps.

So i moved to my laptop that has 1 Gbps NIC but different Operating System (Windows 7) and used a newe Cat 6 cable. I configured the FPGA to send 102 packets to the laptop where the inter-packet delay is configured to be 3000 clock cycles. I was running Wireshark in Windows 7. It turned out that ALL the packets got captured and Wireshark did n't drop even a single packet.  I could not get Gstreamer to work on Windows 7 for some reason. Gstreamer runs but does not produce any output.

Here is the successful 102 packets captured by Wireshark in Windows 7 with inter-packet delay of 3000 clock cycles.

http://pastebin.com/Rw7FXKQd

You may want to look at the older blog post to decipher the above packets.

The point is that successful capture depends upon the NIC, cable and OS. I shall see if can get a recent NIC to reproduce the same behavior with Ubuntu 13.10 machine. Ethtool provides the following information about NIC

http://pastebin.com/VRLUJwLL

The fact that there is no mention of GMII is intriguing. What do you think?


I am on un-official vacation from tomorrow till next Friday. I shall work on this as i get time. The blogs might not be updated regularly. The remaining goal is further testing and documentation.

Today's post will be merged with tomorrow's post

I had to go out of town for most of the day so will update the blog tomorrow (Thursday). Please stay tuned.

Thanks


Has anyone tried reproducing some of this work. Please share your success or failure story.

Last day of debugging 1 Gbps transmitter's MAC

Today the first thing was testing the packetizer via simulation and the last thing that was emulation using Xilinx Chipscope, which took majority of the time as it is tricky and needs lot of debugging. The goal of the today's activity was to make sure that sender (which in this case is FPGA) does not miss packets or drop packets. For example, if FPGA is configured to send 102 packets ( which is sending 34 lines of 720p frame where each line is 1280 pixels and each line is sent as 3 packets ), it sends all the packets in a sequence starting from 0 to 101 without missing any packet from 0 to 101. The reason 102 packets were chosen because receiver PC randomly misses a few of the packets from 102 packets. This was confirmed by Wireshark. Every time, FPGA sends 102 packets, the receiver PC randomly drops 4 to 6 packets from 102 packets. If the number of packets sent from the transmitter FPGA are increased, then receiver drops more packets and these are random every time transmitter sends packets.

So before blaming the receiver for packet drop, it is good to check the transmitter. As mentioned above, i went through the chain of simulation. Please refer to the original proposal where i highlighted the importance of simulation on Pages 10 and 11. Simulation is done after each step of Xilinx Build flow as show in the Figure below.

 

Here are the snapshots that show RTL, post-synthesis, post-translate, post-map and post-pnr simulations. Post-pnr simulation is actually done with SDF delay annotation. All these simulations show that packets went from sequence number 0 to 101

Then after this came the step of verifying with Xilinx Chipscope, which took the whole day. Here are sequence of shots that show sequence number going from 0 to 101. 

After verifying through simulation chain and emulation using Chipscope, we are confident that transmitter is not dropping or missing packets.

Now we either need to slow down the transmitter or make sure receiver is fast enough to capture the transmitter. This is the aim for the rest of the week.

Please share your feedback

Thanks! 

The 2nd last day of 1 Gbps debugging

Today it came to light that complete simulation environment cannot be built as parts of the code actually probe PHY status registers. So the idea of top level design testbench did n't work. So i am back to unit testing and making sure that there is nothing in the code that is responsible for generating conditions that make a packet drop from MAC. After that i shall send traffic from FPGA to PC and see if the receiver PC's MAC is able to catch up or not by doing ifconfig or other utilities that can help detect packet loss. It seems likely that receiver MAC is not able to keep up.

This whole saga should end tomorrow. I am pretty hopeful.

Other than that we celebrated our holy Eid today. So spending some time with the family is good. It is good to take a break from GSoC.

1 Gbps MAC debugging continues on the weekend

Today, it turned out that 1 Gbps MAC was actually dropping packets. I tried to debug this by

1) Creating simulation environment for the entire design (packetizer + MAC)
2) Changing transmit fifo depth in gigmac from 6 to 12 and up
3) Creating internal reset for packetizer as the actual design gets reset externally from FPGA. Look at lines 148-155
http://pastebin.com/HXmiiGU0

Right now, simulation is not working as gemac_ready signal (see above pastebin) which is reset for packetizer is still not toggling. I shall look at it tomorrow Monday, although Monday is a special day.

If this doesn't really work, then  Chipscope is the next step. There is a possibility that 1 Gbps MAC cannot handle such high HDMI throughput but that is what i am after to confirm.

Happy Debugging.

1 Gbps MAC Debugging continues

I tried to debug the MAC. I found something weird and that was Wireshark was capturing all the packets while Gstreamer was not capturing the same packets and still waiting for some packets. I am trying to debug this further for more packets (for a complete 720p frame and back to back frames). I suspect from debugging today that MAC is fine. I'll confirm this soon.

Happy Debugging.

1 Gbps MAC debugging

If you recall, last week, packets sent by Packetizer were getting dropped or were missing. It turned out that 1 Gbps MAC that is integrated with Packetizer was overflowing despite making sure all flow control protocols were followed while interfacing with the MAC. To deal with this, delay was inserted between each RTP packet so MAC never overruns. It fixed everything and you could see entire RED frame, RGB flag and RGB loop. The only thing that got hurt is the frame per second.

So i am trying to figure out how to debug this MAC and get the fps up. I shall see if i could post some calculations of the current implementation.

Happy Debugging.