Bye World!

You might have heard Hello World! in almost all programming languages when the language is introduced. Today is the time to say Bye World! as it is the last official day of Google Summer of Code 2014. This is by no means to say that this blog is dead. This blog will continue to survive and i will continue to contribute after today. It was a good learning exercise and learning should not stop.

Thanks to TimVideos, Google and all those who shaped, mentored and gave feedback about the project.

Looking back, the proposal went through several iterations of proposal before being accepted. Right from the beginning, the expectations were high and that was a good motivation to work. The start of the project was very broad : looking this, evaluating that etc etc. After a while, things started to converge. As the time went by, i started to become more independent: i only had one VC with my mentor after midterm. Being independent is a good thing and i want to thank my mentor that he enforced this. Overall more time was spent debugging as big portion of the project was integration and testing on the host PC.

If you have comments or questions, please keep posting. Besides me, you are going to make this blog alive !

Final Report and Documentation

Today was spent making final report and finishing Documentation. Bits and pieces of Documentation are still left which i will get to later. They don't need to be addressed right away.

Please review these and share your comments. I might not be able to address your comments immediately as i am feeling unwell.

Documentation Continues towards finishing

Today was spent updating the bitbucket repository and Documentation. The Documentation refers to bitbucket quite a bit. Besides that i uploaded the static image Demo (see yesterday's post as well).

Now I hope to get sometime to work on DDR2 as that is in good progress. I hope to finish this bonus step by Friday.

Documentation and adding test images continues

Today, i added a test image generated from VTC_DEMO (see yesterday's post). What this means is that besides RGB loop, now this test image can be seen. You need to select this test image by turning switch sw7 up on the Atlys FPGA. Previously, the ideas was to use switches sw7 and sw6 for future purposes. But today i ended up using sw7 for test image.  Here is a Demo of the test image.

This also brings us closer to DDR2 integration. Right now, we are confident that HDMI traffic from VTC_DEMO is correctly captured in a file which is then read by packetizer using $readmemb. $readmemb is synthesized into BRAM or LUTs. That is how test image is read by packetizer, which eventually sends it to downstream MAC and off to the destination PC.

 Documentation has been updated as well. It will continue this week.

Please share your comments directly on the Documentation or here.

Thanks

Spending sometime working on DDR2 interface and resuming Documentation

If you recall, i had started Specification document for the project. Now is the time to put more writing into it and complete it by the end of this week by Sunday. I also spent sometime working on DDR2 interface today. From now on, more time (or perhaps most of the time) will be spent on the documentation and less on DDR2 integration. If documentation is complete and there is time left, DDR2 integration will continue. For DDR2 integration, we are going to use VTC_DEMO from Xilinx Application Note 495 as HDMI source. So HDMI traffic (720p) from VTC_DEMO will be written to DDR2 which will then be read by RTP packetizer and sent as RTP packets across 1 Gbps link to the host PC. Host PC will decode these packets using Gstreamer and play them back. I hope that was a good summary.

Please look at the Specification document for some new sections on HDMI. The ideas is to put the following sections so anyone (beginner with no knowledge or an expert) can understand the project and be able to reuse it. Following sections are going to be added to the rest of the documentation. This is not a complete list and things may be added or subtracted as needed.

1) HDMI basic explanation (what is a 720p frame, what are the relevant signals and how to know when a line (row) starts and when it is finished and when a frame starts and when it is finished?
2) Gigabit Ethernet MAC
3) RTP Packetizer
4) Frame Calculation
5) Testing on the Host PC using Gstreamer (how to configure the host PC, how to increase or decrease the frame rate using switches on Atlys FPGA)
6) Future Directions


Documentation is to be continued ...

Successful transmission of packets and recovery

Hi,
Good to see you after a break. I was at a Conference last week and after the end of each day, i dedicated some time to figuring out why packets were getting dropped and how to correct that. Finally, i got it working after lot of debugging using simulation, Chipscope pro and Googling. Anyway, i tried a couple of options. Some worked and some didn't. These were:

1) Adding latency (number of milliseconds before a packet is declared dropped) to the Gstreamer command

time gst-launch-1.0 -v udpsrc uri=udp://192.168.1.1:5004  caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGB, depth=(string)8, width=(string)1280, height=(string)720" ! rtpbin latency=1 ! rtpvrawdepay  ! videoconvert !  autovideosink

I also tried with latency of 0 and it worked. Now the inter-packet delay is configurable using Atlys Switches sw0 to sw5. The default delay is 3000 clock cycles which corresponds to approximately 30 fps. Here is the configuration.

sw5 sw4 sw3 sw2 sw1 sw0   Delay in clock cycles
0      0       0    0       0     0            3000
0      0       0    0       0     1            2625
0      0       0    0       1     0            2250
0      0       0    0       1     1            1875
0      0       0    1      0     0             1500
0      0       0    1       0     1            1125
0      0       0    1       1     0              750
0      0       0    1       1     1              375

0      0       1    0       0     0            5500
0      1       0    0       0     0            8000
0      1       1    0       0     0           10500
1      0       0    0       0     0           13000
1      0       1    0       0     0           15500
1      1       0    0       0     0           18000
1      1       1    0       0     0           20500

sw6 is reserved for jumbo frame size and sw7 is reserved for 1080p (for future tweaks)

2) By Tcp tuning in Linux and Windows 7

I  tried without tuning and it worked. The tuning was done to see how much of inter-packet delay can be decreased. It turned out that as you approach to 0 inter-packet delay, Tcp tuning does not work

3) Going from Cat 5 to Cat 6 cable and it didn't make a difference.

4) Going from Linux to Windows 7 as my laptop had Windows 7 and i was stuck with this last week. It was not different going from Linux to Windows 7 except that Gstreamer didn't work in Window 7 for some reason.

Here is a quick demo. You will see frame rate going up (faster RGB pattern) and then going down (slow RGB pattern)


Now i shall work on DDR2 integration and documentation this week.

Please give your valuable feedback

Capturing Ethernet header, adjusting packetizer speed and moving between OS

 I was asked about what is the inter-packet gap (IPG) and can this be varied. I checked and it turns out that IPG is inserted by PHY and not MAC. So MAC has no control over it. Since we are using Marvell 88E1111 PHY, the PHY inserts preamble, start of frame delimiter, CRC and inter-packet gap. The following wikipedia post is really useful

http://en.wikipedia.org/wiki/Ethernet_frame

Then  i was after capturing Ethernet Header ( preamble, start of frame delimiter, CRC and inter-packet gap) but this is not possible given the hardware (NIC + Ubuntu) that i have.

This stackexchnage  post is useful.

http://serverfault.com/questions/521443/can-wireshark-capture-an-entire-ethernet-frame-including-preamble-crc-and-inter

I came across the following raw Ethernet Capture tool.

https://gist.github.com/austinmarton/2862515

I Compiled its C source code to capture raw packets but for some reason, NIC + Ubuntu are stripping this information (preamble, start of frame delimiter, crc and ipg) and packets start from MAC header  (dst mac address, src MAC address ...)

http://pastebin.com/gMPnzy0g

Then i did calculation based on the transmitter packetizer to see how fast its throughput is. It turns out that it is actually 3x faster than 1 Gbps. As mentioned earlier, either the transmitter should slow down or the receiver should catch up. I am sticking to the first approach of slowing down the transmitter. Below is how and by how much?

I calculated the time to send one packet given 30 fps and then found out the delay to insert between packets (3000 clock cycles, where one cycle = 1/125 Mhz). I shall post these calculations tomorrow.


After configuring 3000 cycles delay between packets, i sent 102 packets (0 to 101 ) (see the last to last blog) from FPGA to PC (Ubuntu 13.10 machine). It turned out that Gstreamer got stuck at packet # 97 and never got to receive packet 98 till 101. Wireshark behaved differently from Gstreamer. It showed a couple of missing packets but those packets were successfully captured by Gstreamer AND Wireshark showed couple of packets that Gstreamer could not capture (e.g., packet # 98, 99, 100 and 101).

 It seemed that the transmitter needed to slow down further.So i increased the inter-packet delay  to 5000 clock cycles but still Gstreamer would get stuck at packet # 97. I knew that if i increase the delay to 20,000 clock cycles, it would work but that is not ideal as it would mean < 10 fps.

So i moved to my laptop that has 1 Gbps NIC but different Operating System (Windows 7) and used a newe Cat 6 cable. I configured the FPGA to send 102 packets to the laptop where the inter-packet delay is configured to be 3000 clock cycles. I was running Wireshark in Windows 7. It turned out that ALL the packets got captured and Wireshark did n't drop even a single packet.  I could not get Gstreamer to work on Windows 7 for some reason. Gstreamer runs but does not produce any output.

Here is the successful 102 packets captured by Wireshark in Windows 7 with inter-packet delay of 3000 clock cycles.

http://pastebin.com/Rw7FXKQd

You may want to look at the older blog post to decipher the above packets.

The point is that successful capture depends upon the NIC, cable and OS. I shall see if can get a recent NIC to reproduce the same behavior with Ubuntu 13.10 machine. Ethtool provides the following information about NIC

http://pastebin.com/VRLUJwLL

The fact that there is no mention of GMII is intriguing. What do you think?


I am on un-official vacation from tomorrow till next Friday. I shall work on this as i get time. The blogs might not be updated regularly. The remaining goal is further testing and documentation.

Today's post will be merged with tomorrow's post

I had to go out of town for most of the day so will update the blog tomorrow (Thursday). Please stay tuned.

Thanks


Has anyone tried reproducing some of this work. Please share your success or failure story.

Last day of debugging 1 Gbps transmitter's MAC

Today the first thing was testing the packetizer via simulation and the last thing that was emulation using Xilinx Chipscope, which took majority of the time as it is tricky and needs lot of debugging. The goal of the today's activity was to make sure that sender (which in this case is FPGA) does not miss packets or drop packets. For example, if FPGA is configured to send 102 packets ( which is sending 34 lines of 720p frame where each line is 1280 pixels and each line is sent as 3 packets ), it sends all the packets in a sequence starting from 0 to 101 without missing any packet from 0 to 101. The reason 102 packets were chosen because receiver PC randomly misses a few of the packets from 102 packets. This was confirmed by Wireshark. Every time, FPGA sends 102 packets, the receiver PC randomly drops 4 to 6 packets from 102 packets. If the number of packets sent from the transmitter FPGA are increased, then receiver drops more packets and these are random every time transmitter sends packets.

So before blaming the receiver for packet drop, it is good to check the transmitter. As mentioned above, i went through the chain of simulation. Please refer to the original proposal where i highlighted the importance of simulation on Pages 10 and 11. Simulation is done after each step of Xilinx Build flow as show in the Figure below.

 

Here are the snapshots that show RTL, post-synthesis, post-translate, post-map and post-pnr simulations. Post-pnr simulation is actually done with SDF delay annotation. All these simulations show that packets went from sequence number 0 to 101

Then after this came the step of verifying with Xilinx Chipscope, which took the whole day. Here are sequence of shots that show sequence number going from 0 to 101. 

After verifying through simulation chain and emulation using Chipscope, we are confident that transmitter is not dropping or missing packets.

Now we either need to slow down the transmitter or make sure receiver is fast enough to capture the transmitter. This is the aim for the rest of the week.

Please share your feedback

Thanks! 

The 2nd last day of 1 Gbps debugging

Today it came to light that complete simulation environment cannot be built as parts of the code actually probe PHY status registers. So the idea of top level design testbench did n't work. So i am back to unit testing and making sure that there is nothing in the code that is responsible for generating conditions that make a packet drop from MAC. After that i shall send traffic from FPGA to PC and see if the receiver PC's MAC is able to catch up or not by doing ifconfig or other utilities that can help detect packet loss. It seems likely that receiver MAC is not able to keep up.

This whole saga should end tomorrow. I am pretty hopeful.

Other than that we celebrated our holy Eid today. So spending some time with the family is good. It is good to take a break from GSoC.

1 Gbps MAC debugging continues on the weekend

Today, it turned out that 1 Gbps MAC was actually dropping packets. I tried to debug this by

1) Creating simulation environment for the entire design (packetizer + MAC)
2) Changing transmit fifo depth in gigmac from 6 to 12 and up
3) Creating internal reset for packetizer as the actual design gets reset externally from FPGA. Look at lines 148-155
http://pastebin.com/HXmiiGU0

Right now, simulation is not working as gemac_ready signal (see above pastebin) which is reset for packetizer is still not toggling. I shall look at it tomorrow Monday, although Monday is a special day.

If this doesn't really work, then  Chipscope is the next step. There is a possibility that 1 Gbps MAC cannot handle such high HDMI throughput but that is what i am after to confirm.

Happy Debugging.

1 Gbps MAC Debugging continues

I tried to debug the MAC. I found something weird and that was Wireshark was capturing all the packets while Gstreamer was not capturing the same packets and still waiting for some packets. I am trying to debug this further for more packets (for a complete 720p frame and back to back frames). I suspect from debugging today that MAC is fine. I'll confirm this soon.

Happy Debugging.

1 Gbps MAC debugging

If you recall, last week, packets sent by Packetizer were getting dropped or were missing. It turned out that 1 Gbps MAC that is integrated with Packetizer was overflowing despite making sure all flow control protocols were followed while interfacing with the MAC. To deal with this, delay was inserted between each RTP packet so MAC never overruns. It fixed everything and you could see entire RED frame, RGB flag and RGB loop. The only thing that got hurt is the frame per second.

So i am trying to figure out how to debug this MAC and get the fps up. I shall see if i could post some calculations of the current implementation.

Happy Debugging.

Time for a visual treat !

Last time, i said a picture is worth a thousand words. How about video? Video is frames per second x 1000.

Please see the following RGB Video

                                                 RGB Video

Debugging pays off fully

Picture is worth a thousand words. I am going write less and put two images. You can tell which one is better. I was able to create both using Gstreamer. Again a big thanks to Gstreamer community

Jittery Image

and

Clean Image


Here is a log file if you are really interested in seeing the underlying nitty gritty

Debugging pays off a bit

First, note that the yesterday's blog is marked by Google as today's (Monday blog). Please treat it as Sunday's blog. Today, the day was spent debugging why packets were missing? It finally came to light that the 1 Gbps MAC was overflowing. I adjusted the delay to make it not overflow.

Here is what i get when i send the 720p frame (all RED color) as 6 rows, 44 rows, 100 rows, 200 rows and all the 720 rows. The full image has some artifacts. I am going to see if these can be resolved or not.

Debugging frame sending in Packetizer RTL

I have been debugging why Packetizer is missing some packets randomly by Chipscope pro as well as simulation. Lets pray to hear the good news soon :)

I am trying to setup trigger conditions. It seems i need to work more on that. It should be done by Monday or latest by Tuesday.

Keep looking at the blog.

Debugging Verilog RTP Packetizer frame generation

Packetizer Verilog RTL is somehow missing some packets when i . This was reported in the day before yesterday's blog. I am still trying to debug it. I might go with simulation or chipscope approach. Right now i am making changes and testing in the hardware.

I also posted the instructions on how to reproduce yesterday's blog results of sending one row of 1280 pixels by FPGA and reconstruction in Gstreamer. Please look at the README.md

Debugging will continue on the weekend or next week as i have some commitment on Friday July 18th. See you after a short break !

Gstreamer finally works !

What a day today that ended up with falling in love with Gstreamer ! What an amazing tool !

Mos of today was dominated by the following loop

 Googling, talking to Gstreamer community, going back fixing the RTL code, running hardware, look at Gstreamer output

I must appreciate the community that really helped me with Gstreamer debugging. The problem was "Specification" that i mentioned a while ago that two people can never understand exactly the same specification. This happened today when i got a chance to compare my version of RFC 4175 implementation and Gstreamer's implementation. Other than three things, the implementations were identical. It took a while to figure that out. The three things were

1) Gstreamer counts scan lines from 0 while my implementation started counting lines from 26 and up (See section 3 of the above RFC)

2) My implementation was offsetting pixels by bytes while Gstreamer was offsetting pixels by number of pixels (which is the correct way). So i corrected my implementation

3) Even if one sends one line i.e., 1280 pixels (as three packets of 424,424 and 432 pixels), that last packet of 432 pixels must have marker bit set to 1. My implementation assumed marker bit would be set to 1 for the last packet in the frame (line # 720).

Once these were resolved, FPGA was able to send one line of 1280 pixels (all RED Color i.e., FF_00_00) that was correctly captured by Gstreamer as well as Wireshark. Please take a look at the following snapshots.

Following is the command that is used to launch Gstreamer

$GST_DEBUG=rtp*:9 gst-launch-1.0 -v udpsrc uri=udp://192.168.1.1:5004  caps="

application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)RGB, depth=(string)8, width=(string)1280, height=(string)1" !  rtpvrawdepay  ! videoconvert ! videoscale ! "video/x-raw,height=720" ! autovideosink

OK. Now is the time to send full frame and capture it via Gstreamer. That is the goal for tomorrow.

Thanks for visiting my blog. Please give your precious feedback by commenting.

Debugging Gstreamer and testing packetizer

It was a hectic day trying to get RTP to work using Gstreamer. The receiver is not working yet. I am experimenting with Gstreamer. It is an excellent tool. I really liked it. After reading documentation and bouncing back and forth on #gstreamer, i tried the following and its variation

$gst-launch-1.0 -v GST_DEBUG=udpsrc:5 udpsrc uri=udp://192.168.1.1:1234 ! rtpbin ! rtpvrawdepay ! videoconvert ! autovideosink

but it does not work. It seems that "caps" is missing.It documentation is little hard to digest. I am trying to debug that.


On the other hand, Packetizer is missing some packets. For example, when i am trying to send an entire frame (720x3 packets) via RTP, it misses some packets e.g., packets with sequence number 16-28 are missing. I am trying to debug that as well.

I am hopeful i will be able to debug the Packetizer by tomorrow. I am hopeful to get to some point on Gstreamer by Thursday. If not, i will deal with image reconstruction in Pythonic way.