[ACCEPTED]-Look for fastest video encoder with least lag to stream webcam streaming to ipad-vlc

I have been asked this many times so I will 87 try and answer this a bit generically and 86 not just for mjpeg. Getting very low delays 85 in a system requires a bit of system engineering 84 effort and also understanding of the components.

Some 83 simple top level tweaks I can think of are:

Ensure 82 the codec is configured for the lowest delay. Codecs 81 will have (especially embedded system codecs) a 80 low delay configuration. Enable it. If you 79 are using H.264 it's most useful. Most 78 people don't realize that by standard requirements 77 H.264 decoders need to buffer frames before 76 displaying it. This can be upto 16 for Qcif 75 and upto 5 frames for 720p. That is a lot 74 of delay in getting the first frame out. If 73 you do not use H.264 still ensure you do 72 not have B pictures enabled. This adds delay 71 to getting the first picture out.

Since you 70 are using mjpeg, I don't think this is applicable 69 to you much.

Encoders will also have a rate 68 control delay. (Called init delay or vbv 67 buf size). Set it to the smallest value 66 that gives you acceptable quality. That 65 will also reduce the delay. Think of this 64 as the bitstream buffer between encoder 63 and decoder. If you are using x264 that 62 would be the vbv buffer size.

Some simple 61 other configurations: Use as few I pictures 60 as possible (large intra period). I pictures 59 are huge and add to the delay to send over 58 the network. This may not be very visible 57 in systems where end to end delay is in 56 the range of 1 second or more but when you 55 are designing systems that need end to end 54 delay of 100ms or less, this and several 53 other aspects come into play. Also ensure 52 you are using a low latency audio codec 51 aac-lc (and not heaac).

In your case to 50 get to lower latencies I would suggest moving 49 away from mjpeg and use at least mpeg4 without 48 B pictures (Simple profile) or best is H.264 47 baseline profile (x264 gives a zerolatency 46 option). The simple reason you will get 45 lower latency is that you will get lower 44 bitrate post encoding to send the data out 43 and you can go to full framerate. If you 42 must stick to mjpeg you have close to what 41 you can get without more advanced features 40 support from the codec and system using 39 the open source components as is.

Another 38 aspect is the transmission of the content 37 to the display unit. If you can use udp 36 it will reduce latency quite a lot compared 35 to tcp, though it can be lossy at times 34 depending on network conditions. You have 33 mentioned html5 video. I am curious as to 32 how you are doing live streaming to a html5 31 video tag.

There are other aspects that can 30 also be tweaked which I would put in the 29 advanced category and requires the system 28 engineer to try various things out

What is 27 the network buffering in the OS? The OS 26 also buffers data before sending it out 25 for performance reasons. Tweak this to get 24 a good balance between performance and speed.

Are 23 you using CR or VBR encoding? While CBR 22 is great for low jitter you can also use 21 capped vbr if the codec provides it.

Can 20 your decoder start decoding partial frames? So 19 you don't have to worry about framing the 18 data before providing it to the decoder. Just 17 keep pushing the data to the decoder as 16 soon as possible.

Can you do field encoding? Halves 15 the time from frame encoding before getting 14 the first picture out.

Can you do sliced 13 encoding with callbacks whenever a slice 12 is available to send over the network immediately?

In 11 sub 100 ms latency systems that I have worked 10 in all of the above are used. Some of the 9 features may not be available in open source 8 components but if you really need it and 7 are enthusiastic you could go ahead and 6 implement them.

EDIT: I realize you cannot 5 do a lot of the above for a ipad streaming 4 solution and there are limitations because 3 of hls also to the latency you can achieve. But 2 I hope it will prove useful in other cases 1 when you need any low latency system.

We had a similar problem, in our case it 22 was necessary to time external events and 21 sync them with the video stream. We tried 20 several solutions but the one described 19 here solved the problem and is extremely 18 low latency:

Github Link

It uses gstreamer transcode to mjpeg 17 which is then sent to a small python streaming 16 server. This has the advantage that it 15 uses the tag instead of so it can be viewed 14 by most modern browsers, including the iPhone.

As 13 you want the <video> tag, a simple 12 solution is to use http-launch. That had 11 the lowest latency of all the solutions 10 we tried so it might work for you. Be warned 9 that ogg/theora will not work on Safari 8 or IE so those wishing to target the Mac 7 or Windows will have to modify the pipe 6 to use MP4 or WebM.

Another solution that 5 looks promising, gst-streaming-server. We 4 simply couldn't find enough documentation 3 to make it worth pursuing. I'd grateful 2 if somebody could ask a stackoverflow question 1 about how it should be used!