Audio/Video mixers

• Audio/Video

• Instal·lació / Installation
  • Dependències / Dependencies
    • Mageia 7
      • urpmi git python3-virtualenv
      • urpmi libgirepository-devel gobject-introspection python3-cairo-devel
      • urpmi gstreamer1.0-nice gstreamer1.0-x264 gstreamer1.0-rtmp gstreamer1.0-srtp gstreamer1.0-vp8 lib64gstwebrtc-gir1.0
    • CentOS 7
      • yum install git ...
      • ...
  • cd ~/src
  • git clone https://github.com/bbc/brave.git
  • cd brave
  • install and activate python 3 virtualenv
  • source env/bin/activate
  • pip install --upgrade pip
  • pip install pyyml pygobject uvloop sanic psutil
• Ús / Usage
  • python brave.py
  • GUI
    • http://127.0.0.1:5000/
• REST API

  • 	types	list	create	update	delete
    general		# get all elements curl -X GET http://127.0.0.1:5000/api/all # restart curl -X POST -d '{"config":"original"}'  http://localhost:5000/api/restart
    inputs	uri image TCP Client test_video test_audio	# get all inputs curl -X GET http://localhost:5000/api/inputs	# create an input with a test video curl -X PUT -d '{"type": "test_video"}' http://localhost:5000/api/inputs	# pause input1 curl -X POST -d '{"state": "PAUSED"}' http://localhost:5000/api/inputs/1 # modify pattern in input1 curl -X POST -d '{"pattern": 18}' http://localhost:5000/api/inputs/1 # play input1 curl -X POST -d '{"state": "PLAYING"}' http://localhost:5000/api/inputs/1 # resize input3 curl -X POST -d '{"width": 100, "height": 100}' http://localhost:5000/api/inputs/3
    overlays	text clock effect	# get all overlays curl -X GET http://localhost:5000/api/overlays	# create an overlay (text) over mixer1 curl -X PUT -d '{"type": "text", "source": "mixer1"}' http://localhost:5000/api/overlays # create an overlay (clock) over input1 curl -X PUT -d '{"type": "text", "source": "input1", "text": "camera 1"}' http://localhost:5000/api/overlays # create an overlay (effect) over input2 and make it visible curl -X PUT -d '{"type": "effect", "source": "input2", "effect_name": "agingtv", "visible": true}' http://localhost:5000/api/overlays	# make overlay visible curl -X POST -d '{"visible": true}' http://localhost:5000/api/overlays/1 # make overlay invisible curl -X POST -d '{"visible": false}' http://localhost:5000/api/overlays/1
    mixers		# get all mixers curl -X GET http://localhost:5000/api/mixers		# modify pattern in mixer1 curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1 # cut (switch) to input1 curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source # overlay to input2 (input1 is not removed from mixer) curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source
    outputs	rtmp tcp file image webrtc kvs local	# get all outputs curl -X GET http://localhost:5000/api/outputs	# create a webrtc output curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs

    
    
  • Example
    • # create a webrtc output
curl -X PUT -d '{"type": "webrtc"}' http://localhost:5000/api/outputs
# modify pattern (snow) in mixer1
curl -X POST -d '{"pattern": 1}' http://localhost:5000/api/mixers/1
# create an input with a test video (colour bars)
curl -X PUT -d '{"type": "test_video", "pattern": 0}' http://localhost:5000/api/inputs
# create an input with a test video (balls)
curl -X PUT -d '{"type": "test_video", "pattern": 18}' http://localhost:5000/api/inputs
# cut (switch) to input1
curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/cut_to_source
# cut (switch) to input2
curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/cut_to_source
# create an image input
curl -X PUT -d '{"type": "image", "uri": "file:///tmp/logo.png"}' http://localhost:5000/api/inputs
# put image as a DOG
curl -X POST -d '{"uid": "input3", "zorder": 100}' http://localhost:5000/api/mixers/1/overlay_source
# select (overlay) input1
curl -X POST -d '{"uid": "input1"}' http://localhost:5000/api/mixers/1/overlay_source
# select (overlay) input2
curl -X POST -d '{"uid": "input2"}' http://localhost:5000/api/mixers/1/overlay_source
# restart curl -X POST -d '{"config":"original"}'  http://localhost:5000/api/restart
      ...
  • ...

• Snowmix - Live Video Mixer
• • Snowmix audio guide
  • Reference manuals
  • • Reference mnual for feeds available.
• Instal·lació / Installation
• • Dependencies
  • • CentOS
    • • yum install nmap-ncat bc dejavu-lgc-sans-fonts
      • all other dependencies are automatically managed by bootstrap
    • Mageia
    • • urpmi lib64SDL-devel lib64png-devel lib64cairo-devel lib64pango1.0-devel lib64tcl-devel bc
      • urpmi xterm bwidget
      • urpmi netcat-traditional
  • Get source
  • • curl -o Snowmix-0.5.1.tar.gz -L "http://downloads.sourceforge.net/project/snowmix/Snowmix-0.5.1.tar.gz?r=&use_mirror=netix"
    • tar xzf Snowmix-0.5.1.tar.gz
    • Mageia (to avoid non-closing connection for nc)
    • • sed -i 's/debian )/debian | mageia )/g' scripts/snowmix-settings
    • CentOS (not critical)
    • • sed -i 's/fedora |/fedora | centos |/g' scripts/snowmix-settings
  • Install
  • • cd Snowmix-0.5.1
    • ./bootstrap
    • or use non interactive script bootstrap_ni.sh
      • #!/usr/bin/expect
spawn ./bootstrap
expect "your password to sudo (Y/N)?"
sleep 2
send "y\r"
expect "Press return to execute bootstrapd/bootstrap-centos"
sleep 2
send "\r"
expect "update your Fedora installation (Y/N) ?"
sleep 2
send "n\r"
expect "Update and install (y=yes, n=no, q=quit) ?"
sleep 2
send "y\r"
expect "Press return to continue"
sleep 2
send "\r"
    • make
    • sudo make install
• Additional tools: not dependencies, but used by demo script
• • Gstreamer
• Usage
• • echo "export SNOWMIX=/usr/local/lib/Snowmix-0.5.1" >>~/.bashrc
  • demo
  • • snowmix_demo
  • complete minimal examples
    • using snowmix/ini/minimal-1feed.ini
      • start snowmix:
        • snowmix ini/minimal-1feed.ini
      • check status
        • echo 'system info' | nc 127.0.0.1 9999
      • show output
        • scripts/av_output2screen
      • send file to feed 1
        • ...
      • show inputs
      • ...
    • snowmix/minimal.ini
      • # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved
# verbose
require version 0.5.0
system control port 9999
system geometry 640 360 BGRA
system frame rate 25
system socket /tmp/mixer1
    • snowmix ...
  • regular usage:
  • • Snowmix Libraries
    • • Tcl interpreter
        • tcl eval <tcl command>
        • tcl exec <command macro>
        • tcl help
        • tcl reset
      • scenes.slib (part 1, part 2)
        • Create / Add
          • SceneCreate
          • SceneAddFrame
          • ...
        • Get/Set Scene Info and Scene Commands
          • SceneAddFrameType
          • SceneFrameTypesList
        • Frame commands
          • SceneSetFrameSource
          • ...
      • feeds.slib
        • FeedCreate
        • FeedInfo
        • FeedPAR
        • FeedList
        • FeedDeadImage
        • • Image format
          • • raw RGBA
              • creation
                • ImageMagick
                • frames/make_frame
    • start snowmix (example ini files at ini dir)
    • • snowmix ini/minimal-1feed.ini
      • snowmix ini/demo-scenes
      • snowmix ini/my.ini
      • • my.ini
        • • # Copyright by Peter Maersk-Moller 2012-2015 - All rights reserved
# verbose
require version 0.5.0
# include basis setup
include ini/basis.ini
DigitalClockInit
ClockInit
# include audio setup
include ini/audio
      • minimal_scenes.ini
        • ...
    • output script (using gstreamer)
    • • scripts/av_output2screen
      • Problemes / Problems
      • • no audio
        • • Solució / Solution
          • • av_output2screen
            • •           autovideosink \
          fdsrc fd=0 do-timestamp=true ! \
          $AUDIOFORMAT ! \
    • input script (using gstreamer)
    • • scripts/input2feed <video_feed_id> [<audio_feed_id>] ...
      • get gstreamer-related variables (called from input2feed)
      • • scripts/gstreamer-settings

        • 	output variables
          snowmix	SNOWMIXVIDEOSRC=shmsrc SNOWMIXVIDEOSINK=shmsink
          filters	VIDEOCONVERT=videoconvert
          caps	VIDEO=video/x-raw VIDEOBGRA=video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive VIDEOI420=video/x-raw,format=I420,pixel-aspect-ratio=1/1,interlaced-mode=progressive AUDIO=audio/x-raw AUDIOS16LE=audio/x-raw,format=S16LE,layout=interleaved

      • get snowmix-related variables (called from input2feed)
      • • scripts/snowmix-settings

        • 		input variables (optional)	output variables	example
          system		SNOWMIX_IP (default: 127.0.0.1) SNOWMIX_PORT (default: 9999)	snowmix	feed_id=1 source ./snowmix-settings; echo $geometry ... get_snowmix-settings.sh  #!/bin/bash # usage: get_snowmix-settings <feed_id> [<audio_feed_id>] feed_id=$1 audio_feed_id=$2 source ./snowmix-settings echo echo "SNOWMIX_IP: $SNOWMIX_IP" echo "SNOWMIX_PORT: $SNOWMIX_PORT" echo "  snowmix:           $snowmix" echo echo "video input (feed_id: $feed_id):" echo "  feed_control_pipe: $feed_control_pipe" echo "  feed_width:        $feed_width" echo "  feed_height:       $feed_height" echo echo "video output:" echo "  geometry:          $geometry" echo "  framerate:         $framerate" echo "  ctrsocket:         $ctrsocket" echo "  system_width:      $system_width" echo "  system_height:     $system_height" echo "  ratefraction:      $ratefraction" echo echo "audio input (audio_feed_id: $audio_feed_id):" echo "  feed_rate:         $feed_rate" echo "  feed_channels:     $feed_channels" echo echo "audio output (audio_sink_id: $audio_sink_id)" echo "  rate:              $rate" echo "  channels:          $channels" echo "  AUDIOFORMAT:       $AUDIOFORMAT" exit 0
          video	input	feed_id	feed_control_pipe feed_width feed_height
          	output		geometry framerate ctrsocket system_width system_height ratefraction
          audio	input	audio_feed_id	feed_rate feed_channels
          	output	audio_sink_id (default: 1)	channels rate AUDIOFORMAT

    • ls -l /dev/shm/shmpipe*
    • output
    • • scene #1
      • • frame #1.1
        • frame #1.2
        • ...
      • scene #2
      • • frame #2.1
        • ...
      • ...
  • control using GUI (0.5.1)

  • • tcl/		included files (when modified, you must run: sudo make install)	uses: echo <command> | nc <server> <port>, where command is:
      snowscene.tcl	control and mix video scenes	cs_feed_commands.inc scenes.inc
      snowaudio.tcl	control and mix audio	graph.inc audio.inc	audio mixer status
      snowoutput.tcl	monitor the output statistics: Top graphic: Bottom graphic: Right graphic: Legend: "In Use" white "Min" red "Avg" green "Max" yellow "Last" blue "Duration" cyan	graph.inc output.inc	tcl eval snowmix message [snowmix info system status format] tcl eval snowmix message [snowmix info system status] system output status
      snowrelay.tcl		graph.inc cs_feed_commands.inc relay.inc relay_analyzer.inc
      snowcommentator.tcl		commentator.inc
      snowcub.tcl

    • to control/monitor a remote snowmix server:
      • on remote server, allow your local address (/32) or subnet:
        • snowmix.ini
          • system host allow 127.0.0.1 192.168.1.0/24
      • when calling tcl locally
        • add server:
          • <ip_address_of_instance_running_snowmix>
  • control using netcat / nc

    • group	command				response
      system	system ...				STATS: ...
      video	feed ...
      audio	audio	feed	...
      		mixer	...
      		sink	...
      TCL	tcl	reset
      		help			MSG: ...
      		exec
      		eval	snowmix message <text>
      			snowmix parse <snowmix command>
      			snowmix parse silent <snowmix command>
      			snowmix parses <snowmix command>
      			snowmix info	audio (feed | sink) (info | status | extended | syntax) (format | ids | maxid | nextavail | <id_list>)
      				audio mixer (info | status | extended | source info | source status | source extended | syntax) (format | ids | maxid | nextavail | <id_list>)
      				command ( names | list | at | syntax ) [ format | <name> ]
      				feed (geometry | status | extended | state | syntax) (format | ids | maxid | nextavail | <id_list>)
      				image (load | place | move | extended | syntax) (format | ids | maxid | nextavail | <id_list>)
      				text (string | font | place | move | backgr | linpat | syntax) (format | ids | maxid | nextavail | <id_list>)
      				(vfeed | virtual feed) (place | move | extended | syntax) (format | ids | maxid | nextavail | <id_list>)
      				shape (info | list | place | move) ( format | ids | maxid | nextavail | <id_list> )
      				system (info | status | maxplaces | overlay | syntax) [ format ]
      				shape ( syntax | ( ( info | list | place | move ) ( format | ids | maxid | nextavail | <id_list>) )

    • bash
      • echo '<command>' | nc 127.0.0.1 9999
    • Python
      • dependencies
        • pip install nclib
      • snowmix_info.py
        • import nclib
# python 3
nc = nclib.Netcat(('127.0.0.1', 2000))
nc.recv_until(b'Snowmix version 0.5.1.\n')
command = 'system info'
nc.send_line(command.encode())
response = nc.recv_until(b'STAT: \n')
print(response.decode())
command = 'system output info'
nc.send_line(command.encode())
response = nc.recv_until(b'STAT: \n')
print(response.decode())
command = 'audio mixer source'
nc.send_line(command.encode())
response = nc.recv_until(b'STAT: \n')
print(response.decode())
command = 'audio feed status'
nc.send_line(command.encode())
response = nc.recv_until(b'STAT: \n')
print(response.decode())
nc.close()
  • system commands (ini file)

  • • 	command / line in ini file	echo 'system info' | nc [-q 1] 127.0.0.1 9999	echo 'system output info' | nc 127.0.0.1 9999	variable returned by scripts/snowmix-settings
      		Snowmix version 0.5.1. STAT:  System info STAT:  Snowmix version      : 0.5.1 STAT:  System name          : localhost.localdomain STAT:  Ini file             : .../minimal.ini STAT:  Block size mmap      : 2359296 (=1024*576*4) STAT:  Frame sequence no.   : 0 STAT:  Lagged frames        : 0 STAT:  Open Control conns.  : 2 STAT:  Ctr verbose          : no STAT:  Ctr broadcast        : no STAT:  Output ctr fd        : master 6 client -1 STAT:  Output settings      : mode 0 delay 1 STAT:  Output frames inuse  : 0 of 20 STAT:  Output shm name      : /shmpipe.25528.    0 STAT:  Video feeds          : loaded STAT:  Virtual video feeds  : no STAT:  Video text           : no STAT:  Video image          : no STAT:  Video shapes         : no STAT:  Command interface    : no STAT:  Audio feeds          : no STAT:  Audio mixers         : no STAT:  Audio sinks          : no STAT:  TCL interpreter      : no STAT:  Stack                : 0 STAT:	Snowmix version 0.5.1. STAT: system output info STAT:   Verbose           = 1 STAT:   Socket master     = 6 STAT:   Socket client     = -1 STAT:   Memory handle     = /shmpipe.25528.    0 STAT:   Memory size       = 47185920 (20 blocks, block size 2359296) STAT:   Output mode       = 0 (standard, no thread) STAT:   Freeze            = no (0 frames, 0.00 secs) STAT:   Delay             = 1 frames (40.0 ms) STAT:
      minimal	system control port 9999	STAT:  Control port number  : 9999
      	system host allow 127.0.0.1 192.168.1.0/24	STAT:  Host allow           : 127.0.0.1/32 192.168.1.0/24
      	system geometry 1024 576 BGRA	STAT:  System geometry      : 1024x576 pixels STAT:  Pixel format         : BGRA STAT:  Bytes per pixel      : 4		$system_width $system_height
      	system frame rate 25	STAT:  Frame rate           : 25.000		$ratefraction
      	system socket /tmp/mixer1	STAT:  Output ctr sock name : /tmp/mixer1	STAT:   Socket name       = /tmp/mixer1	$ctrsocket
      	stack ...

      NOTE: deprecated; only for quick setups.
  • video commands

  • • 	command (echo '<command>' | nc 127.0.0.1 9999) / line in ini file	echo 'system info' | nc 127.0.0.1 9999	scripts/snowmix-settings
      creation	feed add [<feed no> <feed name>]		input: $feed_id
      setup	feed socket <feed no> <file name> feed geometry <feed no> <width> <height> feed fast overlay <feed id> <col> <row> <feed col> <feed row> <cut cols> <cut rows> [ <scale 1> <scale 2> [ <par_w> <par_h> [ center ]]] feed filter [ <feed no> (fast | good | best | nearest | bilinear | gaussian)] feed idle <feed no> <timeout in frames> <idle image file> feed name <feed no> <feed name> feed overlay (<id> | <id>..<id> | all | end | <id>..end) [ (<id> | <id>..<id> | all | end | <id>..end) ] ... feed par <feed no> <scale_1> <scale_2> feed (live | recorded) <feed no> feed cutout <feed no> <start col> <start row> <columns> <rows> feed shift <feed no> <column> <row> feed scale <feed no> <scale_1> <scale_2>		output: $feed_control_pipe $feed_width $feed_height
      info	feed help feed buffers DBG: Feed 0 Internal DBG: Feed 1 Feed #1 ... feed info STAT: feed id : state islive oneshot geometry cutstart cutsize offset fifo good missed dropped <name> STAT: feed 0 : STALLED recorded continuously 320x180 0,0 320x180 0,0 0:0 0 266393 0 <Internal> STAT: feed 1 : PENDING live continuously 320x180 0,0 320x180 0,0 0:10 0 266604 0 <Feed #1> ... feed list [verbose] STAT: Feed ID 0  Name: Internal STAT: Feed ID 1  Name: Feed #1 ...

      NOTE: deprecated; only for quick setups.
  • audio commands

  • • 	Audio Feed			Audio Mixer	Audio Sink
      	command (echo '<command>' | nc 127.0.0.1 9999) / line in ini file	echo 'system info' | nc 127.0.0.1 9999	scripts/snowmix-settings	command / line in ini file	command / line in ini file	echo 'system info' | nc 127.0.0.1 9999	scripts/snowmix-settings
      creation	audio feed add [<feed id> [<feed name>]]	STAT: audio feed ...	input: $audio_feed_id	audio mixer add [<mixer id> [<mixer name>]]	audio sink add [<sink id> [<sink name>]]	STAT: audio sink ...	input: $audio_sink_id (default: 1)
      Formats Sample Rate Channels	audio feed format [<feed id> (8 | 16 | 24 | 32 | 64) (signed | unsigned | float)] audio feed rate [<feed id> <rate>] audio feed channels [<feed id> <channels>]		output: $feed_rate $feed_channels	Info: audio mixer rate STAT: audio mixer 1 rate 48000 Hz audio mixer channels STAT: audio mixer 1 channels 2 audio mixer source STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal ... Set: audio mixer rate [<mixer id> <rate>] audio mixer channels [<mixer id> <channels>] audio mixer source [map <mixer id> <source id> <map 1> <map 2> ... <map n>	audio sink format [<sink id> (8 | 16 | 24 | 32 | 64) (signed | unsigned | float)] audio sink rate [<sink id> <rate>] audio sink channels [<sink id> <channels>] audio sink source mixer <...> <...> audio sink file <...> <...>		output: $rate $channels
      Starting	audio feed ctr isaudio <feed id>			audio mixer start [[soft ]<mixer id>]	audio sink start [<sink id>] audio sink ctr isaudio <sink id>
      Muting Volume	audio feed mute [(on | off) <feed id>] audio feed volume [<feed id> <volume 0> ... <volume n>]			Info: audio mixer mute STAT: audio mixer  1 volume 1.000,1.000 STAT: - source audio feed   1 volume 1.000,1.000 muted ... audio mixer volume (same as mute) STAT: audio mixer  1 volume 1.000,1.000 STAT: - source audio feed   1 volume 1.000,1.000 muted ... audio mixer source STAT: audio mixer 1 sourced by audio feed source id 1 av. samp. 0 = 0 ms, min/max 70,280 ms, volume 1.000,1.000 muted, normal ... Set: audio mixer mute [(on | off) <feed id>] audio mixer source [mute (on|off) <mixer id> <source no>] audio mixer volume [<mixer id> <volume 0> ... <volume n>] audio mixer source [volume <mixer id> <source no> <volume 0>... <volume n>]	Info: audio sink mute STAT: audio sink  0 volume 1.000,1.000 muted STAT: audio sink  1 volume 1.000,1.000 audio sink volume Set: audio sink mute [(on | off) <feed id>] audio sink volume [<sink id> <volume 0> ... <volume n>]
      Delay	audio feed delay audio feed delay <feed_id> <ms>			audio mixer source mindelay <mixer_id> <source_no> <ms>  audio mixer source maxdelay <mixer_id> <source_no> <ms>
      Silence	audio feed add silence <feed id> <ms>			audio mixer add silence <mixer id> <ms> audio mixer source add silence <mixer id> <source no> <ms>	audio sink add silence <sink id> <ms>
      Drop	audio feed drop <feed id> <ms>			audio mixer drop <mixer id> <ms> audio mixer drop <mixer id> <source no> <ms>	audio sink drop <sink id> <ms>
      Info and status	audio feed info STAT:  audio feed id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT:  - audio feed 1 : READY, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 1 ... audio feed status STAT: feed_id : state samples samplespersecond avg_samplespersecond silence dropped clipped delay rms STAT: audio feed 1 : READY 0 0 0 0 0 0 0 0.0,0.0 ... audio feed verbose [<level>] audio feed help			audio mixer info STAT:  audio mixer id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT:  - audio mixer 1 : RUNNING, 48000, 2, 4, signed, 1.000,1.000, unmuted, 16384, 0, 2 audio mixer status STAT: mixer_id : state samples sps avg_sps silence dropped clipped delay rms STAT: audio mixer 1 : RUNNING 1057478812 99972 96020 0 0 0 0,0 0.0,0.0 STAT: - source audio feed   1 : READY 0 1 0 0 0 0 0 0.0,0.0 ... audio mixer verbose [<level>] audio mixer help	audio sink info STAT:  audio sink id : state, rate, channels, bytespersample, signess, volume, mute, buffersize, delay, queues STAT:  - audio sink 0 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, muted, 8192, 0, 0 STAT:  - audio sink 1 : RUNNING, 48000, 2, 2, signed, 1.000,1.000, unmuted, 8192, 0, 0 audio sink status STAT: sink_id : state samples samplespersecond avg_samplespersecond silence dropped clipped queue rms STAT: audio sink 0 : RUNNING 0 0 0 0 1068699328 0 0 0.0,0.0 STAT: audio sink 1 : RUNNING 1067938584 95071 96005 0 0 0 0,0 0.0,0.0 audio sink verbose [<level>] audio sink help
      Pausing				audio mixer source [pause <mixer id> <source no> <frames>] audio mixer source [rmsthreshold <mixer id> <source no> <level>]

  • Sincronització àudio/vídeo / Audio/video synchronisation
    
    • Audio
    • Understanding Audio and Video Sync for Snowmix
      • Manual AV Sync
        • audio is ahead of video
          • determine optimal delay (temporarily)
            • add delay/silence several times (+10/20) until audio is in sync -> t1 (e.g. t1=60), t_audio_is_in_sync
            • add delay/silence several times (+10/20) until video is ahead of audio / audio is behind of video -> t2 (e.g. t2=170), audio_is_behind
            • optimal_delay = (2*t1 + t2) / 2 (e.g. optimal_delay=(2*60 + 170)/2=145)
              optimal_delay = (2*t_audio_is_in_sync + t_audio_is_behind) / 2
          • set optimal delay ()
            • audio feed delay 1 <optimal_delay>
            • audio feed delay 1 145
              audio mixer source mindelay 1 2 145
              audio mixer source maxdelay 1 2 325
    • Output buffer problem...
      • audio is ahead of video
        • audio feed delay 1 <t_mean>
        • determine optimal delay with Snowcub (temporarily):
          • add delay several times (+50, audio feed add silence 1 50) until audio is behind of video -> t_audio_is_behind
          • remove delay several times (-30, audio feed ...) until audio is ahead of video -> t_audio_is_ahead
          • optimal_delay = (t_audio_is_ahead + t_audio_is_behind) / 2
        • set optimal delay ()
          • audio feed delay 1 <optimal_delay>
            ...
      • audio is behind the video
        • system output delay ...
  • AV input/output with Gstreamer (shm for video, fd for audio)
  • • Input video to Snowmix
    • Understanding Audio and Video Sync for Snowmix
    • shm size
      • Pipeline for blackmagic card
        • 22 frames
        • echo 'feed info' | nc 127.0.0.1 9999
          • should be: 1:10 - 4:10
        • ... videoconvert ! identity silent=false ! queue ! videorate ...

    • 	input (to first feed)	output
      audio	( echo 'audio feed ctr isaudio 1'; gst-launch-1.0 -v audiotestsrc is-live=true !\ 'audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2' !\ fdsink fd=3 sync=true 3>&1 1>&2 ) | nc 127.0.0.1 9999 audio_feed_id=1 source ./snowmix-settings # audio ( echo "audio feed ctr isaudio $audio_feed_id";   gst-launch-1.0 -v audiotestsrc is-live=true ! $AUDIOFORMAT ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT	(   echo 'audio sink ctr isaudio 1'; cat >/dev/null ) | nc 127.0.0.1 9999 \ (   head -1   gst-launch-1.0 -v fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=48000 num-channels=2 !\     'audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2' !\     audioconvert !\     autoaudiosink )
      video	gst-launch-1.0 -q filesrc location=toto.mp4 ! decodebin name=decoder ! videoconvert ! videoscale ! videoconvert ! 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive, width=704, height=576' ! \ shmsink socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc -l` wait-for-connection=1 sync=true feed_id=1 source ./snowmix-settings # video VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" gst-launch-1.0 -v videotestsrc pattern=ball ! $VIDEOCAPS ! videoconvert ! videoscale ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true	X11 output gst-launch-1.0 -q shmsrcsocket-path=/tmp/mixer1 do-timestamp=true is-live=true !\ 'video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive,width=(int)1024,height=(int)576,framerate=(fraction)25/1' !\ videoconvert ! queue ! autovideosink H.264 + RTP multicast gst-launch-1.0 -v shmsrc socket-path=/tmp/mixer1 do-timestamp=true is-live=true ! video/x-raw,width=1280,height=720,framerate='25/1',format=BGRA ! videoconvert ! x264enc ! rtph264pay config-interval=10 pt=96 ! udpsink host=224.0.0.0 port=5008
      audio + video	test2feed.sh ( echo 'audio feed ctr isaudio 1';   gst-launch-1.0 -v \       videotestsrc pattern=ball ! 'video/x-raw,framerate=25/1,width=704,height=576,format=BGRA' ! videoconvert ! videoscale ! shmsink socket-path=/tmp/feed1-control-pipe shm-size=`echo 704*576*4*22 | bc` wait-for-connection=1 sync=true \       audiotestsrc wave=5 is-live=true ! 'audio/x-raw, format=S16LE, layout=interleaved, rate=48000, channels=2' ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc  127.0.0.1 9999 feed_id=1 audio_feed_id=1 source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio from test ( echo "audio feed ctr isaudio $audio_feed_id";   gst-launch-1.0 -q \       videotestsrc pattern=ball ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \       audiotestsrc is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT feed_id=1 audio_feed_id=1 source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA" AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels" # video and audio from sdp ( echo "audio feed ctr isaudio $audio_feed_id";   gst-launch-1.0 -q \       filesrc location=/tmp/toto.sdp ! sdpdemux name=demux \       demux. ! queue ! "application/x-rtp, media=(string)video" ! decodebin ! videoconvert ! videoscale ! videorate ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \       demux. ! queue ! "application/x-rtp, media=(string)audio" ! decodebin ! audioconvert ! audioresample ! $AUDIOCAPS ! fdsink fd=3 sync=true \       3>&1 1>&2 ) | nc $SNOWMIX_IP $SNOWMIX_PORT	default video + audio output source $SNOWMIX/scripts/snowmix-settings VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1" AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels" ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \ ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \ ( head -1     $gstlaunch -q \     shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! autovideosink \     fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=${rate} num-channels=${channels} ! $AUDIOCAPS ! queue !  audioconvert ! audioresample ! queue ! autoaudiosink \ ) H.264 + opus to RTP source $SNOWMIX/scripts/snowmix-settings # video output VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$system_width,height=$system_height,format=BGRA,pixel-aspect-ratio=1/1" # video coder gop_size=25 video_profile=baseline video_bitrate=1024000 # video rtp config_interval=-1 rtp_video_payload_type=96 # audio output AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$rate,channels=$channels" echo "AUDIOCAPS: $AUDIOCAPS" # audio coder audio_bitrate=200000 # audio rtp rtp_audio_payload_type=$(( rtp_video_payload_type + 1 )) #audio_media_subtype="mpeg4-generic" audio_media_subtype=opus # sdp cat >$sdp_path <<EOF v=0 c=IN IP4 $multicast_address m=video $video_rtp_port RTP/AVP $rtp_video_payload_type a=rtpmap:$rtp_video_payload_type H264/90000 m=audio $audio_rtp_port RTP/AVP $rtp_audio_payload_type a=rtpmap:$rtp_audio_payload_type ${audio_media_subtype}/${rate}/${channels} EOF ( echo "audio sink ctr isaudio $audio_sink_id"; sleep 10000000 \ ) | nc $SNOWMIX_IP $SNOWMIX_PORT | \ ( head -1     $gstlaunch -q \     shmsrc socket-path=$ctrsocket do-timestamp=true is-live=true ! $VIDEOCAPS ! queue ! videoscale ! videoconvert ! x264enc bitrate=$video_bitrate key-int-max=$gop_size ! video/x-h264,profile=${video_profile} ! rtph264pay config-interval=$config_interval pt=$rtp_video_payload_type ! udpsink host=$multicast_address port=$video_rtp_port \     fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=${rate} num-channels=${channels} ! $AUDIOCAPS ! queue ! audioconvert ! audioresample ! queue ! opusenc bitrate=$audio_bitrate ! rtpopuspay pt=$rtp_audio_payload_type ! udpsink host=$multicast_address port=$audio_rtp_port \ )

    • test2feed.sh
    • • #!/bin/bash
MIN_ARGS=2
MAX_ARGS=6
if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
then
    cat <<EOF
Usage: `basename $0` [options] feed_id audio_feed_id
Options:
-p, --pattern video_pattern       video generated pattern (1) (default: snow)
-w, --wave audio_wave             audio generated wave (2) (default: 6 (pink noise))
(1) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videotestsrc.html#GstVideoTestSrcPattern
(2) https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html#GstAudioTestSrcWave
EOF
    exit 1
fi
# defaults
# video
pattern=snow
# audio
wave=6
# options
eval set -- $(getopt -o p:w: --long pattern:,wave: -n $0 -- "$@")
while true
do
  case "$1" in
    -p | --pattern ) pattern="$2"; shift 2;;
    -w | --wave ) wave="$2"; shift 2;;
    -- ) shift; break ;;
    * ) break ;;
  esac
done
# parameters
feed_id=$1
audio_feed_id=$2
if [ "X$SNOWMIX" = X ] ; then
  echo "You need to set the SNOWMIX environment variable"
  exit 1
fi
# get snowmix settings
source $SNOWMIX/scripts/snowmix-settings
echo
echo "SNOWMIX_IP: $SNOWMIX_IP"
echo "SNOWMIX_PORT: $SNOWMIX_PORT"
echo "  snowmix:           $snowmix"
echo
echo "video input (feed_id: $feed_id):"
echo "  feed_control_pipe: $feed_control_pipe"
echo "  feed_width:        $feed_width"
echo "  feed_height:       $feed_height"
echo
echo "video output:"
echo "  geometry:          $geometry"
echo "  framerate:         $framerate"
echo "  ctrsocket:         $ctrsocket"
echo "  system_width:      $system_width"
echo "  system_height:     $system_height"
echo "  ratefraction:      $ratefraction"
echo
echo "audio input (audio_feed_id: $audio_feed_id):"
echo "  feed_rate:         $feed_rate"
echo "  feed_channels:     $feed_channels"
echo
echo "audio output (audio_sink_id: $audio_sink_id)"
echo "  rate:              $rate"
echo "  channels:          $channels"
echo "  AUDIOFORMAT:       $AUDIOFORMAT"
# video
VIDEOCAPS="video/x-raw,framerate=$ratefraction,width=$feed_width,height=$feed_height,format=BGRA"
# audio
AUDIOCAPS="audio/x-raw,format=S16LE,layout=interleaved,rate=$feed_rate,channels=$feed_channels"
# video and audio
rm -f $feed_control_pipe
(
  echo "audio feed ctr isaudio $audio_feed_id";
  gst-launch-1.0 -q \
      videotestsrc pattern=$pattern ! $VIDEOCAPS ! shmsink socket-path=$feed_control_pipe shm-size=`echo $feed_width*$feed_height*4*22 | bc` wait-for-connection=1 sync=true \
      audiotestsrc wave=$wave is-live=true ! $AUDIOCAPS ! fdsink fd=3 sync=true 3>&1 1>&2
) | nc $SNOWMIX_IP $SNOWMIX_PORT
exit 0
• Problemes / Problems
• • DTS discontinuities at the output when loading images
  • shmsink
    • shmsink changes from 1.0.10 to 1.1.4
      • experiments with shm-size
  • a/v out of sync at ouptut
    • intersink/intersrc - A/V delay #56
  • distorted audio when using av output to rtp
    • audio is from time to time for around 2-3min a bit "scrambled"
      • audio_mixer.cpp
    • Solution
      • tolerance in audiorate is too low
  • no audio when using av output
    • Frame xxx - audio sink 1 would block
      • EWOULDBLOCK
      • Error "audio sink 1 would block" (FPM)
        • Wrong dts values for fdsrc before rtpbin (FPM) (gstreamer-devel issues)
        • Wrong dts values for fdsrc before rtpbin (FPM) (Gstreamer issues in GitLab)
        • Solution
          • use rawaudioparse, as suggested by Müller in Wrong dts values for fdsrc before rtpbin, and do not use do-timestamp=true nor audiorate
        • av_output2rtp.sh
          • ... fdsrc fd=0 ! rawaudioparse pcm-format=s16le sample-rate=48000 num-channels=2 ! \
audio/x-raw,format=S16LE,layout=interleaved,rate=48000,channels=2 ! queue ! audioconvert ! audioresample ! queue ! \
opusenc  ...
    • Audio distortion / artifacts when streaming to YouTube
    • av_output2screen is stalling.
  • av output to rtp with gstreamer using rtpbin, to introduce rtcp, does not work (no audio is introduced in rtp):
    • Frame xxx - audio sink 1 would block
  • CVideoOutput out of shm buffers. Setting m_got_no_free_buffers flag.
  • • Frame 13511 : CVideoOutput out of chm buffers. Setting m_got_no_free_buffers flag.
      • monitor on
      • gst-launch
        • queue leaky=2

• Instal·lació / Installation
  • Dependencies
    • Mageia
      • urpmi...
  • cd ~/src
  • git clone https://github.com/voc/voctomix.git
  • cd voctomix
  • mv .python-version .python-version.bo
  • virtualenv env
  • source env/bin/activate
  • pip install pygobject sdnotify scipy
• Ús / Usage
  • cd ~/src/voctomix/
  • source env/bin/activate
  • ./voctocore/voctocore.py -vv
  • ./voctogui/voctogui.py -vv
• API
  • ...
• Problemes / Problems
  • A/V sync
    • GStreamer Inter*-Elements introduce a/v-synchronity by retimestamping Buffers #58
      • solved in voctomix2

• Broadcasting WebRTC Over Low Latency Dash

http://www.francescpinyol.cat/audio_video_mixers.html
Primera versió: / First version: 4.V.2020
Darrera modificació: 16 de desembre de 2020 / Last update: 16th December 2020