VidSynth - synthesis out of video feed

[x.iso]

I'm happy to announce my second, but quite complex patch released in add-ons.
Take advantage of Usine's video capabilities and have yourself a synth that interprets video into live wavetable of it's oscillators. There are many way's to sculpt the generation of wavetable to your liking, so go ahead and experiment!

I've included many ways to adjust this synth for low to high CPU usage depending on demands (video matrix resolution, max voices adjustment, osc disabling, video player disabling), but I recommend using hardware acceleration option in Usine to achieve better performance.

I'll prepare fresh video demo sometime later.

[senso]

I'll try it+++

[joffo78]

It s look very nice

[x.iso]

I've made a mistake on scaling Filter frequency output from modulation matrix, if you chose it from last 'Joker' destination. It's quick to fix, but I'm trying to improve modulation matrix behavior as well so that destination faders could still be used as 'center' of modulation so I'll update it when that's done.
I'm also thinking of replacing Ladder Filter to Simple one for better performance, but I'll wait for full FM implementation in stock Oscillators.

[x.iso]

Here's updated synth:
http://www.sensomusic.org/forums/uploads.php?file=VidSynth_v1-02S.zip
couldn't update existing add-on yet, posted in another topic about it.

Most notable change is that I replaced oli_lab's Ladder Filter with Simple Filter for various reasons, it seems to run smoother that way and easier to get familiar sound. Hope to add FM modulations later.

Here's changelog:
* Fixed incorrect scaling for Filter frequency destination in Modulation Matrix (selectable through Joker combo-box)
* Fixed potential bad behaviour of Filter when using inverted envelope
* Fixed clicks on fast attack
* Fixed edit access for list of added videos/images.
* Added pitchbend and modwheel support. To use modwheel, type 'modwheel' in desired control's binding.
* Replaced Ladder Filter from oli_lab with stock Simple filter to balance performance and ease of use.
* Replaced Spray effect with Noise effect in video effects sliders.
* Two Mod destinations previously reserved for Ladder Filter FM parameters are replaced with Shaper width and shape modulations.
* Rearranged UI for new features
* V-H, RGB-HSL, PWM, LFO freq and Detune are now modulated relatively to their value with visual feedback.
* Added PWM amount for each oscillator for differentiation, also added flip switch and inversion for each OSC individually.
* Added PW switch for Wavetable speed smoother. If turned on, slider smooths overall wavetable (as it did before), but with PW off Shaper and PWM modulations are unaffected.
* Added unison slider
* Added option to loop video for synced period using scan position as start of the loop. See popup-window settings.
* Changed output OScilloscope from 3 osc's view to overall stereo sum after filter, but before envelopes for more comprehensive visual feedback.
* Added scan index splitting for spread slider.

[oli_lab]

>>Most notable change is that I replaced oli_lab's Ladder Filter with Simple Filter for various reasons, it seems to run >>smoother that way and easier to get familiar sound. Hope to add FM modulations later.

I know that the FM input is akward for some, but it is modularesque in essence !

we need audiorate option on many bulk modules !

cheers

Olivar

[SylvainT]

audiorate option is in close discussion with Senso.
I hope it comes soon, at least with HH4.
Please, make a list, to compare with mine and I'll update it.

All the best

Sylvain

[x.iso]

I know that the FM input is akward for some

I actually didn't want to change the filter precisely because of FM effects, but it's a bit quirky and performance of Simple Filter was better, also I think there was a bug that caused filter to glitch, but it might have been due to my mistakes in patching.
I just hope to use FM oscillator modulations in future.

[x.iso]

Something strange happened, can anyone confirm if latest Vidsynth version working properly and not crashing Usine?

[x.iso]

uploaded 1.03S (few tweaks added) and just deleted previous version, I guess that works too.

[x.iso]

I've recently noticed, that although this synth can use CPU a bit too much, if used in separate rack then other patches/racks won't just stack up CPU usage, seems like core distribution is making things even. Hope in HH4 I can achieve better performance though.

Here's a quick video demo that I forgot to add:
https://www.youtube.com/watch?v=C8DW8F0mqEE

[joffo78]

Hello. Thank for sharing but please help us to understant how does it works. And if th patch needs a user module pleas share it with your addon.
Tanks.

[x.iso]

There is actually info included with the patch, if you click 'i' on the patch:

VidSynth 1.02S
This synth takes advantage of Usine's feature set to interpret input video into live wavetable for it's oscillators.

That translates to 3 oscillators per voice (Red Green and Blue or Hue Saturation and Lightness) which can be turned off individually to reduce CPU usage or make less complex sound. Red/Hue oscillator additionally have detune control (which also plays role in spread control) and is cross-mixed with Green/Saturation oscilator for panoramic effect (stereOSC control). Blue oscillator doesn't have special features, can be used as sub-osc.

There's TV section where you either load video or image directly to the list or use input from elswhere (which will save some CPU). Then by selecting a point on the screen you get cross section for Vertical and Horizontal arrays of data, which can be mixed by V-H control and then further mixed between either RGB or HSL colour data interpretation.

Additionally to resulting wave, you can set 'pulsewidth' which will wrap the wave around with optional inversion and flip. Then there's also shaper tools with Shape and Amt controls to add classical shapes to resulting wavetables for more predictable and familiar sound. Also there's speed slider for wavetable change rate to smooth transformations of resulting waves.

Another factor that will affect both video and sound is two effect sliders in TV section - smooth and noise-blur. Smooth is kind of motion blur effect which will result in smear for video and smoother changes to wavetable. Noise-Blur is a combination of two effects, Noise is increased towards the middle section and then classic blur takes effect for second half of the slider, which results in rounder waves.

Notice in TV screen there's Green and Blue dots flying around, those are particles revolving around point of selection and there are settings for their behaviour. You can apply their motions to various controls via Modulation matrix along with other sources and destinations.

This synth can utilise from reasonable to high CPU% depending on settings, main factors are Max resolution slider (found at the bottom-left) and Max voices setting in popup panel (click settings wheel in header). Then you can also turn volume of each oscillator down to 0 to turn it off and use external video input as mentioned earlier.

The patch itself depends on few other add-ons from olilabpack, make sure you have it for synth to work properly.

It relies on oli_lab pack for few modules, which is also in add-ons. The problem is, oli_lab pack may not correctly download automatically from browser, seems like it got too big. So make sure to download it manually if that happens. Unfortunately, I haven't found a way to refer to add-on module in the patch with relative path, so it will inevitably ask for missing modules that aren't standard ones (same happens with many other add-on patches). For same reason I don't think adding it to my own add-on folder will help, it'll still refer to absolute path.

When that is sorted out, here's how it works:
This synth have 3 wavetable oscillators and they receive real-time generated arrays as wavetables. Arrays are generated from scanning input image or video by X and Y axis. Since image can be described as arrays of RGB or HSL values, there's fader to mix between these interpretations, then there's V-H mix for vertical and horizontal slices of images (which again produce different arrays), additionally the slices coordinates are spread between OSC's when 'spread' fader is applied (along with delaying phases for stereo effect).

Finally resulting arrays go through optional transformations: PWM parameter shifts the arrays, affected amount can be set for each OSC, along with optional flipping and mirroring of the shifted array.

Then in the Wavetable display you see Shape and Amt controls that can mix in classic oscillator shapes (when Amt is at the bottom no shaping is applied), and finally there's wavetable transition smoothing parameter (WT change speed) with separate PWM switch so it can be unaffected. That about covers specific of wave forming in this synth, mod matrix is pretty self-explanatory.

Since mod matrix can affect some parameters that have their own faders, those faders work centered values with visual indication of range of modulation applied and actual value floating in it. R and S buttons near LFO button are Retrigger and Sync. Unison is basically Chorus effect applied to OSC's independently, which can produce massive sound and I found it to be 'cheapest' way to achieve.