Need information in "how is a user module handled in async situations"

[hesspet]

Im working on a serial module for Usine

First draft for serial is ready and - let's say - it works in most situations:P. But I'm a little bit confused about a reaction/error. I'm not sure if this is a side effect or by design of usine. Maybe someone can help me out.

I'm testing with this patch:



Simple - I create a counter and send the created number to the serial interface.

This works perfect until this moment the clock is to fast (invervall <50ms). Then I loose call to the callback() method.. I traced out the numbers, usine delivers the correct numbers but the callback has gaps.

Here's the tracelog:

Code: Select all

[8638] [CallBack()()] vor do_write dataout =  => .21151
[8639] [do_write()()] .21151
[8640] [write_start()()] len stream buffer => 6
[8641] [write_start()()] iTxDataLen => 6
[8642] [CallBack()()] nach do_write dataout =  => .21151
[8643] data out[6]46 50 49 49 53 50 
[8644] data out[6]46 50 49 49 53 51 
[8645] data out[6]46 50 49 49 53 52 
[8646] data out[6]46 50 49 49 53 53 
[8647] [CallBack()()] vor do_write dataout =  => .21155
[8648] [do_write()()] .21155
[8649] [write_start()()] len stream buffer => 6
[8650] [write_start()()] iTxDataLen => 6
[8651] [CallBack()()] nach do_write dataout =  => .21155
[8652] data out[6]46 50 49 49 53 54 
[8653] data out[6]46 50 49 49 53 55 
[8654] data out[6]46 50 49 49 53 56 
[8655] data out[6]46 50 49 49 53 57 
[8656] [CallBack()()] vor do_write dataout =  => 5

You can see Usine creates 4 numbers and at the 4th number my callback is "called". This is the reason why I loose numbers. It looks like that usine does not call the callback because ????

this line: [8647] [CallBack()()] vor do_write dataout = => .21155
callback begin

and this line: [8651] [CallBack()()] nach do_write dataout = => .21155
call back end.

So when the next number is traced by Usine, no code of the user module is running. This drives me crazy. When I slow down the counter everything works fine until the point < 50ms, then for unknown reason, my callback isn't called not so often.

Maybe I must do the handling in the process() method? The serial buffer is handled in a asynchronous process, so this can't be a problem of a blocking IO.

Any ideas

This is how the callback is implemented:

Code: Select all

void CallBack (void* pModule, TMessage* Message) 
{ 
	SerialOutModule* pSerialOutModule = ((SerialOutModule*)pModule);

	try {

		switch(Message->wParam) {
		case PARAM_PORT:	
                        ...
			break;

		case PARAM_BAUDRATE:	
                        ....
			break;

		case PARAM_DATAOUT:	

			// if the data changes let's read the data
			if (Message->lParam == MSG_CHANGE) 
			{	
				std::string strDataOut = pSerialOutModule->GetEvtPChar( pSerialOutModule->m_pevtDataOut );
				pSerialOutModule->trace("CallBack()", "vor do_write dataout = ", (PCHAR)strDataOut.c_str());
				if (pSerialOutModule->m_p_SerialBasics != NULL ) {
					try {
						pSerialOutModule->m_p_SerialBasics->do_write(strDataOut);
					}
					catch(exception& e) 
					{
						pSerialOutModule->trace("CallBack()","exception=", (PCHAR)e.what());
					}
					pSerialOutModule->trace("CallBack()", "nach do_write dataout = ", (PCHAR)strDataOut.c_str());
				}
			}
                        break;
		default:
			// do nothing
			break;
		}
	} catch( exception& e) {
		if (pSerialOutModule != NULL)
			pSerialOutModule->trace("CallBack()","exception=", (PCHAR)e.what());
	}
}

Peter

Added:---------------------
I've tested the callback with this line commented:

pSerialOutModule->m_p_SerialBasics->do_write(strDataOut);

same result. If do_write is not called, no serial handling is done. Looks like a Usine behaviour. Maybe it's by design?

[hesspet]

Tomatos on my eyes

I've found the reason.

This is the difference between:

pParamInfo->CallBackType = ctNormal;

and

pParamInfo->CallBackType = ctImmediate;


pParamInfo->CallBackType = ctNormal;
Usine calls you when it have enough time. So in high pressure situations you may get an delay (like my problem )

pParamInfo->CallBackType = ctImmediate;
Usine calls you every time your value changes. Instantly. This was what I want .....

Peter, now happy!

[martignasse]

hi hesspet,

cool to see you already have some result

so, i'll try to understand how your module work :

- you have some async code (working thread ?) to write to the serial line.
- when data change on the inlet 'data out', the callback copy those data (with m_p_SerialBasics->do_write(strDataOut)) to some buffer handled by the async code who deal with the bauderate and the serial line.

and if you refresh data to often, some callback are just ignored ?


ok, first thing to check :

in the GetParamInfo(), for the ParamInfo struct of the 'data out' inlet, make sure to have a line like this :

Code: Select all

pParamInfo->CallBackType    = ctImmediate;

it will ensure that the callback will be called immediately and every time the 'data out' change
(for more info about that, search for 'ctImmediate' in the UsineDefinitions.h file)

let me know if that make it

[hesspet]

Hello martignasse!

Thanks for you reply.

Yes, was late in the night

If forgot the idea of the TFastCallBackType. Using ctNormal was the mistake, I set up this value at the beginning of the project and didn't realize that this was the critical fact. Now it works perfect. I didn't understand the callback handling of Usine at all. After a little bit sleep it's now clear.

I wrote a blog about the TFastCallBackType. Maybe this helps the next developer not to run into the pitfall.

http://journeytounknownsoundscapes.blog ... ing_9.html

Keywords: ctNormal, ctImmediate, ctAsynchronous

The communication between the background thread and the callback is fully covered by a asynchronous handlig. As I wrote, I've removed all my serial code and tested it again and the "problem" occurs again. So I was sure that it has nothing to do with the thread synchronisation. It was a classical beginners problem

[martignasse]

great, can't wait to test your module
my arduino is hot and ready

technically speaking, having three callback mode is one of the killer feature of usine, it make possible the high level of modularity with all the different kind of data we have to deal with inside a module.

ctNormal :
Usine inject events in the windows event loop at the 'refresh speed rate' chosen in the global setup.
But to not flood it, all similar/duplicated event are deleted
that's what you experienced i think, you should have a 'refresh speed rate' set to 50 ms
it's the default choice.

ctImmediate :
just call immediately the concerned callback when the event occur, thus, no event loop involved.
it's the most powerful/accurate way to have params feedback.
But the price is : all the callback code is executed in the audio thread, so memory allocation or heavy process are the devil here

ctAsynchronous :
Usine event are injected in a complete separate thread event loop
it's not fast, it's asynchronous but ideal, for example, to an open file popup or to write to a file.

[hesspet]

Hi!

You wrote:

ctAsynchronous :
Usine event are injected in a complete separate thread event loop
it's not fast, it's asynchronous but ideal, for example, to an open file popup or to write to a file.

Maybe this could also be an alternative for sending slow data, e.g. to an arduino. Depends on what should be solved with an Arduino. If the Arduino is driven with 9600 baud e.g. to send a command to switch on a led or a motor, it's so slow that this mode is ok.

I'm thinking about making 2 different modules. One with the immediate mode and one with the asynchrounous mode. This "high performance" sending of serial information is not needed for every day usages of an Arduino. But for applications like a LED VU meter, or a beat to "whatever" transmission the latency must be reduced to "near zero".

great, can't wait to test your module
my arduino is hot and ready

The raw output module works. Should I post it here as a "alpha" version? Testing serial is allways a little bit difficult, so feedback may help to make it bulletprove. I used the boost libary als underlying framework and I had a lot of problems to solve, but this may help to port the serial module to the upcomming new Usine (mac) version.

Peter

[martignasse]

I'm thinking about making 2 different modules. One with the immediate mode and one with the asynchrounous mode. This "high performance" sending of serial information is not needed for every day usages of an Arduino. But for applications like a LED VU meter, or a beat to "whatever" transmission the latency must be reduced to "near zero".

well, i don't know if two mode are really needed, depends on lots of things
KISS approach here (keep it simple and smart )
first step, make it work with immediate mode
second step, test it a lot
third step, see if it's enough or if need some modifications

in any case, if two mode are needed, the query system is better than two module, it let you ask the user for some options before params are created.

The raw output module works. Should I post it here as a "alpha" version? Testing serial is allways a little bit difficult, so feedback may help to make it bulletprove.

yep, it's a good way to have feedback and in this area, more is better.
personally, i'm quite busy and i'll can't do deep testing, but i'll try it for sure.
you could upload it in the add-ons section.

I used the boost libary als underlying framework and I had a lot of problems to solve, but this may help to port the serial module to the upcomming new Usine (mac) version.

seem's the good choice for Hollyhock and mac compatibility.
what kind of problems have you to solve with boost ?

keep up the good work

[headphoner]

Hi,


I can't help you for the code but i can test the module, i've got 3 arduinos

[hesspet]

seem's the good choice for Hollyhock and mac compatibility.
what kind of problems have you to solve with boost ?

The paradigm of the boost asio is sometimes a little bit confusing and the documentation is so fragmented. You'll find a lot of posts around, but often the info is misleading or "bad programming" .

And... I've stopped programming C++ in 1998 (without STL) and there are so many new things, I must learn to understand the ideas of boost. Now I see a little bit more how the async handling in boost works and how we can use it inside of a Usine module.

There is a lot of magic in the boost::asio lib!
BTW: boost::asio has nothing to do with the ASIO used in audio applications

Code: Select all

	SerialBasics(const char *dev_name, TUserModule* pTUserModule) : 
		m_io_service(),                          // build instance of IO_SERVICE
		m_serial_port(m_io_service, dev_name),   // initialize the COM Port
		m_work(m_io_service)                     // bind IO_SERIVCE to a worker thread pattern, so the run() method is repeated endless 
	  {
		  // must be set before any trace output! No tracing before this point!
		  m_p_user_module = pTUserModule; 

		  // start the endless running worker thread to process the async data.
		  boost::thread t(boost::bind(&boost::asio::io_service::run, &m_io_service));

		  m_tWorker = &t;

		  trace("SerialBasics","constructed");
	  };

	  void runner() {
		  trace("SerialBasics","before run");
		  m_io_service.run();
		  trace("SerialBasics","after run");
	  }

This weaves everything together. Looks pretty simple but the things happening under the hood are strange. If this initialisation is done, the motor (thread) which does a endless repeating of the run (the method which is a blocking call to a sort of a socket processing) ist started. And then - if everything is pretty initialized the sending of the serial data is simple:

Code: Select all

/*
	Tries to send all async data available in the streambuffer.
	*/
	inline void write_start(void)
	{

		// compute the data length to send in the stream
		int iTxDataLen = prepareSendBuffer();

		trace("write_start()","iTxDataLen", iTxDataLen);

		// try to send data much as possible

		boost::asio::async_write(
			m_serial_port,
			boost::asio::buffer(m_TxBuffer, iTxDataLen ),//m_buff,
			boost::bind(
			&SerialBasics::write_complete, // called when all data was send.
			this,
			boost::asio::placeholders::error
			)
			);
	}

The complete serial handling is covered in a portable lib. The horror of boost is, it's totally based on huge template structures which are (for me) impossible to debug. The debugging features in VC2010 express are limited The buffer handling is a little bit tricky, cause it's up to the hosting application to ensure that the buffer is reliable over the complete time of the async processing, needs a little bit juggling with mutex patterns.

BTW: I decided to limit the maximum size of one serial message to 1024 bytes. So the buffer handling is simple. Dynamic buffers will bring a lot of overhead and need garbage collection and and and... KISS....

Peter

[Clearscreen]

Can't say i follow this, but I wanted to say thanks for doing this hesspet, it'll breath new life into my arduino's!