...
All the queued messages and log messages going into a single log file per day.
We did this to batch file I/O writes for performance
This means that if you have say 100 channels with 100 queues that on average less that 1% of the log file relates to one channel.
We call this a sparse data structure - i.e. a lot of space in between entries
In Iguana 6 this created a performance issue when reading the queue.
To solve this problem we made use a database index implemented using a database called SQLite.
...
So one final question is how to we get guaranteed data integrity if the power is cut and yet optimize the speed of the system by batching file I/O writes for performance?
Well we do only periodically writing to each of the neuron’s individual queue/log files, but we write frequently to a journal file for the system. In the event that power is cut to the system we can load the journal file and recover data which has not yet been saved to individual neuron queue/log files.