vorbis.Readsamples question

Mar 6, 2013 at 5:27 AM
Edited Mar 6, 2013 at 6:30 AM
Hi,
Thanks for the great codebase decoding ogg. Badly needed!

I'm testing ogg decoding with a http://unity3d.com/ project.
Your test app test method TestRaw works fine in Unity, decoding and writing.

I have a couple of questions though.

Issue1:
Here's my problem.
As Unity does not include any support for run time loading and playback of audio NVorbis comes in very handy. The way to playback is to feed Unity3D the decoded float array via this method http://docs.unity3d.com/Documentation/ScriptReference/MonoBehaviour.OnAudioFilterRead.html which is called around every 20ms or so with a buffer size of 2048.

I tried using the buffer size in the TestRaw example (var buf = new float[vorbis.SampleRate / 10 * vorbis.Channels];) but what happens is when I call out for a vorbis.readsamples from the OnAudioFilterRead when the current buffer is empty (via a index read in 2048 chunks) it only returns a couple of hundred samples at most.

Yet it works decoding the whole file and writing out to wav as in the TestRaw example. I tested this with a simple debug as it does each buffer read and also the final output wave file is correct and complete as the original ogg.

I tried doing a look ahead read into a second buffer and then swapping them but it also only returns a small amount of samples.

The only workaround I have at the moment is to grab the whole file into a huge buffer before flagging the OnAudioFilterRead into playback mode and play it back via indexed read chunks.
That works but my concern is the memory load may be large for mobiles when you have a 4 minute music track.

Any clues as to why the vorbis.readsamples is under reading when being called from a 20ms thread? Is it due to the Ogg dcecoding taking too much time for the slice available?

Issue2:
Is this mono compatible? Unity is mono based. When I include a dll compiled in VS express 2012 with ,net 3.5 (unity level) and include it in the project unity accepts it ok. If I drop the source code into Unity instead of the precompiled dll I get the following error: Internal compiler error. See the console log for more information. output was:
Unhandled Exception: Mono.CSharp.InternalErrorException: VerifyArgumentsCompat didn't find any problem with rejected candidate Void Return[T[]](T[][] ByRef)
at Mono.CSharp.MethodGroupExpr.OverloadResolve (Mono.CSharp.ResolveContext ec, Mono.CSharp.Arguments& Arguments, Boolean may_fail, Location loc) [0x00000] in <filename unknown>:0
at Mono.CSharp.Invocation.DoResolveOverload (Mono.CSharp.ResolveContext ec) [0x00000] in <filename unknown>:0
at Mono.CSharp.Invocation.DoResolve (Mono.CSharp.ResolveContext ec) [0x00000] in <filename unknown>:0

Cheers,
paul
Coordinator
Mar 6, 2013 at 12:25 PM
Paul,

Issue 1: You probably have found a bug, but before I say for sure can you share a snippet of the offending code? I need to see exactly how you are using NVorbis before I can comment further.

Issue 2: I'm not a Mono / Unity developer, so I have no idea what the error is. Looks like it is choking on the ACache code (which is admittedly a bit complicated for the compiler). Is there a reason you are not using the precompiled binary?
Coordinator
Mar 6, 2013 at 12:31 PM
A further thought on #1: If Unity has a "playback" function (instead of "filter"), you could place your music decode there and use OnAudioFilterRead to mix sfx in... That might improve your decoding scenario.
Mar 6, 2013 at 12:40 PM
Edited Mar 6, 2013 at 12:51 PM
ioctlLR wrote:
Paul,

Issue 1: You probably have found a bug, but before I say for sure can you share a snippet of the offending code? I need to see exactly how you are using NVorbis before I can comment further.

Issue 2: I'm not a Mono / Unity developer, so I have no idea what the error is. Looks like it is choking on the ACache code (which is admittedly a bit complicated for the compiler). Is there a reason you are not using the precompiled binary?
Hi,
re: Issue2
Is there a reason you are not using the precompiled binary?

I am. The binary works fine on PC, Mac, android but on iOS it crashed due to above aot error which is also related to the initial error I posed from Unity editor. Sometimes you can get rid of the AOT error by including the source instead of the binary I've found, but in this case the templated class is choking it I think.

I think it's related to mono AOT on iOS. It works fine on PC/OSX/ and Android, but Unity iOS has AOT restrictions which are choking on the ACache templates.

ref: http://docs.unity3d.com/Documentation/Manual/TroubleShooting.html#iPhoneTroubleShooting (scroll to iOS)
"The game crashes with the error message "ExecutionEngineException: Attempting to JIT compile method 'SometType`1<SomeValueType>:.ctor ()' while running with --aot-only."
The Mono .NET implementation for iOS is based on AOT (ahead of time compilation to native code) technology, which has its limitations. It compiles only those generic type methods (where a value type is used as a generic parameter) which are explicitly used by other code. When such methods are used only via reflection or from native code (ie, the serialization system) then they get skipped during AOT compilation. The AOT compiler can be hinted to include code by adding a dummy method somewhere in the script code. This can refer to the missing methods and so get them compiled ahead of time.
void _unusedMethod()
{
var tmp = new SomeType<SomeValueType>();
}
"

Unfortunately it looks like I can't just use some dummy methods in this case, and maybe I need to rewrite ACache to be non templated methods?



or?
Mar 6, 2013 at 12:47 PM
ioctlLR wrote:
A further thought on #1: If Unity has a "playback" function (instead of "filter"), you could place your music decode there and use OnAudioFilterRead to mix sfx in... That might improve your decoding scenario.
Unity has playback, but it's only for audio assets that have been pre-processed and built into the deliverable, or added later via asset downloading.
My use case is real time mic recording, save to wave, then convert to ogg, and later decode and playback.
In this case the only way to playback a procedural audio stream is feed the floats direct to the onAudioFilterRead.

My code is pretty simple:
public void StartStreaming ()
{
    //Debug.Log (this.ToString () + " Start Streaming");
    oggFilepath = Path.Combine (Application.persistentDataPath, fileName + ".ogg");
    Debug.Log (this.ToString () + " Start Streaming " + oggFilepath);
    StartCoroutine (OpenOggStream (oggFilepath));
}

IEnumerator OpenOggStream (string path)
{
    using (vorbis = new NVorbis.VorbisReader(oggFilepath)) {
        //int size = 2048 * 200;// (int)Math.Round(2048 * 44.1 * vorbis.TotalTime.TotalSeconds, 0); 
        int size = (int)Math.Round (2048 * 44.1 * vorbis.TotalTime.TotalSeconds, 0); 
        //Vorbis File totalTime: 11.703 calculated size: 517703
        Debug.Log ("Vorbis File totalTime: " + vorbis.TotalTime.TotalSeconds.ToString () + " calculated size: " + size.ToString ());
        buf = new float[size];
        //buf2 = new float[size];           
        count = vorbis.ReadSamples (buf, 0, buf.Length);
        index = 0;  
        lastBuffer = false;
        fillBuf2 = false;
        buf2full = false;
        streamOutput = true;
        yield return true;
    }       
}
then :
void OnAudioFilterRead (float[] data, int channels)
{       if (streamOutput) {
        //count = vorbis.ReadSamples(buf, 0, buf.Length);
        Debug.Log ("OnAudioFilterRead Stream count: " + count.ToString () + " index: " + index.ToString () + " buf.Length: " + buf.Length.ToString () + " data.Length: " + data.Length.ToString ());
        //Need to implement very large buffers and some sort of round robin look ahead reading 
        if (count > 0) {
            //OK
            for (int i = 0; i < data.Length; i++) {
                data [i] = buf [index + i];             
            }
            index += data.Length;
            if (index > count - 1) {
                count = -1;
                index = 0;  
                streamOutput = false;
                Debug.Log ("OnAudioFilterRead Buffer finished");
            }

        } else {
            streamOutput = false;
            Debug.Log ("Stream end");
        }
    }
}

This works, but needs huge initial buffer depending how long the initial file is - at most it will be a few minutes though.
Coordinator
Mar 6, 2013 at 12:47 PM
Strange... NVorbis explicitly uses ACache (i.e., no reflection or native code). You might try putting dummy methods in ACache for byte, float, and int (create a buffer, then return it).

I still think you should use the precompiled binary (VS2012) and pass it through Unity to get the native binary for iOS... Unless that isn't possible or gets the same error.
Mar 6, 2013 at 12:53 PM
ioctlLR wrote:
Strange... NVorbis explicitly uses ACache (i.e., no reflection or native code). You might try putting dummy methods in ACache for byte, float, and int (create a buffer, then return it).

I still think you should use the precompiled binary (VS2012) and pass it through Unity to get the native binary for iOS... Unless that isn't possible or gets the same error.
I have, then on iOS it gets the ""The game crashes with the error message "ExecutionEngineException: Attempting to JIT compile method 'SometType`1<SomeValueType>:.ctor ()' while running with --aot-only."" from ACache methods.
Coordinator
Mar 6, 2013 at 1:13 PM
Unity's playback is lame... ;-)

My typical recommendation for a situation like this is to decode to a circular buffer on a separate thread. That way your playback filter doesn't have to wait for the decoder. Not sure how well it'll work on Unity/iOS, though...
Coordinator
Mar 6, 2013 at 1:17 PM
If the GC in Unity/iOS is intelligent enough, you can try dropping ACache entirely. NVorbis isn't as efficient on Windows without it, but maybe you can get away with it...

On the circular buffer, make sure it's big enough to ensure you don't get buffer underruns, but don't make it so big you have to wait more than 100 - 300 ms for it to fill up... I would expect a 1 second buffer to be plenty, though mobile devices are tricky and may need a slightly larger buffer. In no case would I go past 5 seconds.
Mar 6, 2013 at 1:25 PM
ioctlLR wrote:
If the GC in Unity/iOS is intelligent enough, you can try dropping ACache entirely. NVorbis isn't as efficient on Windows without it, but maybe you can get away with it...

On the circular buffer, make sure it's big enough to ensure you don't get buffer underruns, but don't make it so big you have to wait more than 100 - 300 ms for it to fill up... I would expect a 1 second buffer to be plenty, though mobile devices are tricky and may need a slightly larger buffer. In no case would I go past 5 seconds.
Thanks for the info.

reL Dropping ACache. How do you mean?
Coordinator
Mar 6, 2013 at 1:33 PM
Edited Mar 6, 2013 at 1:34 PM
Comment-out the entirety of ACache.cs, then fix all the broken "Get"s with array initializers (ACache.Get<float>(2048) becomes new float[2048]) and the broken "Return"s with varName = null.

You'll need to set up loops in a few places to initialize all the 2nd & 3rd ranks. Once you've done that, there are some significant optimizations you can apply if you can make guarantees about how NVorbis will be used.
Mar 6, 2013 at 1:50 PM
ioctlLR wrote:
Comment-out the entirety of ACache.cs, then fix all the broken "Get"s with array initializers (ACache.Get<float>(2048) becomes new float[2048]) and the broken "Return"s with varName = null.

You'll need to set up loops in a few places to initialize all the 2nd & 3rd ranks. Once you've done that, there are some significant optimizations you can apply if you can make guarantees about how NVorbis will be used.
ah ok I get you with the array initializers for , int[] or int[] or int[][][] + pdi.FloorData = ACache.Get<VorbisFloor.PacketData>(_channels); for ex

but not sure what you mean "Return"s with varName = null
ex: ACache.Return(ref cls); where cls = int [][] you mean ?

ty!
Coordinator
Mar 6, 2013 at 1:53 PM
Actually, it's simpler than that... "ACache.Return(ref cls)" becomes "cls = null".
Mar 6, 2013 at 1:59 PM
ioctlLR wrote:
Actually, it's simpler than that... "ACache.Return(ref cls)" becomes "cls = null".
kk, I'll give that a shot. Thanks for the conversation/advice;-)
Mar 6, 2013 at 3:10 PM
Edited Mar 6, 2013 at 3:46 PM
Happy to report it now works fine on iOS as well.
Used as a dll once I removed the ACache templates as you suggested above and I also had to replace the Huffman.BuildLinkedList as an straight int due to AOT issue.

re: Circular buffer I'll implement later. Looks like a decent one at http://circularbuffer.codeplex.com/
EDIT: ah..I see you have a ringbuffer.cs in NVorbis already;-)

Thanks for the help and the cool tool!

Cheers,
Paul
Coordinator
Jul 15, 2013 at 3:11 PM
sonicviz,

Just FYI, the newest release of NVorbis should perform much better and should be usable in unity3d as-is.

Thanks for the interest!
Jul 16, 2013 at 1:19 AM
A1. TY for the update!