Latest release consumes memory endlessly?

Mar 21, 2013 at 7:29 PM
Edited Mar 21, 2013 at 7:50 PM
Hi,

I tried updating to the latest, and while streaming blocks, ACache seems to grow without end until I hit an OutOfMemory error...

Using a memory profiler I isolated the problem to Residue2::Decode(), the float cache had over a Gb of memory over 30 seconds of streaming a few tracks.
(Edit : Actually this was one of the float[] instances, doesn't mean Decode is the culprit. There are a LOT of instances...)

I'm calling ReadSamples() from two threads, but never at the same time (locked with a mutex). One thread does most of the calls.

Any idea about how to go about solving this?

Edit : For the moment, since I ensure that I'm never calling ReadSample() on two threads at a time, I'll remove usages of ACache and use instance-local buffers. I know this is potentially unsafe, but it should be fine for my use.
Mar 22, 2013 at 12:39 AM
I found a way simpler workaround, in ACache :
            internal static Scope Current
            {
                get { return _global; }
            }

        internal static void BeginScope()
        {
                // do nothing
        }

        internal static void EndScope()
        {
                Scope.Current.Dispose();
        }
Dispose is more of a cleanup operation because the object is reused over and over.
Coordinator
Mar 22, 2013 at 2:51 PM
Your change to ACache is not really "safe"... scopes can be nested (and I think actually are nested), which creates a possible reuse collision in the outer scope.

That said, changing everything you can over to instance-local will probably do the job. It loses the multi-threaded packet decode scenario I originally designed for, but that's probably OK... I never wrote the code to make use of that scenario and probably won't. If you go through and audit the use of ACache, you might find it can be replaced with instance-local or fully local buffers. Performance benchmarks are your friend here.
Jun 26, 2013 at 9:17 PM
I think I'm having the same problem here as well, was this problem resolved?
Coordinator
Jun 26, 2013 at 9:28 PM
Nope. :(

I've not touched this project much since the conversation above (work & life are getting in the way).

The issue is probably related to ACache. I haven't yet profiled it to find out, but that's my expectation. That said, we should probably dump it and go with instance and local buffers. It adds complexity and is hard to maintain. If it has a memory leak, that's even more reason to dump it.

All that said, I'll accept patches if you have the time... :)
Coordinator
Jun 27, 2013 at 1:28 PM
Well, strike that... I profiled it this morning and found two things:
  • ACache improves performance & memory usage tremendously vs. just allocating as needed, and
  • The packet data buffering added in commit 8cb786b1165a made memory usage go through the roof.
That being the case, I've reverted the packet data buffering and will work on eliminating the need for ACache. Once that is done, I'll probably work on re-implementing the Ogg reader so it isn't as I/O heavy but still uses a "semi-fixed" amount of memory. That should address all the outstanding memory usage issues.

Thanks for your interest!
Jun 27, 2013 at 7:52 PM
Hey, awesome! :D I just tried the newest version, you seem to have nailed the memory problem!