Garbage Collection - Memory Leaks

Question

I've written a C# application takes a string and converts it into a voice buffer using MS SpeechLib (knowledge of MS SpeechLib is irrelevant to my question) and does some work with it.

I have a custom voice class where the voice buffer gets created that also contains cleanup code (IDipose and Finalize). The buffer is in the form of a byte array.

The main thing the cleanup code does is set the voice buffer to null. All other variables are value types.
Once every second I create a new voice object (roughly 1MB to 5MB each), do some work with it, call the cleanup code, and then set the object to null. Well after running for an hour or so, my application is taking up anywhere from 150MB to 500MB of physical memory.

I used a program to find out where a memory leak might be occurring and it appears it could be with the byte array audio buffer, regardless of my cleanup code. I went through my code to make sure nothing references this, but apparently there must be some memory leaks in other places.

My question is, what are some other things I can check or do to help prevent this from happening?

Also, it appears the Garbage Collecter is doing a poor job with all my allocations so is there anything else I can do besides forcefully call the GC?