C# and Pointers - experienced question

Question

I know c# isn't a great place to do pointer arithmetic, but I am writing an app that interacts with a server that returns a great deal of data with pointers actually in the returned data. The data is predictable for the first few fields, but after a little while, the dynamic part of the data stream comes into play. The data stream consists of the data requested and then a pointer to the start of the data points associated with it. My only problem is that there is no good way for me to jump straight to the address pointed to by this pointer they included. And it is not an offset, it is literally a pointer in that if we were programming in c/c++, we could assign a pointer this value and it would jump straight to the data. Any ideas? I've tried IntPtr, casting the Int32 to an object and then to Int32[], and a couple other things, but nothing's working. I always get an invalid cast exception. Thanks!