Thanks for all the tips. unset and memory_get_usage were the solution
(didn't know about these two functions).
I managed to track the memory usage with memory_get_usage and used unset
to dispose of unnecessary variables. The peak memory usage is now down
to 21MB from 41MB.