-[UIView addSubview:]
causes the subview to be retained, and -[UIView removeFromSuperview]
causes the view to be released. This is direct from Apple's UIView documentation. At that point, one of your objects needs to retain the CALayer or it will be deallocated.
I assumed that the CALayer was created by you directly. All UIViews are associated with CALayers; the CALayer is what a UIView draws into, and then its CALayer is composited onto the screen. If you're not working with CALayers directly, this is a symptom of another problem.
The core problem you're describing is an overrelease. It's possible you're not retaining a UIView when you should. I suggest you run the Xcode analyzer (in Xcode 4, choose the Product > Analyze) menu item and see what gets reported. Pay special attention to all of the memory management errors it reports.