At that point, having done everything else, the remaining issue was clearly Core Data. So I tried more things, re-read everything I could about Core Data performance (for the nth time), ran experiments, spent tons more time in Shark. Trying to get it good. No go.
Finally I realized I had to switch away from Core Data and use SQLite more directly.
Like Gus Mueller, I’m really glad Brent wrote this up.
Brent’s points can be reduced to creating objects in memory just to manipulate rows in storage is inefficient.
This pains me. Not because it isn’t true – it very much is. It pains me for the same reason I bet it pains Aaron Hillegass – Apple already solved this problem. In 1995.
Enterprise Objects Framework – Core Data’s predecessor – allows raw SQL access. This escape hatch mostly allows you to use lovely happy objects everywhere until you hit the wall. Then you measure and sprinkle in small, focused SQL bits in just the right places. Tada, performance issues solved without having to jettison a goodly chunk of your code.
Aside to Brent and developers in similar circumstances: it’s in your interest to check out BNRPersistence. It can be much faster than even using SQLite directly in many scenarios.
※ ※ ※
That said, I don’t fault Core Data for not including EOF’s raw-SQL escape hatch. It’s an inelegant hack that necessarily totally bypasses EOF’s machinery, leaving its caches out of sync with DB reality.
A better method would be if Core Data’s NSPersistentStoreCoordinator offered a method that accepted an NSPredicate or NSFetchRequest (I haven’t decided which would be better) along with an NSArray of NSManagedObjectOperations.
NSManagedObjectOperation would be a new, lightweight class that declaratively describes how a matched object should be operated upon. The idea is it would be easily compiled down to raw SQL, yet still be compatible with all the other (and future) store types.
Think of it as OpenCL, for Core Data.