The only way to guarantee catching early allocator use is to switch the system's allocator (ie, libc itself) to the new one. Otherwise, the application will end up with two allocator implementations being used: the application's custom one and the system's, included and used within libc (and other system libraries, of course.)
So I don't know a ton about how this stuff works, but Firefox does redirect allocations made by system libraries to the mozjemalloc allocator. I know because I've been fighting with it recently because it wasn't always doing it for MinGW and it's mismatch the alloc/free. This is https://bugzilla.mozilla.org/show_bug.cgi?id=1547519 and dependencies.
Fingerprinting: It is most likely possible to be creative enough to fingerprint what memory allocator is used. If we were to choose from different allocators at runtime, I don't think that fingerprinting is the worst thing open to us - it seems likely that any attacker who does such a attack could also fingerprinting your CPU speed, RAM, and your ASLR base addresses which depending on OS might not change until reboot.
My post was more along the lines of: what system-level components, if replaced, have a potentially visible effect on current (or future) fingerprinting techniques?
I imagine that we have not seen the limit of creativity when it comes to fingerprinting hardware characteristics of the user's machine. These would include graphics card performance, CPU performance, cache sizes (CPU and RAM), FPU operation (?), perhaps even disk speed. Allocator, sure too.
And: If, or how, does breaking monocultures affect fingerprinting? Breaking monocultures is typically done to help secure an environment through diversity, causing an attacker to have to spend more resources in quest for success.
The only reason I can think of to choose between allocators at runtime is to introduce randomness into the allocation strategy. An attacker relying on a blind overwrite may not be able to position their overwrite reliably AND it has the cause the process to crash otherwise they can just try again.
Allocators can introduce randomness themselves, you don't need to choose between allocators to do that.
I'm assuming you're talking about randomness of the address space?
No, randomization of the allocations.
Imagine a simplistic example of grabbing 1MB of memory, and requesting 3 allocations of 100KB each. In a deterministic allocator you'll always have the allocations at <base>, <base+100Kb>, <base+200Kb> In a randomized allocator the allocations could be at <base>, <base+100Kb+r1>, <base+200Kb+r1+r2>
This removes determinism for the attacker in laying out the heap exactly how they want it.
As I mention below, this randomness is easily bypassed in the content process (where the attacker has a JIT engine to work with) and may provide some security on the other side of an IPC boundary.
In virtually all browser exploits we have seen recently the attacker creates exploitation primitives that allow partial memory read/write and then full memory read/write. Randomness introduced is bypassed and ineffective. I've seen a general trend away from randomness for this purpose. The exception is when the attacker is heavily constrained - like exploiting over IPC or in a network protocol. Not when the attacker has a full Javascript execution environment available to them.
In conclusion, while it's possible hardened_malloc could provide some small security increase over mozjemalloc, the gap is much smaller than it was when I advocated for allocator improvements 5 years ago, the effort is definitely non-trivial, and the gap is closing.
I'm curious about how breaking monocultures affect attacks. I think supporting hardened_malloc (or <insert arbitrary allocator here>) would provide at least the framework for academic exercises.
At Mozilla in the past we have evaluated exploit mitigations by hiring an exploit developer to write or adapt an exploit to bypass a mitigation and give us their opinion. The replace_malloc framework is effectively the framework for performing such an evaluation.
Exploits have become more and more frameworked. They abstract away the exploitation primitives and write the exploits against an API. Then for each vulnerability they construct the same primitives using different or slightly different techniques and use mostly the same exploit.
'Breaking the monoculture' to me feels like "The attacker doesn't know X so they have to guess and they might guess wrong and lose their ability to exploit." This assumes a) they have to guess and b) they lose their ability to exploit.
(a) does not seem true. When they have a JIT to work with, they can almost always safely inspect the system before taking any risks. (b) also does not seem true. Reading memory is fairly safe so the probability of crashing is low.
I think there is *significant* advantage to trying new approaches and experimenting. Alternate implementations and experimentation. However to toss those experiments in for no clear reason besides 'diversity' does not seem advantageous.
-tom