Why does my mushroom compost smell of ammonia and fail?

A strong ammonia smell means the composting fermentation is not finished and there is still free ammonia in the compost, which is toxic to the mushroom mycelium and will stop the spawn from running, exactly as you describe. The usual causes are too much nitrogen or manure for the straw, not enough turns, or not letting the compost ferment and then pasteurise long enough to drive off and convert the ammonia. The fix is to complete the process, give the compost the proper turns so it heats evenly and the microbes use up the free ammonia, and pasteurise and condition it until it smells sweet and earthy with no ammonia before you spawn. Never spawn compost that still smells of ammonia. Getting the straw to nitrogen balance and the turning schedule right is the skill here. Learn the correct recipe, turning and conditioning hands on from DMR Solan or your KVK so your next batch finishes properly.