Trailing 2-1 from the first leg, Erik ten Hag’s side stormed into an aggregate lead after just 18 minutes through forwards Hakim Ziyech and David Neres.
The electric Dusan Tadic, provider of his side’s first two goals, then added a brilliant third as he whipped an effort into the top corner from the edge of the area.
Chasing a fourth consecutive Champions League triumph, Real looked to have gained a lifeline when Marco Asensio scored with 20 minutes remaining.
But Ajax midfielder Lasse Schone scored a spectacular free-kick just two minutes later to leave Santiago Solari’s side requiring three goals to progress.
Gareth Bale, jeered by fans in Saturday’s defeat by Barcelona, was dropped by Solari but struck the post after emerging as a 29th-minute substitute for the injured Lucas Vazquez.
Real looked fragile without captain Sergio Ramos, serving a two-match ban “for clearly receiving a yellow card on purpose” in the tie’s first leg.
Their miserable night was compounded as Nacho was sent off in injury time for a second bookable offence, as Ajax became the first team to overturn a 2-1 first-leg home defeat in the Champions League.
This was a performance of sheer quality by a young Ajax side who arrived at the home of the European champions with no fear as they danced through to the quarter-finals.
A last-16 meeting with Real Madrid was the club’s first knockout tie in 13 years, and with six of their starting side aged 22 and under, there is the sense something special could be achieved this season – although the challenge of holding on to those players when the next transfer window opens will be significant.
The visitors needed at least two goals in Madrid and learned from their wastefulness in the first leg to silence the home support inside 18 minutes.
Ajax had to weather early pressure, but Ziyech’s fine finish came as a powerful blow to their bewildered hosts.
Tadic, almost unrecognisable from his Southampton days, produced a moment of excellence to set up Neres before seemingly putting the tie out of reach with a finish that epitomised his performance.
There was a long wait before the video assistant referee eventually confirmed Noussair Mazraoui had managed to keep the ball in play during the build up to Tadic’s goal – as nervous tension paved the way for unbridled joy in the away end.
Barcelona-bound Frenkie de Jong shone brightly too – with his pending £65m move to La Liga already a concern for Real, but it was fellow midfielder Schone who added the deserved gloss just as Real may have sensed hope.
Ajax could have had even more as goalkeeper Thibaut Courtois was forced to stop efforts from Donny van de Beek and Tadic, while the impressive Matthijs de Ligt stood firm alongside Daley Blind under periods of pressure.
The visitors more than held their own against the holders, and could play a huge part in the final stages of this season’s competition should they maintain such a devastating level of performance.
After three consecutive Champions League titles, the most successful side in the competition’s history did not see this coming.
Ajax head coach Ten Hag had suggested that absence of Ramos for Real was “not only a tactical loss for them but also a mental one”. Ultimately, he was was proven correct.
Real’s aggregate advantage disappeared quickly without their captain’s authority, despite a promising start in which Raphael Varane had headed on to the bar from a cross by Lucas Vazquez, who drew a fine save from Andre Onana before injury forced him off, to be replaced by Bale.
The Wales forward received a warmer reception than he had at the weekend, with Asensio also sharpening the hosts’ attack when he came on for Vinicius Junior before half-time.
When Asensio made it 4-3 on aggregate, he appeared to have dragged his side – with a reputation of churning out results in Europe – back into the tie.
But Ajax crushed any hope of a comeback just moments later, effectively ending Real’s season, given that they fell 12 points behind La Liga leaders Barcelona at the weekend.
Nacho’s dismissal and an injury to Bale in the closing stages rounded off an ugly night for the home side.
Having gone eight campaigns without failing to reach the Champions League semi-finals, it now looks as if a summer of change awaits at the Bernabeu.