It marks the first time that Germany have ever lost a penalty shootout at a World Cup.
Julio Enciso headed Paraguay into a shock lead before half-time, but Kai Havertz levelled for Germany on 54 minutes with his third goal of the tournament.
Jonathan Tah saw a goal disallowed after a VAR review in extra time, before Paraguay held on to shock Germany in a wild, nerve-jangling shootout as the momentum swung back and forth.
Paraguay goalkeeper Orlando Gill saved from Havertz and Nick Woltemade, but his teammates blew two chances to win it before Jose Canale blasted in the winning spot-kick after Tah had blazed his effort over the bar.
It marked another premature exit for Germany, who were playing in their first knockout-stage appearance since winning the 2014 World Cup.
Paraguay celebrated arguably their greatest win at the tournament, but they face the daunting prospect of running into an in-form France next should Les Bleus beat Sweden.
Julian Nagelsmann handed Deniz Undav his first start of the tournament, after the Stuttgart striker scored three times in the opening two games as a substitute.
He came in for Jamal Musiala as Nathaniel Brown returned at left-back, having missed the Ecuador loss as a precaution due to a minor injury.
Miguel Almiron was recalled by Paraguay following a one-match ban after becoming the first player to be sent off for covering his mouth during an on-field confrontation under new Fifa rules.
Paraguay coach Gustavo Alfaro had urged his players to grasp the opportunity of a lifetime, and they came close to scoring barely a minute into the game.
Junior Alonso found space at the back post from a corner but was unable to beat Manuel Neuer.
Germany’s 7-1 win over Curacao in their opener offered a glimpse of their attacking firepower, but it papered over what has otherwise been an often blunt forward line.
Undav floated an effort wide as Germany struggled to break down a compact Paraguay defence, and they paid the price when they lost their shape before half-time, allowing Enciso to head home.
Paraguay recycled possession after Neuer punched clear a corner, working it out wide on the right for Matias Galarza to fizz in a cross that picked out a completely unmarked Enciso.
Remarkably, it was Paraguay’s first goal in the World Cup knockout stage, having failed to score in their previous five matches, including a 1-0 defeat by Germany in the last 16 in 2002.
Germany, toothless in the first half, brought Leon Goretzka on at the break but almost conceded again when an under-hit backpass from Joshua Kimmich forced Neuer to race out and stop Enciso doubling the lead.
However, they at last breached Paraguay’s dogged rearguard when Florian Wirtz cut in from the left and delivered a cross that Havertz glanced into the far corner to ease Germany’s nerves.
Paraguay lost Enciso to injury, and with it their main outlet, leading to a growing sense of inevitability that Germany’s pressure would eventually tell.
But Orlando Gill reacted sharply to claw away Havertz’s header from another Wirtz cross and Paraguay bravely clung on to send the match into extra time.
Tah thought he had put Germany in front with a powerful back-post header from Nathaniel Brown’s looping corner, but the goal was disallowed after a VAR review for a foul by Waldemar Anton on the goalkeeper.
Anton headed straight at Gill from another corner, but Paraguay withstood Germany’s set-piece barrage and then kept their cool to spring a monumental World Cup upset. (AFP)
