The Polish third seed went down 7-5, 6-1 to the 15th seed from Ukraine.
Kostyuk, the Madrid Open winner, moved into the quarter-finals at Roland Garros for the first time in her career.
“I’m still in shock to beat such an unbelievable player who won four times here and I lost four times to her before this, I never took a set off her,” 23-year-old Kostyuk said.
Swiatek will now go another year without the title she first won in 2020 and then three times consecutively between 2022 and 2024.
With her fourth-round exit, it was the joint worst performance at Roland Garros for the 25-year-old, equal with the last-16 departure she suffered on her debut in Paris seven years ago.
The first set went on serve until the seventh game when Swiatek pounced on Kostyuk’s service to move in front.
From there, the Pole’s own delivery deserted her as the world number 15 hit back to level before the pair again exchanged another pair of breaks.
Kostyuk then dug in to hold and force the former world number one to try and serve to stay in the set.
Two Swiatek double faults gave Kostyuk the chance to claim the opener and she did with a low cross-court backhand that whizzed past her opponent at the net.
Swiatek swiftly departed the court at the interval in a bid to regroup.
She came back out firing as she immediately nosed ahead with a break to 15.
But Kostyuk was not to be denied as she sealed a three-game streak with a bruising forehand winner past a stranded Swiatek to lead 3-1.
A stunning rally concluded with a reflex volley at the net that Swiatek could not equal. In the next game a dominant Kostyuk held to love to move Swiatek to the brink.
The match was up for the six-time major winner as Kostyuk consistently pinned the Pole back with her powerful groundstrokes to win the final two games.
In another match, Sorana Cirstea continued her renaissance this season by downing Chinese world number 148 Wang Xiyu 6-3, 7-6 (7/4) to book a spot in the quarter-finals.
Cirstea, who will retire at the end of the season, is enjoying a golden last year on tour. She backed up a semi-final run at the Italian Open by equalling her previous best performance at Roland Garros and progressing to the last eight for the first time since 2009.
Cirstea got a double break ahead to lead 5-1 in the first set before Wang hit back as the Romanian served for the opener.
The 18th seed made no such error the second time around as she polished off the opener in just under an hour on Court Suzanne Lenglen.
The second set followed much the same rhythm as the first with Cirstea storming to five games before Wang fought back.
This time the 25-year-old took Cirstea all the way as the veteran had to dig deep to stave off her opponent in a tie-break. (AFP)
