Anders Ygeman said charter aircraft would be used to deport the refugees but it would take several years.
Some 163,000 refugees applied for asylum in Sweden in 2015, the highest per capita number in Europe.
The numbers have fallen significantly since Sweden imposed tighter border controls in 2016.
Along with Germany, Sweden is a prime destination for refugees and other refugees entering the EU illegally.
Of the approximately 58,800 asylum cases processed in Sweden in 2015, 55% were accepted.
Of those facing expulsion, Anders Ygeman was quoted in Swedish media as saying: “We are talking about 60,000 people but the number could climb to 80,000.”
He later tweeted to say he had not taken a position on how many refugees had grounds for asylum, it being a matter for the authorities and the courts.
Sweden earlier this week became the latest of a number of European nations to see tensions over migrants heightened by violence. A 15-year-old asylum seeker was arrested in Molndal, near Gothenburg, after a 22-year-old asylum centre employee was stabbed to death.
More than one million refugees and migrants travelled to Europe in 2015, most fleeing conflicts in Syria, Iraq and Afghanistan.
The UN says another 46,000 people have arrived in Greece so far this year, with more than 170 killed making the dangerous crossing from Turkey.
In the latest such accident, at least 11 migrants drowned – mostly children – after their boat capsized off the island of Samos, the Greek coast guard says. Several are still missing.
On January 27, a draft European Commission report said Greece “seriously neglected” its obligations to control the external frontier of Europe’s passport-free Schengen zone.
Greek government spokeswoman Olga Gerovasili accused the Commission of “blame games” and said it had failed to act on a program agreed in 2015 to relocate tens of thousands of migrants and refugees stranded in Greece.
Denmark has also faced criticism this week after approving legislation to seize the valuables of refugees in the hope of limiting the influx of migrants.
Image source: Wikimedia Commons President-elect Donald Trump celebrated his election victory at the Ultimate Fighting…
Millions of voters across the US chose to return Donald Trump to the White House…
Donald Trump declares victory in the US election as he addresses jubilant supporters in Florida.…
Stocks around the world are rising as Donald Trump appears to be on the cusp…
Donald Trump has won Pennsylvania, North Carolina and Georgia and taken a lead over Kamala…
Quincy Jones, the celebrated musician and producer who worked with Michael Jackson, Frank Sinatra, Ray…