Anders Ygeman said charter aircraft would be used to deport the refugees but it would take several years.
Some 163,000 refugees applied for asylum in Sweden in 2015, the highest per capita number in Europe.
The numbers have fallen significantly since Sweden imposed tighter border controls in 2016.
Along with Germany, Sweden is a prime destination for refugees and other refugees entering the EU illegally.
Of the approximately 58,800 asylum cases processed in Sweden in 2015, 55% were accepted.
Of those facing expulsion, Anders Ygeman was quoted in Swedish media as saying: “We are talking about 60,000 people but the number could climb to 80,000.”
He later tweeted to say he had not taken a position on how many refugees had grounds for asylum, it being a matter for the authorities and the courts.
Sweden earlier this week became the latest of a number of European nations to see tensions over migrants heightened by violence. A 15-year-old asylum seeker was arrested in Molndal, near Gothenburg, after a 22-year-old asylum centre employee was stabbed to death.
More than one million refugees and migrants travelled to Europe in 2015, most fleeing conflicts in Syria, Iraq and Afghanistan.
The UN says another 46,000 people have arrived in Greece so far this year, with more than 170 killed making the dangerous crossing from Turkey.
In the latest such accident, at least 11 migrants drowned – mostly children – after their boat capsized off the island of Samos, the Greek coast guard says. Several are still missing.
On January 27, a draft European Commission report said Greece “seriously neglected” its obligations to control the external frontier of Europe’s passport-free Schengen zone.
Greek government spokeswoman Olga Gerovasili accused the Commission of “blame games” and said it had failed to act on a program agreed in 2015 to relocate tens of thousands of migrants and refugees stranded in Greece.
Denmark has also faced criticism this week after approving legislation to seize the valuables of refugees in the hope of limiting the influx of migrants.
The US House Ethics Committee has voted to release its report on former Republican Representative…
ABC News has agreed to pay $15 million to President-elect Donald Trump to settle a…
South Korea’s parliament has voted to impeach President Yoon Suk Yeol over his failed attempt…
Israeli war planes have carried out more than 100 air strikes in Syria on December…
President-elect Donald Trump has threatened to impose 100% tariffs on the BRICS countries if they…
Syrian troops have withdrawn from the city of Aleppo following an offensive by rebels opposed…