Black smoke rising from a chimney above the Sistine Chapel indicated that their ballot had been inconclusive.
The 115 cardinal-electors will vote four times daily until two-thirds can agree on a single candidate.
The election was prompted by the surprise abdication of Benedict XVI. There is no clear frontrunner to take over from him as head of the Church.
The vote was the first held by the cardinals since they entered the conclave on Tuesday afternoon, and was not expected to produce a positive result.
Crowds who had braved rain and storms to watch the start of the conclave on big screens in St Peter’s Square cheered as the black smoke appeared.
The electors will now return to their hotel for the night and go back to the chapel on Wednesday morning to resume voting.
White smoke from the chimney will indicate that a new pope has been chosen.
Pope Benedict’s resignation and the recent damage to the Church’s reputation make the choice of the cardinal-electors especially hard to predict.
At 16:30 local time on Tuesday, 115 cardinal-electors – all under 80, as those over 80 are excluded – entered the Sistine Chapel, chanting the traditional Litany of the Saints.
Each man in turn stepped up and placed his hands on the Gospel to swear an oath in Latin.
Afterwards Msgr Guido Marini, papal master of ceremonies, called out the words “Extra omnes” – “Everybody out” – and the chapel doors were locked to outsiders.
From now on the cardinals will eat, vote and sleep in closed-off areas until a new pope is chosen.
Jamming devices in the Sistine Chapel should block all electronic communication and anyone tweeting would in any case risk being excommunicated.
Earlier on Tuesday the cardinals attended a “Mass for the Election of the Supreme Pontiff” in St Peter’s Basilica.
In his homily, the Dean of the College of Cardinals, Cardinal Angelo Sodano, praised the “brilliant pontificate” of Pope Benedict and implored God to grant another “Good Shepherd” to lead the church.
MOcRfK5Kv44Image source: Wikimedia Commons President-elect Donald Trump celebrated his election victory at the Ultimate Fighting…
Millions of voters across the US chose to return Donald Trump to the White House…
Donald Trump declares victory in the US election as he addresses jubilant supporters in Florida.…
Stocks around the world are rising as Donald Trump appears to be on the cusp…
Donald Trump has won Pennsylvania, North Carolina and Georgia and taken a lead over Kamala…
Quincy Jones, the celebrated musician and producer who worked with Michael Jackson, Frank Sinatra, Ray…