The ship spent a few hours south of the border, nearing a South Korean border island, before returning after repeated warnings from the South, they added.
North Korea disputes the maritime border and has sent boats across it in the past.
The incident comes amid joint military drills between the US and South Korea which are opposed by Pyongyang.
The North Korean patrol ship crossed the Northern Limit Line (NLL), which South Korea considers the maritime border between the two sides, at around 22:46 local time, South Korea’s defense ministry said.
The South broadcast warnings 10 times before the ship returned, at around 02:25 local time, the ministry added.
The NLL was drawn unilaterally by the United Nations Command at the end of the 1950-1953 Korean War.
However, North Korea has disputed the NLL and drawn its own border further south of the line.
“The North Korean ship’s NLL violation is seen as part of military drills,” defense ministry spokesman Kim Min-seok said.
“It is believed that [the ship] intended to test the South Korean military.”
The ship was said to have come within 15 miles of South Korea’s border island of Baengnyeong.
Monday’s incident comes on the same day joint annual US-South Korea military exercises began.
More than 12,500 US troops will take part in the exercises, which include Key Resolve, a computer-based simulation, and Foal Eagle, which involves air, ground and naval drills.
Pyongyang is opposed to the drills and has previously called them “exercises of war”.
Tuesday is also the last day of rare family reunions for North and South Korean relatives separated after the Korean War.
VqE2KhIJEIcThree Indian nationals have been arrested and charged over the killing of Sikh separatist leader…
President Joe Biden has urged pro-Palestinian protesters on university campuses to uphold the rule of…
Blue Ivy Carter has joined the voice cast of The Lion King prequel Mufasa: The…
At least five people, including a four-month-old baby, have been killed after dozens of tornadoes…
Harvey Weinstein has been hospitalized just days after his 2020 rape conviction in New York…
Hamas has published a video showing the first proof of life of US and Israeli…