The court announced Monday it cannot consider the merits of the claim by six same-sex couples that New Jersey's civil union law is unconstitutional until there is a trial record.
Gay couples unsuccessfully sued New Jersey four years ago for the right to marry. They claim that by creating civil unions, the state has not fulfilled a court order to treat them the same as heterosexual couples seeking to marry.
The justices were split 3-3, one vote shy of the four needed for a motion to be granted.
Chief Justice Stuart Rabner is among those who voted to deny the motion.
Justice Virginia Long, one of the justices who dissented, agreed there is an insufficient record for debating the merits of the claim, but she said hearing oral arguments would have helped guide the justices on how best to go about creating a judicial record.
Steven Goldstein, who leads the state's largest gay rights group, Garden State Equality, said the decision perpetuates the unequal legal status of same-sex couples and their children.
"Same-sex couples will continue to be denied the consistent right to visit one another in the hospital, to make medical decisions for one another, and to receive equal health benefits from employers, all because of the deprivation of the equality and dignity that uniquely comes with the word marriage," he said in a statement.
Len Deo, president of the New Jersey Family Policy Council, which supports the traditional view of marriage as between one man and one woman, saw the court ruling as a partial victory for gay rights advocates.
"We were hoping the Supreme Court would just reject the application outright," he said. However, "we believe they have a higher thresholdof proof now."
Hayley Gorenberg, the couples' Lambda Legal lawyer, said abundant proof exists that civil unions aren't equal to marriage.
She said more than 100 people testified before the state's Civil Review Commission about having experienced discrimination under the civil union law.
Opponents dismissed the group and its 2008 report as having a built-in bias in favor of expanding gay rights.
For the better part of the past decade, New Jersey has been one of the states at the center of the fight over expanding recognition of gay relationships. It's one of a handful of states where social conservatives have not been able to amend the state constitution to ban gay marriage.
In 2006, the state Supreme Court ruled unanimously that gay couples deserved equal treatment. But the court ruled by a 4-3 vote that the state did not have to recognize them as married. Instead, the court left the details of their legal status up to the Legislature, which responded by making the state the third to offer civil unions.
More than 4,400 couples have registered.
New Jersey is now the only state with civil unions. Gay marriage is legal in Connecticut, Iowa, Massachusetts, New Hampshire, Vermont and Washington, D.C.
Gay rights groups tried to get New Jersey lawmakers to reconsider and recognize gay marriage, but the bill died after the state Senate voted it down in January. Supporters say they had the momentum until November, when Republican Chris Christie, who opposes gay marriage, was elected governor.