Serena was playing her first match since Wimbledon, while Venus has played singles and doubles at three hard-court events in recent weeks.
The pair made a poor start against Ukraine's Kostyuk and American Stearns before shaking off what Serena called "a little rust" in the second set.
They quickly went down 6-0 in the deciding 10-point tie-break but fought back impressively.
Serena produced two service winners to force the tie-break back on serve, but Kostyuk and Stearns held firm before securing victory on a double fault from Venus.
"I thought it was fun, it was special, it was good," Serena said.
"We were starting to feel it. It was really cool to be able to do this again."
The sisters would need wildcards to play either singles or doubles at the US Open, which begins on Sunday, 30 August.
Serena - who described her knee injury as "a little traumatic experience, but what doesn't kill you makes you stronger" - has only played one singles match since her return.
Venus has lost her past 13 singles matches but has had more success in the doubles, reaching the quarter-finals of the women's tournament in New York last year.
At their peak, they won 14 Grand Slam doubles titles together and three Olympic gold medals.
In the singles, world number two Elena Rybakina beat Magdalena Frech 6-4 7-6 (7-2) either side of a lengthy rain delay to reach the fourth round.
Australian Open champion Rybakina could replace Aryna Sabalenka at the top of the world rankings in Cincinnati.
Rybakina must reach the final, external to have a chance at surpassing Sabalenka. If Sabalenka reaches the quarter-finals, Rybakina would need to win the title to become world number one.
Canadian Open champion Iga Swiatek shook off a poor first set to beat Greece's Maria Sakarri 4-6 6-1 6-1, while third seed Jessica Pegula beat fellow American Emma Navarro 7-5 6-2.

4 hours ago
3









