The visit comes two weeks after President Ghani invited the Taliban to begin peace talks without preconditions to end the 16-year-old war in the Kabul Process Conference last month.
The visit, which was not announced in advance due to security concerns, comes as the US injects new resources into the stalemated war. "We've had some groups of Taliban - small groups - who have either started to come over or expressed an interest in talking".
Afghanistan's President Ashraf Ghani recently offered a comprehensive peace deal to the Taliban, but the militants have not responded and have increased their attacks.
In return, the militants would have to recognize the Kabul government and respect the rule of law.
They said that while Afghan forces are getting better, the Taliban is likely to threaten Afghan stability in 2018.
The group has insisted it would only negotiate with the United States, which it calls a "foreign occupying force". "Not a military victory, the victory will be a political reconciliation with the Taliban, which has achieved a stalemate in recent years and shown little interest in conceding to the Kabul government", Mattis added.
An Afghan official says at least five Afghan police forces have been killed after insurgents attacked their checkpoint in western Farah province.
The Afghan government and the Taliban held peace talks in 2015, but they broke down nearly immediately.
The military is battling a Taliban insurgency while also fighting the Islamic State group's gradual expansion in the country's north and its regular attacks in the capital Kabul.
The UN said on Tuesday that 30,672 people have been displaced in Afghanistan since the beginning of the year due to continued conflict.
"Right now we want the Afghans to lead and to provide the substance of the reconciliation effort", Mattis said.