Mills: I seem to remember us knocking on your door.
Doe: Oh, that's right. And I seem to remember breaking your face.

---

Mills is telling Doe that they would have eventually solved the case on their own and that he should remember they did find out where he lived.

---