Ambrose of Milan
"[T]hey [the Novatian heretics] have not the succession of Peter, who hold not the chair of Peter, which they rend by wicked schism; and this, too, they do, wickedly denying that sins can be forgiven [by the sacrament of confession] even in the Church, whereas it was said to Peter: ‘I will give unto thee the keys of the kingdom of heaven, and whatsoever thou shalt bind on earth shall be bound also in heaven, and whatsoever thou shall loose on earth shall be loosed also in heaven’[Matt. 16:19]" (Penance 1:7:33 [A.D. 388]).
 
Ignatius of Antioch
"Ignatius . . . to the church also which holds the presidency, in the location of the country of the Romans, worthy of God, worthy of honor, worthy of blessing, worthy of praise, worthy of success, worthy of sanctification, and, because you hold the presidency in love, named after Christ and named after the Father" (Letter to the Romans 1:1 [A.D. 110]).
"You [the church at Rome] have envied no one, but others you have taught. I desire only that what you have enjoined in your instructions may remain in force" (ibid., 3:1).

Irenaeus
"But since it would be too long to enumerate in such a volume as this the succession of all the churches, we shall confound all those who, in whatever manner, whether through self-satisfaction or vainglory, or through blindness and wicked opinion, assemble other than where it is proper, by pointing out here the successions of the bishops of the greatest and most ancient church known to all, founded and organized at Rome by the two most glorious apostles, Peter and Paul, that church which has the tradition and the faith which comes down to us after having been announced to men by the apostles. With that church, because of its superior origin, all the churches must agree, that is, all the faithful in the whole world, and it is in her that the faithful everywhere have maintained the apostolic tradition" (Against Heresies 3:3:2 [A.D. 189]).

Augustine
"There are many other things which rightly keep me in the bosom of the Catholic Church. The consent of the people and nations keeps me, her authority keeps me, inaugurated by miracles, nourished in hope, enlarged by love, and established by age. The succession of priests keep me, from the very seat of the apostle Peter (to whom the Lord after his resurrection gave charge to feed his sheep) down to the present episcopate [of Pope Siricius]" (Against the Letter of Mani Called "The Foundation" 5 [A.D. 397]).
 
Peter Chrysologus
"We exhort you in every respect, honorable brother, to heed obediently what has been written by the most blessed pope of the city of Rome, for blessed Peter, who lives and presides in his own see, provides the truth of faith to those who seek it. For we, by reason of our pursuit of peace and faith, cannot try cases on the faith without the consent of the bishop of Rome" (Letters 25:2 [A.D. 449]).

Council of Ephesus

"Philip, presbyter and legate of [Pope Celestine I] said: ‘We offer our thanks to the holy and venerable synod, that when the writings of our holy and blessed pope had been read to you . . . you joined yourselves to the holy head also by your holy acclamations. For your blessednesses is not ignorant that the head of the whole faith, the head of the apostles, is blessed Peter the apostle’" (Acts of the Council, session 2 [A.D. 431]).

"Philip the presbyter and legate of the Apostolic See said: ‘There is no doubt, and in fact it has been known in all ages, that the holy and most blessed Peter, prince and head of the apostles, pillar of the faith, and foundation of the Catholic Church, received the keys of the kingdom from our Lord Jesus Christ, the Savior and Redeemer of the human race, and that to him was given the power of loosing and binding sins: who down even to today and forever both lives and judges in his successors. The holy and most blessed pope Celestine, according to due order, is his successor and holds his place, and us he sent to supply his place in this holy synod’" (Acts of the Council, session 3 [A.D. 431]).