In 1621 the old village church of St. Victor en Gezellen ('St. Victor and company') of Afferden was confiscated to serve as a protestant church, despite the fact there was only a handful of protestants. The catholic population of Afferden resorted to churches in the free catholic enclaves Megen and Ravenstein, except for a short period of religious freedom under French occupation in 1672-1674, when the old church was catholic once more. After the French had left the protestant rulers returned, the catholic religion was banned again and the priest was put in jail before being banished. But somehow the protestants had learned that oppressing the catholics had not helped, and that tolerance was a better way to keep the catholics loyal. In ca.1710 they allowed the catholics to build a barn church on the border between Afferden and Deest, a village that had always been a part of the Afferden parish.

In 1801, during a second occupation by the French, the old church was restituted to the catholics. However, it was only used on Sundays and holidays, the barn church on the other side of the village still being used on all the other days. In 1833 the barn church was falling apart while the old church was too small and needed maintenance. A new church, this time recognizable as such, was built near the barn church, and when it was completed the barn church was torn down and the old church was abandoned. In 1838 it was demolished with the exception of the tower.

In 1858 Deest had become a seperate parish, and this left Afferden with a church located at the edge of the village. In 1886 a plot of land on a more central location was purchased. In 1890 architect Carl Weber was commissioned to design a new church, with the explicit instruction that it had to be a cruciform basilican church in neo-Gothic style. This was not simply a matter of taste. By that time Weber's style had moved away from straight neo-Gothic, The architect, at that time almost 70 years of age, had recently earned a reputation with several impressive, but very costly, dome-churches in neo-Romanogothic style which were a far cry from the neo-Gothic, in some ways even early neo-Gothic churches of the first three decades of his career, like te church of Puiflijk a few kilometers to the west.

Weber delivered what he was asked, but did incorporate some Romanesque elements. Although the St. Victor is a cruciform basilica and is mostly neo-Gothic on the outside, the interior shows major influences of Romanesque, like the alternating piers and columns. Details on the tower are close to Romanesque and more in line with Weber's other work of that time. The cloverleaf shape ground-plan of the transept and choir is basically late-Romanesque and refers to the Munsterkerk in Roermond.  

To compensate for the absence of a dome and double towers, Weber built a tall church. The single tower was meant to become the tallest of the area, higher than the 62 metres tall tower of the church of nearby Druten. But despite warnings by the locals, Weber gave the tower insufficient foundation, and already after it reached 17 metres it began to sink in the unsteady ground. Although the construction was strengthened there was no money left to complete the tower. Instead, it was raised to the same height as the nave and covered with a flat roof, never to get a spire.

The last mass in the church was held in October 2016. A few months later it was bought by a Buddhist foundation, which turned it, together with the presbytery, into a monastery.