While Transformers are not new to deep learning, their successful application to computer vision is. Transformers holding the SOTA in a vision benchmark is certainly a massive breakthrough, but it’s unclear whether they’ll be able to compete with convolutional networks in the (relatively) “low-data low-compute” regime long term.
Even more interesting is the potential for a convergence of NLP and CV around similar architectural components. If this trend is to continue, it could rapidly accelerate the progress of the field as a whole as the DL community, as its many niches and sub-categories begin to adopt similar techniques to solve wildly different problems.
Transformers have only been around for 4 years, but it’s clear that their impact on DL research will resonate for years to come.
