Big data contains a tremendous amount of dark knowledge. The community has realized that effectively exploring and using such knowledge is essential to achieving superior intelligence. How can we effectively distill the dark knowledge from ultra-large-scale data? One possible answer is: “through Transformers”. Transformers have proven their prowess at extracting and harnessing the dark knowledge from data. This is because more is truly different when it comes to Transformers. In this talk, I will showcase our recent work on transformers named ViTAE, on many dimensions of “more” including: model parameters, labeled and unlabeled data, prior knowledge, computing resource, tasks, and modalities. Specifically, ViTAE has more model parameters and more input modality support; ViTAE can absorb and encode more data to extract more dark knowledge; ViTAE is able to adopt more prior knowledge in the form of biases and constraints; ViTAE can be easily adapted to larger-scale parallel computing resources to achieve faster training.

ViTAE has been applied to many computer vision tasks and has proven its promise, such as image recognition, object detection, semantic segmentation, image matting, pose estimation, scene text understanding, and remote sensing.

You can find the source code for this work at here.