Brain alignment in deep neural networks emerges early and independently of object classification

Deep convolutional neural networks are leading models of biological vision, largely because of their strong brain alignment: their features predict neural responses better than earlier models. Yet they are believed to recognize objects differently, relying on texture where humans rely on shape and failing on perturbations humans handle effortlessly. What then does alignment reflect? Tracking three architectures densely during training, we find that alignment with human fMRI, EEG, and macaque electrophysiology is already largely present at initialization, when networks classify at chance, and reaches a plateau within one to five epochs; thereafter it changes only modestly while classification accuracy continues to climb to 75%. Network lesioning shows that a kernel's contribution to alignment is essentially uncorrelated with its contribution to classification throughout training. Brain-network alignment therefore appears to reflect the structure of the visual environment both systems encode, rather than a shared solution for classification.