Damage identification for structural health monitoring (SHM) is explored through applying the x-vector speaker recognition technique in the structural domain. Using the progressive damage tests from the Z24 Bridge Benchmark dataset, a time-delay neural network (TDNN) is trained as an acoustic model to classify the provided global damage scenarios. The outputs of a pre-final layer, called x-vectors, are used as damage-sensitive features for identification of damage presence and mechanisms. Since the developed TDNN has learned the underpinning dynamics of the damage mechanisms in the Z24 tests, we apply it as a basis for damage identification problems tangential to the Z24 progressive damage classification task. Transfer learning and domain transfer are investigated via application of the developed TDNN towards local damage identification of the Z24 Bridge, and global and local damage identification for the unseen LANL SHM Alamosa Canyon Bridge, UC-Irvine Bridge Column, and Bookshelf studies. Supervised and unsupervised classification techniques are explored to assess this method, and strong results in damage detection are obtained for these SHM problems.