According to legend; when Ly Thai To established Hanoi in 1010; the refurbished city walls kept falling down until a magical white horse appeared and indicated where new fortifications should be built. In reverence and gratitude; Ly Thai To established Bach Ma Temple; and the white horse became a guardian spirit for the capital.

In its present form the temple dates from the 19th century when it was restored with the help of contributions from the Hang Buom Chinese community. The white horse still figures prominently in the pantheon of deities represented here and there in the temple; but the Chinese also introduced a martial divinity of their own; Ma Vien; the general who had defeated the Trung Sisters in AD43 and re-established Chinese control over Vietnam.