A semiconductor device includes a semiconductor fin, a gate structure, and a dielectric isolation plug. The semiconductor fin extends along a first direction above a substrate and includes a silicon germanium layer and a silicon layer over the silicon germanium layer. The gate structure extends across the semiconductor fin along a second direction perpendicular to the first direction. The dielectric isolation plug extends downwardly from a top surface of the silicon layer into the silicon germanium layer when viewed in a cross section taken along the first direction.