A method includes providing first and second channel layers in a p-type region and an n-type region respectively, forming a gate dielectric layer around the first and second channel layers, and forming a sacrificial layer around the gate dielectric layer. The sacrificial layer merges in space between the first channel layers and between the second channel layers. The method further includes etching the sacrificial layer such that only portions of the sacrificial layer in the space between the first channel layers and between the second channel layers remain, forming a mask covering the p-type region and exposing the n-type region, removing the sacrificial layer from the n-type region, removing the mask, and forming an n-type work function metal layer around the gate dielectric layer in the n-type region and over the gate dielectric layer and the sacrificial layer in the p-type region.