When choosing a high voltage capacitor, factors like voltage rating, capacitance value, tolerance, dielectric material, size, and environmental conditions need to be considered. It is important to select a capacitor that meets the requirements of the specific application.