A commonly used rule of thumb states that the thread engagement length for a screw in a tapped hole should be 1.5D, where D is the major diameter of the screw. A variation on this recommends an engagement length of 1.0D for steel, 1.5D for cast iron, and 2.0D for aluminum.
A more accurate approach is suggested in a technical bulletin from the Industrial Fastener Institute titled "Calculating Thread Strength." This bulletin explains how to determine the needed thread engagement length based on the shear strength of the tapped material.
Shear strength is a material property which is not often listed on datasheets. If you don't have data, you can assume that the shear strength is 50 percent of the ultimate tensile strength. The stress on the internal threads must not exceed this value.
To calculate the stress on the internal threads, divide the tensile load by the internal thread stripping area. Internal thread stripping areas for standard fastener sizes can be found in IFI Inch Fastener Standards, 7th Edition; or in IFI Metric Fastener Standards, 3rd Edition. For the tensile load, you can use the proof load of the screw, or, for a more conservative approach, multiply the tensile strength of the screw by its tensile stress area (also found in the IFI Fastener Standards books). The second approach ensures that the screw will always break before the threads do, no matter what.
This approach assumes that the load is divided evenly among all of the engaged threads, which isn't completely true. Also, internal thread areas in cast materials may be less than the published values due to porosity. However, this calculation is a good starting point.
Designing joints with the proper thread engagement length can help ensure that threads don't strip or yield. This can prevent parts from failing on the assembly line or in service.