Microsoft never released the format specification (.CHM), but there are a few reverse engineered descriptions of the format. CHM is a ITS (InfoTech Store) format.
Unofficial CHM Specification
Matthew Russotto's Microsoft's HTML Help format description and more ...