A study on word-based and integral-bit Chinese text compression algorithms

Kwok Shing CHENG, Gilbert H. YOUNG, Kam-Fai WONG

Research output: Contribution to journalArticlespeer-review

32 Citations (Scopus)


Experimental results show that a word-based arithmetic coding scheme can achieve a higher compression performance for Chinese text. However, an arithmetic coding scheme is a fractional-bit compression algorithm which is known to be time consuming. In this article, we change the direction to study how to cascade the word segmentation model with a faster alternative, the integral-bit compression algorithm. It is shown that the cascaded algorithm is more suitable for practical usage. Among several word-based integral-bit compression algorithms, WLZSSHUF achieves the best compression results. Not only can it achieve a comparable compression ratio with a PPM compressor, COMP-2, it demonstrates a faster compression and decompression speed. In the last part of this article, the relation between the accuracy of the word segmentation model (match ratio) and the performance of the compression algorithm (compression ratio) are analyzed. By varying the match ratio, it was discovered that the growth rate of the compression ratio is content-dependent and close to linear. The results of our study will help the practitioners of information retrieval to design word-based compression algorithms for Chinese. This is particularly useful to multilingual digital libraries in which a massive volume of data is often involved. Copyright © 1999 John Wiley & Sons, Inc.

Original languageEnglish
Pages (from-to)218-228
JournalJournal of the American Society for Information Science
Issue number3
Early online dateFeb 1999
Publication statusPublished - 1999


Cheng, K.-S., Young, G. H., & Wong, K.-F. (1999). A study on word-based and integral-bit Chinese text compression algorithms. Journal of the American Society for Information Science, 50(3), 218-228. https://doi.org/10.1002/(SICI)1097-4571(1999)50:3%3C218::AID-ASI4%3E3.0.CO;2-1


Dive into the research topics of 'A study on word-based and integral-bit Chinese text compression algorithms'. Together they form a unique fingerprint.