This paper studies the performance of hierarchical-modulation-based image and video transmission incognitive radio systems with imperfect channel sensing results under constraints on both transmit and interference power. Data intended for transmission is first compressed via source coding techniques and then divided into two priority classes, namely high priority (HP) data and low priority (LP) data, by taking into consideration the unequal importance of bits in the output codestream. After dividing the compressed data into packets of equal size, turbo coding is applied. Finally, the resulting packets are modulated using hierarchical quadrature amplitude modulation (HQAM).
In this setting, closed-form bit error probability expressions for HP data and LP data are derived over Nakagami-m fading channels in the presence of sensing errors. Subsequently, the effects of probabilities of detection and false alarm on error rate performance of cognitive transmissions are evaluated. In addition, tradeoffs between the number of retransmissions and peak signal-to-noise ratio (PSNR) quality are analyzed numerically. Moreover, performance comparisons of multimedia transmission with conventional QAM and hierarchical QAM are carried out in terms of the received data quality and number of retransmissions.