Advanced Signal Processing Techniques for Two-Way Relaying Networks and Full-Duplex Communication Systems

To enable ultra-high data rate and ubiquitous coverage in future wireless networks, new physical layer techniques are desired. Relaying is a promising technique for future wireless networks since it can boost the coverage and can provide low cost wireless backhauling solutions, as compared to traditional wired backhauling solutions via fiber and copper. Traditional one-way relaying (OWR) techniques suffer from the spectral loss due to the half-duplex (HD) operation at the relay. On one hand, two-way relaying (TWR) allows the communication partners to transmit to and/or receive from the relay simultaneously and thus uses the spectrum more efficiently than OWR. Therefore, we study two-way relays and more specifically multi-pair/multi-user TWR systems with amplify-and-forward (AF) relays. These scenarios suffer from inter-pair or inter-user interference. To deal with the interference, advanced signal processing algorithms, in other words, spatial division multiple access (SDMA) techniques, are desired. On the other hand, if the relay is a full-duplex (FD) relay, the spectral loss due to a HD operation can also be compensated. However, in practice, a FD device is hard to realize due to the strong loop-back self-interference and the limited dynamic range at the transceiver. Thus, advanced self-interference suppression techniques should be developed. This thesis contributes to the two goals by developing optimal and/or efficient algebraic solutions for different scenarios subject to different utility functions of the system, e.g., sum rate maximization and transmit power minimization. In the first part of this thesis, we first study a multi-pair TWR network with a multi-antenna AF relay. This scenario can be also treated as the sharing of the relay and the spectrum among multiple operators assuming that different pairs of users belong to different operators. Existing approaches focus on interference suppression. We propose a projection based separation of multiple operators (ProBaSeMO) scheme, which can be easily extended when each user has multiple antennas or when different system design criteria are applied. To benchmark the ProBaSeMO scheme, we develop optimal relay transmit strategies to maximize the system sum rate, minimize the required transmit power at the relay, or maximize the minimum signal to interference plus noise ratio (SINR) of the users. Specifically for the sum rate maximization problem, gradient based methods are developed regardless whether each user has a single antenna or multiple antennas. To guarantee a worst-case polynomial time solution, we also develop a polynomial time algorithm which has been inspired by the polynomial time difference of convex functions (POTDC) method. Finally, we analyze the conditions for obtaining the sharing gain in terms of the sum rate. Then we study the sum rate maximization problem of a multi-pair TWR network with multiple single antenna AF relays and single antenna users. The resulting sum rate maximization problem, subject to a total transmit power constraint of the relays in the network, yields a similar problem structure as in the previous scenario. Therefore the optimal solution for one scenario can be used for the other. Moreover, a global optimal solution, which is based on the polyblock approach, and several suboptimal solutions, which are more computationally efficient and approximate the optimal solution, are developed when there is a total transmit power constraint of the relays in the network or each relay has its own transmit power constraint. We then shift our focus to a multi-pair TWR network with multiple multi-antenna AF relays and multiple dumb repeaters. This scenario is more general because the previous two scenarios can be seen as special realizations of this scenario. The interference management in this scenario is more challenging due to the existence of the repeaters. Interference neutralization (IN) is a solution for dealing with this kind of interference. Thereby, necessary and sufficient conditions for neutralizing the interference are derived. Moreover, a general framework to optimize different system utility functions in this network with or without IN is developed regardless whether the AF relays in the network have a total transmit power limit or individual transmit power limits. Finally, we develop the relay transmit strategy as well as base station (BS) precoding and decoding schemes for a TWR assisted multi-user MIMO (MU-MIMO) downlink channel. Compared to the multi-pair TWR network, this scenario suffers from the co-channel interference. We develop three suboptimal algorithms which are based on channel inversion, ProBaSeMO and zero-forcing dirty paper coding (ZFDPC which has a low computational complexity, provides a balance between the performance and the complexity, and suffers only a little when the system is heavily loaded, respectively. In the second part of this thesis, we investigate self-interference (SI) suppression techniques to exploit the FD gain for a point-to-point MIMO system. We first develop SI aware transmit strategies, which provide a balance between the SI suppression and the multiplexing gain of the system. To get the best performance, perfect channel state information (CSI) is needed, which is imperfect in practice. Thus, worst case transmit strategies to combat the imperfect CSI are developed, where the CSI errors are modeled deterministically and bounded by ellipsoids. In real word applications, the RF chain is imperfect. This affects the performance of the SI suppression techniques and thus results in residual SI. We develop efficient transmit beamforming techniques, which are based on the signal to leakage plus noise ratio (SLNR) criterion, to deal with the imperfections in the RF chain. All the proposed design concepts can be extended to FD OWR systems.