## Abstract

In this paper, we highlight that it is inadequate to describe the rotation of the state of polarization (RSOP) in a fiber channel with the 2-parameter description model, which was mostly used in the literature. This inadequate model may result in problems in polarization demultiplexing (PolDemux) because the RSOP in a fiber channel is actually a 3-parameter issue that will influence the state of polarization (SOP) of the optical signal propagating in the fiber and is different from the 2-parameter SOP itself. Considering three examples of the 2-parameter RSOP models typically used in the literature, we provide an in-depth analysis of the reasons why the 2-parameter RSOP model cannot represent the RSOP in the fiber channel and the problems that arise for PolDemux in the coherent optical receiver. We present a 3-parameter solution for the RSOP in the fiber channel. Based on this solution, we propose a DSP tracking and equalization scheme for the fast time-varying RSOP using the extended Kalman filter (EKF). The proposed scheme is proved to be universal and can solve all the PolDemux problems based on the 2- or 3-parameter RSOP model and exhibits good performance in the time-varying RSOP scenarios.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

In recent years, the rapid development of new services, such as cloud computing, online high–definition video and virtual reality, has produced an explosive growth of the global IP data traffic. To support the ever-growing bandwidth demands, the capacity of optical fiber communication system in single wavelength channel has increased exponentially from 1 Gb/s to 100 Gb/s, even exceeding 400 Gb/s [1–3]. As a result, polarization effects play an increasingly important role (either a positive role or a negative role) in an optical communication system. For the positive role, the polarization can be used as a physical dimension in the polarization division multiplexing (PDM) optical fiber system to double the spectral efficiency. In contrast, the negative impact, which should not be ignored, is that polarization-related impairments will lead to the degradation of the system performance. The three main polarization impairment effects in an optical fiber communication system are the rotation of state of polarization (RSOP), polarization mode dispersion (PMD) and polarization dependent loss (PDL) [4,5], all of which are required to be equalized at the receiver end. Among these effects, the RSOP is essential to a PDM system for the requirement of polarization demultiplexing (PolDemux). Therefore, in this paper, we focus on the RSOP modelling for fiber channel, and the equalization issues related to RSOP. The main trouble of RSOP for the PDM system is its high variation speed that imposes a heavier burden on the ordinary MIMO (multiple input and multiple output) equalization algorithm to keep pace. Indeed, there are some extreme scenarios such as vibration sources nearby and lightning strike, in which the speed of RSOP will be up to 5.1 Mrad/s for an optical link [6–8]. It is essential for us to make a deep analysis about the appropriate RSOP modelling for fiber channel, and hence the fast response and effective RSOP equalization schemes, in order for us to cope with the ultra-fast RSOP variation in the extreme environments.

In the literature, in Jones space the RSOP impairment is modeled as 2 × 2 matrix *J* = (*j _{xx}*,

*j*;

_{xy}*j*,

_{yx}*j*), mostly with only 2 independent parameters [9–14]. We find that it comes from the confusion or misunderstanding regarding the concept of SOP and RSOP. The SOP refers to the state of polarization of optical signals itself in a fiber and requires only 2 parameters for its normalized vector representation. Alternatively, the RSOP refers to a polarization effect that causes the rotation of a signal’s SOP in a fiber channel and requires 3 parameters for its matrix representation. This paper will answer why an RSOP modelling should be with 3 independent parameters, and show the mostly used 2-parameter RSOP models are the special cases of 3-parameter model.

_{yy}As for the RSOP equalization scheme, the widely used schemes are MIMO equalization algorithms, such as constant modulus algorithm (CMA). It can equalize the RSOP impairment to some extent and have some limited tolerance to other impairments, such us residual chromatic dispersion, PMD and PDL [15]. However, the performance of CMA will decline dramatically or even be completely insufficient when the speed of RSOP exceeds several hundreds of kilo-rad per second.

On the other hand, recently, an efficient method of PolDemux in Stokes space has been proposed that has attracted widespread attention [9–14]. The operation principle of this type of PolDemux technique is based on the calculation of the Stokes parameters of the received signal samples in Stokes space to obtain a best fitting plane, whose normal vector identifies the two orthogonal polarization states at the receiver. The normal vector is forced to its original direction to achieve polarization demultiplexing [9]. The main advantage of this approach is that it does not require pre-identification of the modulation format transmitted and has a high convergence speed. In particular, Stokes space-based geometric approach proposed by [11] is robust and compatible to time-varying RSOP.

Nevertheless, among these Stokes-based PoDemux algorithms, as mentioned above, the confusion regarding the concept of SOP and RSOP will result in problems for the implementation of RSOP equalization. We can, for example, find this misunderstanding or confusion regarding this discrepancy between the SOP and the RSOP in [9–14]. The RSOP fiber channel models adopted are Eq. (9) in [9] and Eqs. (7)-(9) in [11], all of which include only 2 parameters for the RSOP matrix representation. This confusion between the 2-parameter SOP and the 3-parameter RSOP leads to problems for PolDemux in Stokes space; addressing this confusion is the motivation of this paper.

In this paper, first, we will highlight the reasons why those previous works used a 2-parameter RSOP for a fiber channel and the resulting issues in following that approach. Second, we propose a 3-parameter solution for this issue in PolDemux in Stokes space combined with the extended Kalman filter. Finally, numerical verifications are demonstrated.

## 2. RSOP model in a fiber channel

#### 2.1 SOP representation with 2-parameters

In [9,11–14], nearly all the RSOP models for fiber channel are established based on the idea that, through the RSOP transformation in a fiber channel, a pair of orthogonal x and y linearly polarized states can be transformed into an arbitrary pair of orthogonal elliptically polarized states. And hence these transformations were regarded as the RSOP models for a fiber channel. The x-polarized and y-polarized states that are orthogonal to each other can be represented as

According to Fig. 1 we can also use the principal *ξ*-*η* coordinate system using another pair of parameters $\theta $ and$\beta $to describe an arbitrary elliptical polarization state, which is denoted as $|E\left(\theta ,\beta \right)\u3009$ [18]. Here, $\theta $ denotes the orientation angle or azimuth angle, and $\beta $ denotes the ellipticity angle with the relation$\mathrm{tan}\beta ={B}_{\eta}/{B}_{\xi}$. Using $\theta $and$\beta $, the arbitrary two orthogonal elliptical polarization states are

_{$|E\left(\alpha ,\delta \right)\u3009$} or $|E\left(\theta ,\beta \right)\u3009$ can be mapped to Stokes space either on the observable polarization sphere (using${S}_{2}\text{-}{S}_{3}\text{-}{S}_{1}$coordinate system) or on the Poincaré sphere (using ${S}_{1}\text{-}{S}_{2}\text{-}{S}_{3}$coordinate system) [16], as shown in Fig. 2, and with the relations given in Eq. (5).

We note that, in Stokes space, whenever we represent a state of polarization (SOP) using the observable polarization sphere or the Poincaré sphere, two parameters are adequate to describe an SOP. We can conclude that an SOP requires 2-parameter representation. Under the idea that the$J$matrix of the RSOP model for a fiber channel should satisfy$|{E}_{1}\left(\alpha ,\delta \right)\u3009=J|x\text{-polarized}\u3009$or $|{E}_{1}\left(\theta ,\beta \right)\u3009=J|x\text{-polarized}\u3009$, can one conclude from the above reasoning that an RSOP$J$matrix must also be a 2-parameter representation? Before we answer this question, we first consider some examples of the forms in which the$J$matrix can take.

#### 2.2 The RSOP model for a fiber channel adopted in previous works

### 2.2.1 $(\alpha ,\delta )$-two-step RSOP model

According to Fig. 3, we can establish an RSOP *J* matrix through a two-step process using parameters$(\alpha ,\delta )$. The first step: an x-linearly polarized SOP for the input signal is converted into *α*-linearly polarized SOP by *α* angle rotation through a rotation matrix. The second step: the *α*-linearly polarized SOP (has an outer frame with lengths of $2\mathrm{cos}\alpha $ and $2\mathrm{sin}\alpha $ with zero phase difference between its x and y components) is converted into an arbitrary elliptical SOP by introducing a phase retarder, as shown in Fig. 3. For the reason that we take two steps to obtain the final representation of polarization rotation, we call this RSOP model an $(\alpha ,\delta )$-two-step RSOP model, as shown in Eq. (6); this model can be found in [9,11–13].

### 2.2.2 $(\alpha ,\delta )$-one-step RSOP model

Following Fig. 4, we can also establish a RSOP *J* matrix through a one-step process using parameters$(\alpha ,\delta )$. This time, we find that any pair of orthogonal elliptical SOPs ${\left(\mathrm{cos}\alpha \text{,}\mathrm{sin}\alpha {e}^{j\delta}\right)}^{T}$ and ${\left(-\mathrm{sin}\alpha {e}^{j\delta},\mathrm{cos}\alpha \right)}^{T}$ in Eq. (2) can be the columns of transformation matrix in Eq. (7) such that the horizontal and vertical linearly SOPs ${\left(1,0\right)}^{T}$ and ${\left(0,1\right)}^{T}$ can be transformed into above arbitrary pair of orthogonal elliptical SOPs. Because the process uses a one-step process, we call this model the $(\alpha ,\delta )$-one-step RSOP model, with the result that

This model can be found in [11,12,14]. Two parameters$\alpha $and$\delta $in Eqs. (6)-(7) correspond to the angles of$2\alpha $and$\delta $in the observable polarization sphere [16], as shown in Fig. 2(a).

### 2.2.3 $(\theta ,\beta )$-RSOP model

According to Fig. 5, we can establish an RSOP$J$ matrix through a two-step process using parameters$(\theta ,\beta )$. The first step: an x-linearly polarized SOP for the input signal is converted into a right-handed zero angle orientation elliptical SOP ${\left(\mathrm{cos}\beta ,j\mathrm{sin}\beta \right)}^{T}$ through a transformation matrix $\left(\begin{array}{cc}\mathrm{cos}\beta & j\mathrm{sin}\beta \\ j\mathrm{sin}\beta & \mathrm{cos}\beta \end{array}\right)$ where $2\mathrm{cos}\beta $ and $2\mathrm{sin}\beta $ denote the outer frame of the SOP and *j* denotes $\pi /2$ phase difference between its x and y components. The second step: the zero angle orientation ellipse is transformed to a $\theta $angle orientation ellipse through a rotation matrix. The RSOP $J$ matrix of this model has the following form:

This model can be found in [18]. For this $(\theta ,\beta )$ RSOP model, two parameters correspond to the angles of $2\theta $and $2\beta $in Poincaré sphere, as shown in Fig. 2(b).

#### 2.3 Why should an RSOP model be a 3-parameter representation for a fiber channel?

As we know, for an optical fiber, if we do not take PDL into account, the Jones transformation matrix representing polarization effect should be unitary matrix $U$, whose Cayley-Klein form is [4]

From another point of view, in Stokes space, any SOP transformation through a fiber without PDL can be considered as a rotation of the SOP Stokes vector with the angle $\phi $around an axis denoted by a unit vector $\widehat{r}$. The corresponding rotation 3 × 3 matrix in Stokes space has the counterpart 2 × 2 Jones matrix in Jones space given in Eq. (10) [19].

*φ*around $\widehat{r}$ is another independent parameter. Therefore, an RSOP representation in the fiber channel requires three independent parameters.

In fact [20], had proved analytically and experimentally that at least 3 instead of 2 degrees of freedom for a polarization controller (PC) are needed to achieve the goal of SOP transformation from any input SOP into any other output SOP in Stokes space, which can be regarded as equivalent SOP transformation in a fiber channel. Indeed [20] proved that there are some special input SOPs that cannot be transformed into other arbitrary output SOPs on the Poincaré sphere using the PC with only 2 degrees of freedom. Therefore, we can conclude that an RSOP model matrix should be a 3-parameter representation other than 2-parameter representation.

#### 2.4 The problem induced by the 2-parameter RSOP representation

### 2.4.1 RSOP effects on optical signals with different 2-parameter RSOP models

In this paper, we only focus on the RSOP for the fiber polarization effect that does not include PMD and PDL. If we let $|{r}_{Tx}\u3009$,$|{r}_{Rx}\u3009$ and $J$ be the dual-polarization signal vectors (2 × 1) at the transmitter, receiver, and the RSOP rotations matrix (2 × 2), then

corresponds to the signal distortion procedure from transmitter to receiver through fiber channel.To evaluate the effects of different 2-parameter RSOP models, as mentioned in 2.2, on the optical signals propagating in fiber channel, we let the signals suffer from the three different RSOP matrices mentioned above in Eqs. (6)-(8). The evaluations and comparisons are implemented both in Stokes space and in constellation space in Fig. 6, with the static RSOPs shown in Figs. 6(a)-6(c), and the time-varying RSOPs shown in Figs. 6(d)-6(f), respectively. In all the cases, the start SOPs at the transmitter end are dual-polarization with$|x\text{-polarized}\u3009$ and $|y\text{-polarized}\u3009$ QPSK signals, and in Stokes space, all the QPSK signal samples are located in the ${S}_{2}\text{-}{S}_{3}$ plane with its fitting plane’s normal vector pointing along the ${S}_{1}$ axis [9]. We can see in Figs. 6(a)-6(c) that when a QPSK signal experience the RSOPs described by Eqs. (6)-(8) (with angles$\{\alpha =-pi/6;\text{\hspace{0.17em}}\delta =-pi/3\}$and the corresponding set $\{\theta =\mathrm{arctan}(-\sqrt{3}/2)/2;$$\{\beta =\mathrm{arcsin}(4/3)/2\}$ according to Eq. (5)), although the fitting plane of the signal constellation samples in Stokes space has the same orientation, and the normal vector is $\widehat{n}={\left[0.498-\text{0}\text{.432}\text{0}\text{.751}\right]}^{T}$(left pictures in Figs. 6(a)-6(c)), but the constellation points in constellation space show different patterns (right pictures in Figs. 6(a)-6(c)). For the cases in Figs. 6(d)-6(f), one parameter is fixed ($\alpha $ in the $(\alpha ,\delta )$ model, and $\theta $ in the $\left(\theta ,\beta \right)$ model), and another is linearly increased with time ($\delta $in the $(\alpha ,\delta )$ model, $\beta $in the $\left(\theta ,\beta \right)$ model). We can see in Figs. 6(d)-6(f) that the time-varying trajectories of both signal constellation samples in Stokes space, and constellation points in the constellation plane are different one another. Therefore, from this point of view, a 2-parameter representation is insufficient for describing the RSOP in the fiber channel because different models based on Eqs. (6)-(8) have different impacts on the optical signals propagating in the fiber.

### 2.4.2 RSOP compensation based on the 2-parameter representation

The signal equalization procedure (assuming perfect equalization) can be expressed as follows:

where $|{r}_{Eq}\u3009$ represents the signal after equalization. The task for equalization is to monitor or find the exact form of $J$ matrix and make a transformation using${J}^{-1}$.The following question arises: if the 2-parameter RSOP representations mentioned in Section 2.2 is acceptable, can following equations be satisfied simultaneously?

In the literature, the authors of [9-10] have proposed and developed a PolDemux technique based on Stokes space. The idea behind the PolDemux technique is that, for a PDM-M-QAM modulation format, if there is no RSOP impairment, then its corresponding constellation samples in Stokes space constitute a lens-like disk whose symmetrical plane (or fitting plane) lies in the ${S}_{2}\text{-}{S}_{3}$plane, with its normal vectors pointing along the $\text{+}{S}_{1}$and$-{S}_{1}$ directions. The two ends of the normal vectors $\overrightarrow{H}\text{=}{(1,0,0)}^{{\rm T}}$and $\overrightarrow{V}={\text{(}-\text{1,0,0)}}^{{\rm T}}$represent the horizontal polarization state (H-state) and the vertical polarization state (V-state), respectively. In particular, for the PDM-QPSK format, its constellation samples are located at ${(0,\pm 1,0)}^{{\rm T}}$and ${(0,0,\pm 1)}^{{\rm T}}$. An RSOP makes the orientation (also the normal vectors) of the plane deviate from the ${S}_{2}\text{-}{S}_{3}$ plane. As a result, the normal vectors become ${\overrightarrow{G}}_{1}\text{=}{(a,b,c)}^{{\rm T}}$ and ${\overrightarrow{G}}_{2}\text{=}{(-a,-b,-c)}^{{\rm T}}$. Therefore, the PolDemux technique based on Stokes space involves forcing this plane to go back to the ${S}_{2}\text{-}{S}_{3}$ plane or its normal vectors to go back to $\overrightarrow{H}$ and $\overrightarrow{V}$ to achieve polarization demultiplexing of the optical signals.

To implement the PolDemux method based on Stokes space, we must (i) find the best fitting plane and its normal vectors ${\overrightarrow{G}}_{1}$ and ${\overrightarrow{G}}_{2}$ in Stokes space and obtain the 2-parameters $\alpha $ and $\delta $ (or $\theta $ and $\beta $) by

However, what if we conduct the equalization according to Eq. (13)? Equation (13) indicates that, if the 2-parameter representation for the RSOP in the fiber channel is acceptable, then all the RSOP model forms are equivalent, in particular, any equalization transformation matrix ${J}_{j}^{-1}$ can recover the RSOP impairments induced by any other RSOP models${J}_{i}$,when $j\ne i$.

To evaluate the question of equalization according to Eq. (13), we performed a simulation for a PDM-QPSK coherent optical communication system. The RSOP model we chose is the $(\alpha ,\delta )$-one-step RSOP model represented by ${J}_{2}(\alpha ,\delta )$:$\{\alpha =-pi/6;\text{\hspace{0.17em}}\delta =-pi/3\}$. We used two equalization transformation matrices ${J}_{1}^{-1}(\alpha ,\delta )$ and ${J}_{3}^{-1}(\theta ,\beta )$, and checked the results. The results are shown in Fig. 7.

Figure 7(a) shows the constellation samples in Stokes space under the situation of no RSOP impairment. We can see that the 4 constellation samples are located at ${(0,\pm 1,0)}^{{\rm T}}$and ${(0,0,\pm 1)}^{{\rm T}}$, constituting a plane lying in the ${S}_{2}\text{-}{S}_{3}$plane. Figure 7(b) shows that the RSOP ${J}_{2}(\alpha ,\delta )$$\{\alpha =-pi/6;\delta =-pi/3\}$ makes the plane deviate from the ${S}_{2}\text{-}{S}_{3}$ plane. Figure 7(c)-7(d) are the results when we perform the equalizations using ${J}_{1}^{-1}(\alpha ,\delta )$ and ${J}_{3}^{-1}(\theta ,\beta )$. We can see that, after equalization, although the constellation sample planes all return to the ${S}_{2}\text{-}{S}_{3}$plane, and their normal vectors return to $\overrightarrow{H}$ and $\overrightarrow{V}$, the 4 constellation samples shift from${(0,\pm 1,0)}^{{\rm T}}$ and ${(0,0,\pm 1)}^{{\rm T}}$, which means the impairment induced by the RSOP is not recovered completely.

The phenomena shown in Fig. 7 and the above discussion make us realize that the 2-parameter RSOP models are not sufficient and not universal; they are only some special cases of the 3-parameter representation of the RSOP. Therefore, the equalization matrix we choose must be a form with 3 independent parameters. We cannot obtain these parameters from the information in Stokes space because the number of independent parameters in Stokes space is only 2 ($(\alpha ,\delta )$ or $(\theta ,\beta )$). In fact, for a spherical coordinate system, only two angles are independent. The previous equalization schemes [9,10] by step (i) to (iii) mentioned above are incomplete.

## 3. A solution scheme

In this section, we propose a PolDemux scheme or a RSOP equalization scheme based on Stokes space and the Kalman filter. The important key issues using the Kalman filter are: (i) choose the appropriate state parameters to be monitored by the Kalman Filter; (ii) perform the correct equalization using the right transformation matrix; (iii) adopt the right measurement space to measure the measurement innovation or residual; (iv) make the appropriate initialization.

We first focus on the issues in (i) to (iii). In [12], which is a typical PolDemux scheme using both Stokes space and the Kalman filter, the authors choose the state vector as ${\overrightarrow{G}}_{1}$,${\overrightarrow{x}}_{k}={\left({a}_{k},{b}_{k},{c}_{k}\right)}^{T}$ defined in Stokes space, where only 2 parameters are independent because ${a}_{k}{s}_{1,k}+{b}_{k}{s}_{2,k}+{c}_{k}{s}_{3,k}=0$ (constitute a plane) and ${a}_{k}^{2}+{b}_{k}^{2}+{c}_{k}^{2}=1$ (unit vector). They performed equalization with ${J}_{1}{}^{-1}\left(\alpha ,\delta \right)$ in Jones space. They adopted Stokes space as the measurement space in which the measurement innovation is

We can see that, for issue (i), Stokes space can only offer 2 parameter information, and ${J}_{1}{}^{-1}\left(\alpha ,\delta \right)$ cannot cover the universal RSOP models.

Next, we describe our RSOP equalization scheme. We start with issue (ii). According to the discussions above, the equalization transformation matrix should be based on 3 parameters and unitary and can take the following form [4]:

The Kalman filter possesses flexibilities in choosing the spaces adopted for the state vector, the equalization and the measurement. As mentioned above [12], chose Stokes space for state vector ${\left(\alpha ,\delta \right)}^{T}$, the Jones space for the equalization (process as in Eq. (12)) and Stokes space again for the measurement (process as in Eq. (14)). In our proposed scheme, we choose the state vector in the Jones space as

We also choose the equalization in the Jones space as in Eq. (18), and the measurement space as Stokes space with measurement vector ${\overrightarrow{z}}_{k}$ and innovation vector ${\overrightarrow{e}}_{k}$ asIn this paper, because $\overrightarrow{h}k$ is nonlinear with the state vector ${\overrightarrow{x}}_{k}$, the extended Kalman filter (EKF) is utilized. The main estimation formulas of EKF are given as follows [10,21,22]:

*a priori*estimate, $\widehat{\overrightarrow{x}}k$is the

*a posteriori*state estimate, ${P}_{k}$ denotes the error covariance matrix, and $H$ is the Jacobi matrix of measurement equation expanded at prior estimation point ${\widehat{\overrightarrow{x}}}_{k}^{-}$. $Gk$is the Kalman gain matrix. The matrices $Q$and $R$describe the state covariance matrix and the measurement covariance matrix, respectively.

## 4. Simulation results

As a means to verify the theory described above, we performed numerical simulations in the PDM-QPSK coherent optical transmission system at 28 GBaud. The time-varying RSOP models were numerically emulated by four different matrices of ${J}_{1}(\alpha ,\delta )$, ${J}_{2}(\alpha ,\delta )$, ${J}_{3}(\theta ,\beta )$, and $U\left(\xi ,\eta ,\kappa \right)$mentioned above in Eqs. (6)-(8) and (17) under the OSNR of 15 dB. As shown in Fig. 8, in the receiver, the DSP mainly includes modules of resampling and normalization, orthogonalization, CD compensation, PolDemux, recovery of frequency offset and phase noise, decision, and BER calculation. In order to focus on the equalization of RSOP, we assume all other impairments such as CD are already compensated in previous equalization parts in DSP. PDL and PMD are also not taken into account in this paper. The frequency offset between the transmitter and receiver lasers is set as 500MHz, and the linewidths of them are all set as 100kHz. The simulation in this paper is mainly involved in the brown shaded parts of DSP. In the PolDemux module, the proposed Kalman scheme based on ${U}_{Eq}\left(\xi ,\eta ,\kappa \right)$, CMA and the Stokes method are implemented separately for comparison. We exploit the fourth-power method for the frequency offset estimation and the Viterbi-Viterbi phase equalizer (VVPE) for the phase noise recovery. For the Kalman scheme based on ${U}_{Eq}\left(\xi ,\eta ,\kappa \right)$, some parameters must be initialized. The state covariance matrix **Q** and the measurement covariance matrix **R** giving the optimum performance are set to be$Q=\text{diag}\left({10}^{-\text{5}},{10}^{-\text{5}},{10}^{-\text{5}}\right)$and $R=\text{diag}\left(\text{200},\text{200}\right)$.

Firstly, we have simulated the PolDemux performances respectively using the proposed 3-parameter ${U}_{Eq}\left(\xi ,\eta ,\kappa \right)$ based EKF method, CMA, and the Stokes space based algorithms to compensate for the signal distortions induced by RSOP models ${J}_{1}(\alpha ,\delta )$,${J}_{2}(\alpha ,\delta )$, ${J}_{3}(\theta ,\beta )$ and $U\left(\xi ,\eta ,\kappa \right)$, respectively. The results are shown in Fig. 9(a), 9(b), 9(c), and 9(d). We take the algorithm chosen in [12] to represent the Stokes space based algorithms in [9–14]. Actually, the equalization matrix in [12] is the inverse matrix of $(\alpha ,\delta )$-two-step RSOP model ${J}_{1}^{-1}(\alpha ,\delta )$. The step sizes of CMA is set as 5e-4, the filter lengths of CMA is set as 11. We can see in Fig. 9 that the proposed 3-parameter based method performs best and behaves universal to all the RSOP models mentioned above. It has the RSOP variation speed tolerance more than 130 Mrad/s. We can also find that the Stokes space based algorithm behaves better in the cases of ${J}_{1}(\alpha ,\delta )$ and ${J}_{2}(\alpha ,\delta )$ models, and performs worse in the cases of ${J}_{3}(\theta ,\beta )$ and $U\left(\xi ,\eta ,\kappa \right)$ models. The reason is that the parameters $(\alpha ,\delta )$ chosen in Stokes space based algorithm are the same as in models of ${J}_{1}(\alpha ,\delta )$ and ${J}_{2}(\alpha ,\delta )$, and different from those in models of ${J}_{3}(\theta ,\beta )$ and $U\left(\xi ,\eta ,\kappa \right)$.

Then we make the comparison of the convergence performance of the proposed method, CMA, and the Stokes space based method at the RSOP of 1Mrad/s. For the proposed method and the Stokes method, the convergence curves of the estimated parameters $(\alpha ,\delta )$are given. For CMA, the absolute values of the first row elements |*h*_{11}| and |*h*_{12}| of the equalization matrix are tracked. The results are shown in Fig. 10. It can be seen that the proposed method and the Stokes method provides a quicker convergence rate. The tracked parameters $(\alpha ,\delta )$ achieve their convergence within just tens of samples, which represents an important advantage over the CMA algorithm.

## 5. Conclusions

In this paper, we found that the 2-parameter representation models are inadequate to describe the RSOP in a fiber channel. The 2-parameter SOP is quite different from the 3-parameter RSOP, and the description of the RSOP with 3 parameters is required. The mistaken use of 2-parameter RSOP instead of the 3-parameter RSOP leads to problems for PolDemux or RSOP equalization. It is proved analytically that the 2-parameter RSOP representations that used in the literature are actually the special cases of the 3-parameter RSOP representation used in this paper. Focused on the problems in RSOP equalization, we proposed a solution scheme using a 3-parameter RSOP representation and a Kalman filter. The proposed scheme was implemented and verified in the 28 Gbaud PDM-QPSK optical fiber coherent system. The numerical simulation results show that the proposed scheme based on the 3-parameter RSOP model is universal and independent of the RSOP models used in the literature. The RSOP tracking speed of the proposed scheme can be as high as approximately 180, 170, 160 and 130 Mrad/s for the four models considered in this paper.

## Funding

National Natural Science Foundation of China (61571057, 61527820, 61575082); Huawei Technology Project (YBN2017030025); Open Fund of the Guangdong Provincial Key Laboratory of Optical Fiber Sensing and Communications (Jinan University); State Grid Corporation of China (5101/2017-3205A).

## References

**1. **M. Birk, P. Gerard, R. Curto, L. Nelson, X. Zhou, P. Magill, T. Schmidt, C. Malouin, B. Zhang, E. Ibragimov, S. Khatana, M. Glavanovic, R. Lofland, R. Marcoccia, R. Saunders, G. Nicholl, M. Nowell, and F. Forghieri, “Real-time single-carrier coherent 100 Gb/s PM-QPSK field trial,” J. Lightwave Technol. **29**(4), 417–425 (2011). [CrossRef]

**2. **P. Winzer, “High-spectral-efficiency optical modulation formats,” J. Lightwave Technol. **30**(24), 3824–3835 (2012). [CrossRef]

**3. **G. Raybon, “High symbol rate transmission systems for data rates from 400 Gb/s to 1Tb/s,” in *Optical Fiber Communication Conference*, OSA Technical Digest (online) (Optical Society of America, 2015), paper M3G.1. [CrossRef]

**4. **J. N. Damask, *Polarization Optics in Telecommunications* (Springer,2005).

**5. **Y. Feng, L. Li, J. Lin, H. Xu, W. Zhang, X. Tang, L. Xi, and X. Zhang, “Joint tracking and equalization scheme for multi-polarization effects in coherent optical communication systems,” Opt. Express **24**(22), 25491–25501 (2016). [CrossRef] [PubMed]

**6. **M. Kuscherov and M. Herrmann, “Lightning affects coherent optical transmission in aerial fiber,” (lightwave, 2016), http://www.lightwaveonline.com/articles/2016/03/lightning-affects-coherent-optical-transmission-in-aerial-fiber.html.

**7. **H. Yaffe, “Are ultrafast SOP events affecting your coherent receivers?” https://newridgetech.com/are-ultrafast-sop-events-affecting-your-receivers.

**8. **D. Charlton, S. Clarke, D. Doucet, M. O’Sullivan, D. L. Peterson, D. Wilson, G. Wellbrock, and M. Bélanger, “Field measurements of SOP transients in OPGW, with time and location correlation to lightning strikes,” Opt. Express **25**(9), 9689–9696 (2017). [CrossRef] [PubMed]

**9. **B. Szafraniec, B. Nebendahl, and T. Marshall, “Polarization demultiplexing in Stokes space,” Opt. Express **18**(17), 17928–17939 (2010). [CrossRef] [PubMed]

**10. **B. Szafraniec, T. Marshall, and B. Nebendahl, “Performance monitoring and measurement techniques for coherent optical systems,” J. Lightwave Technol. **31**(4), 648–663 (2013). [CrossRef]

**11. **N. Muga and A. Pinto, “Adaptive 3-D stokes space-based polarization demultiplexing algorithm,” J. Lightwave Technol. **32**(19), 3290–3298 (2014). [CrossRef]

**12. **N. Muga and A. Pinto, “Extended Kalman filter vs. geometrical approach for Stokes space-based polarization demultiplexing,” J. Lightwave Technol. **33**(23), 4826–4833 (2015). [CrossRef]

**13. **Z. Yu, X. Yi, Q. Yang, M. Luo, J. Zhang, L. Chen, and K. Qiu, “Polarization demultiplexing in stokes space for coherent optical PDM-OFDM,” Opt. Express **21**(3), 3885–3890 (2013). [CrossRef] [PubMed]

**14. **Z. Yu, X. Yi, J. Zhang, D. Zhao, and K. Qiu, “Experimental demonstration of polarization-dependent loss monitoring and compensation in Stokes space for coherent optical PDM-OFDM,” J. Lightwave Technol. **32**(23), 3926–3931 (2014).

**15. **S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Express **16**(2), 804–817 (2008). [CrossRef] [PubMed]

**16. **E. Collett, *Polarized Light in Fiber Optics* (SPIE,2003).

**17. **A. Kumar and A. Ghatak, *Polarization of Light with Applications in Optical Fibers* (SPIE,2011).

**18. **R. Noé, *Essentials of Modern Optical Fiber Communication* (Springer,2016).

**19. **J. P. Gordon and H. Kogelnik, “PMD fundamentals: Polarization mode dispersion in optical fibers,” Proc. Natl. Acad. Sci. U.S.A. **97**(9), 4541–4550 (2000). [CrossRef] [PubMed]

**20. **X. Zhang and Y. Zheng, “The number of least degrees of freedom required for a polarization controller to transform any state of polarization to any other output covering the entire Poincaré sphere,” Chin. Phys. B **17**(7), 2509–2513 (2008). [CrossRef]

**21. **S. S. Haykin, *Adaptive Filter Theory, *4th ed. (Pearson Education India, 2008).

**22. **H. Xu, X. Zhang, X. Tang, C. Bai, L. Xi, W. Zhang, and H. Zheng, “Joint scheme of dynamic polarization demultiplexing and PMD compensation up to second-order for flexible receivers,” IEEE Photonics J. **9**(6), 7204615 (2017). [CrossRef]