In this thesis, we focus on the area of segment-based non-rigid object tracking using statistical learning, spatial-temporal constraints and image processing methods. Object tracking is a potential research area and has a wide range of applications. Owning to the difficulties of modeling objects and the complexity of various scenarios and their backgrounds in the videos, the performance of existing non-rigid object tracking methods in the literature are still far from satisfaction and practical deployment. We proposed two effective segment-based non-rigid object trackers in this thesis. Rather than using a conventional bounding box, both proposed trackers are based on segments of images, which facilitate outputting segmented target object along its contour. In the first tracker, target objects and candidates are regarded as combinations of segments, where the hierarchical hue-saturation-value histograms are extracted as descriptive features. The objectness method is employed and integrated into the tracker as sampling component to generate candidates for a similarity measure. Moreover, segment-based motion weights are introduced to give higher weights to candidates with motion consistency. A confidence-collecting scheme is proposed for similar candidates. In the second tracker, the proposed method is implemented using an imbalanced learning method and motion saliency. The imbalanced learning model is learned in the initialization stage using imbalanced training set since background and foreground are often of sheer different sizes. During tracking, a similarity map is generated by measuring multiple scale segments in the frames while a motion saliency map is computed using spatial-temporal information. Two maps are fused with an adaptive scheme. The target object in video is deduced using a specific Bayesian inference framework. To validate our method, experiments were conducted using several image sequences with different non-rigid iii challenges. The experimental results show that the proposed scheme can achieve better performance than other state-of-the-art methods.