Due to the high robustness against various image transformations, Scale Invariant Feature Transform (SIFT) has been widely employed in many computer vision and multimedia security areas to extract image local features. Though SIFT has been extensively studied from various perspectives, its security against malicious attack has rarely been addressed. In this work, we demonstrate that the SIFT keypoints can be effectively removed, without introducing serious distortion on the image. This is achieved by formulating the SIFT keypoint removal as a constrained optimization problem, where the constraints are well-designed to suppress the existence of local extrema and prevent generating new keypoints within a local cuboid in the scale space. We show that such optimization problem in the ideal case is unfortunately non-convex. To make the computation feasible, we propose a relaxation technique to convexify the original problem, while maximally preserving the solution space. As demonstrated experimentally, our proposed SIFT removal algorithm significantly outperforms the state-of-the-arts in terms of keypoint removal rate-distortion (KRR-D) performance. Finally, a case study of malfunctioning a copy-move forgery detection is provided to further demonstrate the effectiveness of our scheme. Our results imply that an authorization mechanism is required for SIFT-based systems to verify the validity of the input data, so as to achieve high reliability.