No the CSV does not work like a check valve. The check valve on the pump is leaking back and needs to be replaced. You probably just have the wrong CSV. The standard CSV2W1.25 is made for 3-5 HP pumps and has a minimum flow of 5 GPM. Your house probably never uses more than 5 GPM, so it is like the CSV is not even there. A CSV1A would never let the pump shut off when using more than 1 GPM, while the CSV2W you have won't even work unless you use more than 5 GPM, which is why the pump is cycling.
However, we do make a 1 GPM seat for the CSV2W, and we could easily swap it out for you so that valve would work. And I am just assuming here, what size pump do you have?