Be sure to take a look at the d3-zoom docs. There you’ll find an example to reset a zoom transform, which you need to apply before you remove the handlers so that your SVG can still update:
} else {
svgOrigin.call(zoom.transform, d3.zoomIdentity);
// Remove all *.zoom event handlers
svgOrigin.on(".zoom", null);
}