Hi there Simon,
With enough elbow grease, you can do a lot with D3 - even recreate much of what Leaflet does. There are a few examples of zooming & panning with tiles (d3.tile), and a few years back I worked on one of the examples of d3 as a full-fledged map browsing & editing client (iD).
The most natural breaking-off point for D3 versus something like Leaflet, Google Maps or Mapbox, is world-scale data. If you need to zoom into the street level and show details anywhere, then you’ll need to load additional data dynamically, and for that you’ll need the things Leaflet and similar map clients have pre-built: tile loading, unloading, figuring out URLs, and so on. They’ll also bring things like pre-built zoom & pan controls, the ability to store the map’s centerpoint using a hash in the URL, and a pretty wide range of ‘other stuff’.
The other big difference is how things are rendered: if you’re planning on showing a lot of data on a map - thousands of markers, lots of text, raster data, and so on - there are very few ways to beat the performance of something like Mapbox GL, which has many person-years spent optimizing how it carefully loads data and uses WebGL, instead of SVG or Canvas, to display it. (disclosure: previous employer!)
The other advantage of a map-specific library is that they all (Leaflet, Google, Mapbox, OpenLayers, etc) tend to implement a certain set of interactions that people have grown used to: double-click zooming, double-right-click zoom out, scroll zoom with fixed centerpoint, etc. There are no technical reasons why you can’t recreate all these behaviors in D3, but it might devolve into a sort of whack-a-mole to get everything your stakeholders think is ‘standard’ implemented. Plus, you have a bunch of extremely performant clustering algorithms right off the open-source shelf, like supercluster and MarkerClusterer
But, there’s plenty of praises to sing about the D3 approach. First of all, it tends to be self-contained: mapping frameworks tend to rely on some company or organization, and that means relying on them for reliability, and potentially forking over some money, on an ongoing basis. D3 on the other hand, you can typically set up to just load data locally, without any additional infrastructure.
And if you’re already building other visualizations on the page, it’s great to handle both geo- and non-geo- visualizations with the same toolkit, so you can share things like color scales for those choropleth maps - and if your project is inherited by a D3-wizard developer, then they’ll have one less tool to learn.
And - one final thing - D3 handles geographical projects like a pro, whereas typical mapping software is really built around Web Mercator (the projection of Google Maps) first. It’s possible to use most of the map-specific tools with other projections, but at this point in time, it’s typically inconvenient. So if your maps are in Albers, or some other projection, and really need to be, D3 is likely a good choice.
Hope that’s helpful! There are a lot of points in each column, but neither’s a bad choice - it’s just a matter of what you want to build 
-Tom
