Another way of approaching this would be to use Plot.bin and Plot.rect. Then you can do you aggregation within the visualization itself, where x is a temporal axis (not ordinal) and y is a quantitative axis (also not ordinal, representing minutes since midnight):
This still takes a few seconds to bin the 5.6M records… I suspect that could be optimized, though. I didn’t investigate yet to see whether there is obvious room for improvement. But at least it shouldn’t be slow from the rendering side; you can adjust the thresholds to make larger bins.