I couldn't capture VGA but I did keep flipping input from a V3-loaded system to my LCD to check.
3dfx's old MiniGL for GLQuake/Quake2/andtherestoftheidtech2games features a software 4x1 filter which is applied to screenshots. It's a little more exaggerated than actual dac output but it's a good starting point to reverse engineer off of. SuprGrab does the same. HyperSnapDX has 4x1 and 2x2 filters though when I tried them they produced suspect results with some imprecise colors
When first starting my rewrite I replaced Quake's help pages with 256x128 images of 50% dither with black bars in between to work my HLSL shader on. I also replaced the color palette to get a RGB gradient going. I used a difference layer and shifted 1px to the right to expose what was going on
my first shader's method was very inaccurate which made me finally realize what I was doing was wrong... then I rewrote it in Dec 2014 using a method to check the next pixel for differences then add/sub it. I tried to finetune my threshhold by eyeing Voodoo3 output by switching monitor display inputs between a P2 box
it's still not right thoughhttp://www.vogons.org/viewtopic.php?p=392728#p392728
As for the shaders themselves... this has the most mature implementation
(u_CC_Brightness here is just used a variable to indicate the engine's pass number to switch the filter's direction)
There's also always a faint purple scanline effect for some reason (dithering breakup perhaps) on the 4x1 which isn't present on the 2x2. that's probably a factor in it's "superior image quality" hype etc.
and finally it's called leifx because it's my clean room approximation effort
and because i didn't use any official info (which is apparently super secret/super scarce as not even searching newsgroups help) it would be misinforming call it the
I hope this all helps in including (externally) or creating your own filter.