It is generally recommended to mux raw streams (.h264, .h265, etc.) into containers (such as using MKVToolNix) before loading with source filters.

There are a few source filters that can be used to load videos into VapourSynth. If you have an NVIDIA GPU, DGDecNV is recommended:

src = core.dgdecodenv.DGSource("src.dgi")

.dgi index file is generated for the source video file using the DGIndexNV program in the package.

If you don't have an NVIDIA GPU, L-SMASH-Works is recommended:

src = core.lsmas.LWLibavSource("src.mkv")

ffms2 can also be be used, although certain versions add extra black frames to the beginning of video:

src = core.ffms2.Source("src.mkv")

For MPEG2 sources, d2vsource and DGMPGDec can be used.