Classical models vs neural networks for tabular data
Intro
I started working with machine learning around 2018, back then it was a transition period from classical ML to neural networks. Given the problem we had to work with at the time, classical ML was better since we needed the models to be explainable to comply with regulations. At one point we hit a dead-end when trying to improve model performance, so I tried to use neural networks to see how it compares to our existing random forest regressor setup. The NN model didn't outperform our random forest regressor.
But it stuck with me - given the same dataset, how do classical ML and NN perform differently for training and inference. I got curious, so here we are.
Experiment Design
The dataset is NYC Taxi Trip. Train data is 3 million rows (sorted by time, it's a time-series dataset after all).
- Classical ML models (Random Forest, XGBoost, LightGBM) use default config.
- NN (Pytorch, Tensorflow, Jax) is a feed-forward with two hidden layers (128 and 64 units) using ReLU activations
- Followed by a single-unit linear output layer.
- Adam optimizer (lr=0.02) and MSE for loss.
- Inference is run in a single loop for 100 times, with 20 iterations for warmup.
- NN inference includes optimizations since default config is unoptimized.
Benchmarks were ran on Thinkpad X13 Gen 6
- CPU:
AMD Ryzen AI 7 PRO 350 (16) - Memory:
32 GB
SBOM
Libraries used for this experiment.
dependencies = [
"fastai>=2.8.6",
"flax>=0.12.2",
"jax>=0.9.0",
"joblib>=1.5.3",
"lightgbm>=4.6.0",
"loguru>=0.7.3",
"marimo>=0.19.4",
"mlflow>=3.8.1",
"polars>=1.37.1",
"pyspark>=4.1.1",
"scikit-learn>=1.8.0",
"seaborn>=0.13.2",
"tensorflow>=2.20.0",
"torch>=2.9.1",
"xgboost>=3.1.3",
]Click to expand
NAME VERSION TYPE
absl-py 2.3.1 python
aiofiles 25.1.0 python
alembic 1.18.1 python
annotated-doc 0.0.4 python
annotated-types 0.7.0 python
anyio 4.12.1 python
apsw 3.51.2.0 python
apswutils 0.1.2 python
astunparse 1.6.3 python
beartype 0.22.9 python
beautifulsoup4 4.14.3 python
blinker 1.9.0 python
blis 1.3.3 python
cachetools 6.2.4 python
catalogue 2.0.10 python
certifi 2026.1.4 python
cffi 2.0.0 python
charset-normalizer 3.4.4 python
chex 0.1.91 python
click 8.3.1 python
cloudpathlib 0.23.0 python
cloudpickle 3.1.2 python
colorama 0.4.6 python
confection 0.1.5 python
contourpy 1.3.3 python
cryptography 46.0.3 python
cycler 0.12.1 python
cymem 2.0.13 python
databricks-sdk 0.78.0 python
docker 7.1.0 python
docutils 0.22.4 python
etils 1.13.0 python
fastai 2.8.6 python
fastapi 0.128.0 python
fastcore 1.12.11 python
fastdownload 0.0.7 python
fastlite 0.2.4 python
fastprogress 1.1.3 python
fasttransform 0.0.2 python
filelock 3.20.3 python
flask 3.1.2 python
flask-cors 6.0.2 python
flatbuffers 25.12.19 python
flax 0.12.2 python
fonttools 4.61.1 python
fsspec 2026.1.0 python
gast 0.7.0 python
gitdb 4.0.12 python
gitpython 3.1.46 python
google-auth 2.47.0 python
google-pasta 0.2.0 python
graphene 3.4.3 python
graphql-core 3.2.7 python
graphql-relay 3.2.0 python
greenlet 3.3.0 python
grpcio 1.76.0 python
gunicorn 23.0.0 python
h11 0.16.0 python
h5py 3.15.1 python
httpcore 1.0.9 python
httptools 0.7.1 python
httpx 0.28.1 python
huey 2.6.0 python
humanize 4.15.0 python
idna 3.11 python
importlib-metadata 8.7.1 python
importlib-resources 6.5.2 python
itsdangerous 2.2.0 python
jax 0.9.0 python
jaxlib 0.9.0 python
jedi 0.19.2 python
jinja2 3.1.6 python
joblib 1.5.3 python
keras 3.13.1 python
kiwisolver 1.4.9 python
libclang 18.1.1 python
lightgbm 4.6.0 python
loguru 0.7.3 python
loro 1.10.3 python
mako 1.3.10 python
marimo 0.19.4 python
markdown 3.10 python
markdown-it-py 4.0.0 python
markupsafe 3.0.3 python
matplotlib 3.10.8 python
mdurl 0.1.2 python
ml-dtypes 0.5.4 python
mlflow 3.8.1 python
mlflow-skinny 3.8.1 python
mlflow-tracing 3.8.1 python
mpmath 1.3.0 python
msgpack 1.1.2 python
msgspec 0.20.0 python
murmurhash 1.0.15 python
namex 0.1.0 python
narwhals 2.15.0 python
nest-asyncio 1.6.0 python
networkx 3.6.1 python
numpy 2.4.1 python
nvidia-cublas-cu12 12.8.4.1 python
nvidia-cuda-cupti-cu12 12.8.90 python
nvidia-cuda-nvrtc-cu12 12.8.93 python
nvidia-cuda-runtime-cu12 12.8.90 python
nvidia-cudnn-cu12 9.10.2.21 python
nvidia-cufft-cu12 11.3.3.83 python
nvidia-cufile-cu12 1.13.1.3 python
nvidia-curand-cu12 10.3.9.90 python
nvidia-cusolver-cu12 11.7.3.90 python
nvidia-cusparse-cu12 12.5.8.93 python
nvidia-cusparselt-cu12 0.7.1 python
nvidia-nccl-cu12 2.27.5 python
nvidia-nvjitlink-cu12 12.8.93 python
nvidia-nvshmem-cu12 3.3.20 python
nvidia-nvtx-cu12 12.8.90 python
oauthlib 3.3.1 python
opentelemetry-api 1.39.1 python
opentelemetry-proto 1.39.1 python
opentelemetry-sdk 1.39.1 python
opentelemetry-semantic-conventions 0.60b1 python
opt-einsum 3.4.0 python
optax 0.2.6 python
optree 0.18.0 python
orbax-checkpoint 0.11.32 python
packaging 25.0 python
pandas 2.3.3 python
parso 0.8.5 python
pillow 12.1.0 python
pip 26.0 python
plum-dispatch 2.6.1 python
polars 1.37.1 python
polars-runtime-32 1.37.1 python
preshed 3.0.12 python
protobuf 6.33.4 python
psutil 7.2.1 python
py4j 0.10.9.9 python
pyarrow 22.0.0 python
pyasn1 0.6.2 python
pyasn1-modules 0.4.2 python
pycparser 2.23 python
pydantic 2.12.5 python
pydantic-core 2.41.5 python
pygments 2.19.2 python
pymdown-extensions 10.20 python
pyparsing 3.3.1 python
pyspark 4.1.1 python
python-dateutil 2.9.0.post0 python
python-dotenv 1.2.1 python
python-fasthtml 0.12.39 python
python-multipart 0.0.22 python
pytz 2025.2 python
pywin32 311 python
pyyaml 6.0.3 python
requests 2.32.5 python
rich 14.2.0 python
rsa 4.9.1 python
scikit-learn 1.8.0 python
scipy 1.17.0 python
seaborn 0.13.2 python
setuptools 80.9.0 python
simplejson 3.20.2 python
six 1.17.0 python
smart-open 7.5.0 python
smmap 5.0.2 python
soupsieve 2.8.3 python
spacy 3.8.11 python
spacy-legacy 3.0.12 python
spacy-loggers 1.0.5 python
sqlalchemy 2.0.45 python
sqlparse 0.5.5 python
srsly 2.5.2 python
starlette 0.50.0 python
sympy 1.14.0 python
tabular-data-model-benchmark 0.1.0 python
tensorboard 2.20.0 python
tensorboard-data-server 0.7.2 python
tensorflow 2.20.0 python
tensorstore 0.1.80 python
termcolor 3.3.0 python
thinc 8.3.10 python
threadpoolctl 3.6.0 python
tomlkit 0.14.0 python
toolz 1.1.0 python
torch 2.9.1 python
torchvision 0.24.1 python
tqdm 4.67.2 python
treescope 0.1.10 python
triton 3.5.1 python
typer-slim 0.21.1 python
typing-extensions 4.15.0 python
typing-inspection 0.4.2 python
tzdata 2025.3 python
urllib3 2.6.3 python
uvicorn 0.40.0 python
uvloop 0.22.1 python
waitress 3.0.2 python
wasabi 1.1.3 python
watchfiles 1.1.1 python
weasel 0.4.3 python
websockets 16.0 python
werkzeug 3.1.5 python
wheel 0.45.1 python
win32-setctime 1.2.0 python
wrapt 2.0.1 python
xgboost 3.1.3 python
zipp 3.23.0 pythonResults
Random Forest has the largest model size (around 18 GB). It is decorrelated - the nodes are independent (Wisdom of the Crowd paradigm).
As for other models, the size is more or less comparable - less than 1 MB. NN models tend to be smaller than classical models.
For training metrics, Random Forest is considered to be very resource-heavy compared to other models.
Pytorch takes slightly longer to train than Random Forest, but other models are very fast to train - less than 16 seconds.
Random Forest and XGBoost use the most CPU, other models use less. As for memory, the usage is considered to be very low (around 1-2 GB), except for Random Forest which consumes a lot of memory (around 20 GB)
LightGBM is fastest to train (6 seconds) and it uses very small amount of CPU and memory. For NN, Jax is fastest, with only 10 seconds for training time.
All classical models perform significantly better than NN models, even when NN models utilizes heavy optimizations. NN models performance does not come close to classical models. The performance difference is 12x worst for NN, which is A LOT.
When comparing training metrics with accuracy, we can see that LightGBM and XGBoost yield the best accuracy while using the least memory, and they don't take very long to train.
For inference, only XGBoost, LightGBM and Jax can fit within 0.5 GB of memory. Random Forest model size is around 18 GB (not pictured in the chart), which isn't economical to use for real-time inference.
If you are looking for fast inference time, LightGBM and Pytorch are good options. Jax is also ok but it has slightly higher latency. XGBoost has the highest latency.
Starting with the best model in terms of accuracy, LightGBM is a clear winner due to the least training time and very low inference latency.
Conclusion
Machine learning model development comprises two parts - model development and inference optimization. You have to account for both, otherwise you might run into a situation where your model is very fast to train, but it does not perform well during inference. The reverse can also happen.
The trick is to find the sweet spot between acceptable model training time and inference time, this way you can get the best of both worlds.
Experiment Limitations
The dataset used are considered to be uniformed. Results from a dataset with high outliers might be different.