Python virtual environments explained for data analysts

Most analysts meet virtual environments the hard way. A colleague sends over a notebook, you run the first cell, and you get a wall of red text about incompatible versions of numpy. Nothing in the notebook is wrong. The trouble is that every package you install goes into the same Python installation by default, shared with every other project you have ever touched. An isolated environment fixes that in about thirty seconds of typing.

Why dependency conflicts happen

Python installs packages into a global site-packages folder attached to a single interpreter. Install pandas 1.5 for one piece of work and pandas 2.2 for another, and whichever went in last wins — for both projects.

Version constraints make this worse. One package needs numpy below 1.24; another was built against 1.26. Both cannot be satisfied in one folder. You will not always get a loud error, either. Sometimes a function returns subtly different results: a changed data type, a different default for missing values, a column order that shifts. Those are the expensive incidents, because nothing looks broken.

Upgrades carry the same risk. Running pip install --upgrade pandas to fix one script can quietly change the behaviour of three others. With a single shared installation, every update is an update to everything.

What a virtual environment actually is

A virtual environment is a folder. Inside it sits a link to (or a copy of) a Python interpreter, its own site-packages directory, and its own record of installed versions. Activate it and the python and pip commands in your terminal point at that folder instead of the system-wide installation.

That is the whole trick. It is not a container and not a virtual machine, and it does not duplicate Python in any meaningful way. Environments are small — a few megabytes, occasionally a few hundred — so one per project costs you nothing worth worrying about.

Because each one is self-contained, it is also disposable. If an environment becomes a mess, delete the folder and rebuild it from your list of requirements. Nothing else on your machine is disturbed.

Creating one, step by step

The venv module ships with Python, so there is nothing to install. From your project folder:

1. Open a terminal in your project directory — the one holding your notebooks or scripts.
2. Run python -m venv .venv. A new .venv folder appears. The leading dot keeps it out of sight and, more usefully, out of your import path.
3. Activate it. On Windows: .venv\Scripts\activate. On macOS or Linux: source .venv/bin/activate. Your prompt should now begin with (.venv).
4. Update the installer once: python -m pip install --upgrade pip.
5. Install what you need: pip install pandas matplotlib jupyterlab.
6. Work. Everything you run from this terminal uses the environment.

When you are done, type deactivate. Closing the terminal does the same job.

Two habits make this reliable. First, create the environment inside the project folder so it travels with the work, and add .venv to .gitignore — never commit it. Second, if you use notebooks, register the environment as a kernel with python -m ipykernel install --user --name myproject, then select that kernel in JupyterLab. Otherwise your notebook runs against whatever interpreter launched the server, which is a very common source of “but I installed that already”.

Keeping a record of what you installed

Freeze your exact versions with pip freeze > requirements.txt, and rebuild anywhere using pip install -r requirements.txt. This file is the thing you commit. It is also the fastest way to get a colleague's project running or to move work onto a server.

One caution: freeze records everything, indirect dependencies included, pinned to exact versions. That is ideal for reproducing an analysis and less ideal as a long-term specification, because it will hold pandas at the same release forever. For analytical work, reproducibility usually matters more than freshness, so this is a reasonable trade. If you want a looser, readable list, write a requirements.in containing the packages you actually chose and pin only where it counts.

Which tool, and which mistakes to avoid

Choosing a tool

• venv — built in, nothing to install, fine for almost every analysis project. Start here.
• conda or mamba — handy when you need compiled libraries or non-Python dependencies, and common in scientific teams. It mixes awkwardly with pip if you are careless.
• pipenv, poetry or uv — add lock files and dependency resolution. Good for shared codebases, extra machinery for a folder of notebooks.

You do not need to pick a winner. What matters is that each project has its own environment and a file describing it.

Mistakes that cost time

• Installing with administrator rights, or using sudo pip install. That writes into the system interpreter and undoes the isolation you just built.
• Creating the environment but forgetting to activate it. If pip list shows packages you do not remember installing, check your prompt.
• Copying an environment folder between machines or operating systems. Rebuild it from requirements.txt instead.
• Expecting an activated terminal to affect a notebook that is already running. Restart the kernel or pick the right one.
• Committing the .venv folder to version control. It bloats the repository and breaks on other machines.

A routine that takes a minute

For each new analysis: make a folder, run python -m venv .venv, activate it, install pandas plus whatever else the job needs, register the kernel if you are working in notebooks, and commit requirements.txt alongside your code.

Six months later, when you cannot remember which version of a library handled dates the way your chart assumes, that small folder and text file answer the question. It is one folder per project, one command to activate, one file to share — and far less time than unpicking a broken global installation.

Photo: StockSnap / Pixabay