In this Python tutorial, we will discuss how to optimize your Pyinstaller EXE’s using Virtual Environments. We will be using the “venv” library to create the Virtual environment for Pyinstaller, which is actually already included in every Python installation by default (because it’s part of the standard library).
We will walk you through the entire process, starting from “what” virtual environments are, “why” we need them and “how” to create one.
Understanding Pyinstaller
Let me start by telling you how Pyinstaller works. We all know that Pyinstaller creates a standalone EXE which bundles all the dependencies, allowing it to run on any system.
What alot of people do not know however, is “HOW” Pyinstaller does this.
Let me explain.
What Pyinstaller does is “freezes” your Python environment into what we call a “frozen” application. In non-technical terms, this means to take bundle everything in your Python environment, like your Python installation, libraries that you have installed, and other dependencies (e.g DLL’s or Data files) you may be using, into a single application.
Kind of like taking a “snap-shot” of your program in it’s running state (with all the dependencies active) and saving it.
With this understanding, we can now safely explain the benefits of Virtual Environments.
What are Virtual Environments in Python?
Imagine for a moment that you have a 100 libraries installed for Python on your device. You might think you do not have many, but the truth is that when you download a big library (e.g Matplotlib) it downloads several other libraries along with it (as dependencies).
To check the currently installed libraries on our systems (excluding the ones included by default), run the following command.
pip listIt will give you something like that following output.
altgraph 0.17.3
astroid 2.12.9
async-generator 1.10
attrs 22.2.0
auto-py-to-exe 2.24.1
Automat 22.10.0
autopep8 1.7.0
Babel 2.10.3
beautifulsoup4 4.11.1
bottle 0.12.23
bottle-websocket 0.2.9
bs4 0.0.1
cad-to-shapely 0.3.1
cairocffi 1.3.0
CairoSVG 2.5.2
certifi 2022.12.7
cffi 1.15.1
chardet 4.0.0
colorama 0.4.5
constantly 15.1.0
cryptography 38.0.4
cssselect 1.2.0
cssselect2 0.6.0
cx-Freeze 6.13.1
cycler 0.11.0
Cython 0.29.32
defusedxml 0.7.1
dill 0.3.5.1
Eel 0.14.0
et-xmlfile 1.1.0
exceptiongroup 1.1.0
ezdxf 0.18
filelock 3.8.0
fonttools 4.34.4
future 0.18.2
geomdl 5.3.1
gevent 22.10.2
gevent-websocket 0.10.1
greenlet 2.0.1
h11 0.14.0
hyperlink 21.0.0
idna 2.10
incremental 22.10.0
isort 5.10.1
itemadapter 0.7.0
itemloaders 1.0.6
Jinja2 3.0.1
jmespath 1.0.1
kiwisolver 1.4.4
lazy-object-proxy 1.7.1
lief 0.12.3
lxml 4.9.1
MarkupSafe 2.1.1
matplotlib 3.5.3
mccabe 0.7.0
more-itertools 8.14.0
MouseInfo 0.1.3
mpmath 1.2.1
Nuitka 1.2.4
numexpr 2.8.3
numpy 1.23.1
openpyxl 3.0.10
ordered-set 4.1.0
outcome 1.2.0
packaging 21.3
pandas 1.4.3
pandastable 0.13.0
parsel 1.7.0
pefile 2022.5.30
Pillow 8.4.0
pip 22.2.1
platformdirs 2.5.2
Protego 0.2.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
PyAutoGUI 0.9.53
pycodestyle 2.9.1
pycparser 2.21
PyDispatcher 2.0.6
pygal 3.0.0
pygame 2.1.2
pygame-menu 4.2.8
PyGetWindow 0.0.9
pyinstaller 5.6.2
pyinstaller-hooks-contrib 2022.13
pylint 2.15.2
PyMsgBox 1.0.9
PyMuPDF 1.20.2
pyOpenSSL 22.1.0
pyparsing 3.0.9
pyperclip 1.8.2
PyQt5 5.15.7
PyQt5-Qt5 5.15.2
PyQt5-sip 12.11.0
PyQt6 6.4.0
PyQt6-Qt6 6.4.1
PyQt6-sip 13.4.0
PyRect 0.2.0
PyScreeze 0.1.28
PySocks 1.7.1
pytest-check 1.0.6
python-dateutil 2.8.2
pytweening 1.0.4
pytz 2022.1
pywin32-ctypes 0.2.0
queuelib 1.6.2
requests 2.25.1
requests-file 1.5.1
rhino-shapley-interop 0.0.4
rhino3dm 7.15.0
scipy 1.9.0
Scrapy 2.7.1
sectionproperties 2.0.3
selenium 4.7.2
service-identity 21.1.0
setuptools 63.2.0
Shapely 1.8.2
six 1.16.0
sniffio 1.3.0
sortedcontainers 2.4.0
soupsieve 2.3.2.post1
sympy 1.10.1
tabulate 0.8.10
tinycss2 1.1.1
tkcalendar 1.6.1
tkdesigner 1.0.6
tkinter-tooltip 2.1.0
tldextract 3.4.0
toml 0.10.2
tomli 2.0.1
tomlkit 0.11.4
triangle 20220202
trio 0.22.0
trio-websocket 0.9.2
Twisted 22.10.0
twisted-iocpsupport 1.0.2
typing_extensions 4.3.0
urllib3 1.26.13
w3lib 2.1.1
webencodings 0.5.1
whichcraft 0.6.1
wrapt 1.14.1
wsproto 1.2.0
xlrd 2.0.1
zope.event 4.5.0
zope.interface 5.5.2
That was quite a long list right? I don’t even recognize half of those libraries (they were installed as dependencies). And this is on a relatively new Python installation (3-4 months old). Running this on my old device might have given me double the above amount.
Now you might have already put two-and-two together and realized the problem here.
When we normally use Pyinstaller to bundle our applications, it ends up including ALL of the libraries that we have installed. Regardless of whether they are actually needed, or not.
Now obviously this is a big problem, especially if you have several large libraries lying around which are not actually being used.
The Solution?
Virtual Environments!
Now, what we could do is setup a new Python installation on your PC and only install the required packages (which you know are being used). But this is an extra hassle, and can cause issues with your current Python installation if you are not careful.
Instead, we use Virtual environments which basically create a “fresh copy” of your current Python version, without any of the installed libraries. You can create as many virtual environments as you want!
We can then compile our Pyinstaller EXE’s inside these virtual environments (just like how we normally do). This time the EXE will only include the bare minimum number of libraries.
It is actually recommended to have a virtual environment for each major project you have. This is to ensure that there is no library conflict, and to ensure version control (the version of the libraries you are using).
Version Control in Virtual Environments with Venv
For example, a common issue that can happen is when you install a new library “A” (unrelated to your application) and it requires a dependency “B”, which is also required by library “C”.
Library “A” requires the dependency to be at atleast version 1.3 (random version number i picked), whereas library “C” only works with the dependency “B” up-to version 1.2 (1.3 onwards not supported).
Hence, we now have a conflict issue. There are many other scenarios like this under which problems can occur. This is just one of them.
Virtual environments help isolate projects and dependencies into separate environments, minimizing the risk of conflict.
Creating a Pyinstaller Virtual Environment with Venv
Now for the actual implementation part of the tutorial. The first thing we will do is setup our Virtual environments.
For users with Python added to PATH, run the following command.
python -m venv tutorial
In the above command, “tutorial” is the name of the virtual environment. This is completely your choice what you choose to name it. Also pay attention to which folder you are running this command in. The virtual environment will be created there.
For users who do not have Python added to PATH, you need to find the path to your Python installation. You can typically find it in a location like this:
C:\Users\CodersLegacy\AppData\Local\Programs\Python\Python310To create a virtual environment you need to run the following command (swapping out “python” for the location of your python.exe file)
C:\Users\CodersLegacy\AppData\Local\Programs\Python\Python310\python.exe -m venv tutorial
Activating the Virtual Environment
We aren’t done yet though. The Virtual environment needs to be activated first! If you completed the previous step, we should have a folder structure something like this.
-- virtuals_envs_folder
-- tutorialvirtual_envs_folder is simply a parent folder where we ran the previous commands for creating the virtual environment.
We will now add a new Python file to the virtual_envs_folder (not the tutorial folder). So now our file structure is something like this.
-- virtuals_envs_folder
-- tutorial
-- file.pyfile.py is where all the code will go, which we want want to convert to a pyinstaller exe. Any supporting files, folder or libraries you have created can also be added here.
Now we need to run another command which will activate the virtual environment. The command can vary slightly depending on what terminal/console/OS you are using.
Command Prompt: (Windows)
C:\Users\CodersLegacy\virtual_envs_folder> tutorial\Scripts\activate.batWindows PowerShell:
C:\Users\CodersLegacy\virtual_envs_folder> tutorial\Scripts\Activate.ps1Linux:
C:\Users\CodersLegacy\virtual_envs_folder> tutorial\bin\activateCongratulations, now your Virtual environment is now activated and ready to run! Our command-line will now be pointing to the Python installation inside our Virtual environment instead of the main Python installation. (This effect will end once you close the command line/terminal)
Your virtual environment folder (tutorial) should look something like this:
-- tutorial
-- Include
-- Lib
-- Scripts
-- file.pyThe two important files here are “Lib” and “Scripts”. “Lib” is where all of our installed libraries will go. “Scripts” is where our Python.exe file is.
Your command prompt/terminal should also look something like this:
(tutorial) C:\Users\CodersLegacy\virtual_envs_folder>Notice the “(tutorial)” which is now included right in the start. If this has appeared, your Venv Virtual Environment is ready to use with Python and Pyinstaller.
Setting up our Application in the Virtual Environment
Now we will begin installing the required libraries we need. Here is some sample code we will be using in our file.py file.
import tkinter as tk
from tkinter.filedialog import askopenfilename, asksaveasfile
import numpy as np
from matplotlib.figure import Figure
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
from matplotlib.path import Path
from matplotlib.patches import PathPatch
from matplotlib.collections import PatchCollection
from pandastable import Table
import pandas as pd
class Window():
def __init__(self, master):
self.main = tk.Frame(master, background="white")
self.rightframe = tk.Frame(self.main, background="white")
self.rightframe.pack(side=tk.LEFT)
self.leftframe = tk.Frame(self.main, background="white")
self.leftframe.pack(side=tk.LEFT)
self.rightframeheader = tk.Frame(self.rightframe, background="white")
self.button1 = tk.Button(self.rightframeheader, text='Import CSV', command=self.import_csv, width=10)
self.button1.pack(pady = (0, 5), padx = (10, 0), side = tk.LEFT)
self.button2 = tk.Button(self.rightframeheader, text='Clear', command=self.clear, width=10)
self.button2.pack(padx = (10, 0), pady = (0, 5), side = tk.LEFT)
self.button3 = tk.Button(self.rightframeheader, text='Generate Plot', command=self.generatePlot, width=10)
self.button3.pack(pady = (0, 5), padx = (10, 0), side = tk.LEFT)
self.rightframeheader.pack()
self.tableframe = tk.Frame(self.rightframe, highlightbackground="blue", highlightthickness=5)
self.table = Table(self.tableframe, dataframe=pd.DataFrame(), width=300, height=400)
self.table.show()
self.tableframe.pack()
self.canvas = tk.Frame(self.leftframe)
self.fig = Figure()
self.ax = self.fig.add_subplot(111)
self.graph = FigureCanvasTkAgg(self.fig, self.canvas)
self.graph.draw()
self.graph.get_tk_widget().pack()
self.canvas.pack(padx=(20, 0))
self.main.pack()
def import_csv(self):
types = [("CSV files","*.csv"),("Excel files","*.xlsx"),("Text files","*.txt"),("All files","*.*") ]
csv_file_path = askopenfilename(initialdir = ".", title = "Open File", filetypes=types)
tempdf = pd.DataFrame()
try:
tempdf = pd.read_csv(csv_file_path)
except:
tempdf = pd.read_excel(csv_file_path)
self.table.model.df = tempdf
self.table.model.df.columns = self.table.model.df.columns.str.lower()
self.table.redraw()
self.generatePlot()
def clear(self):
self.table.model.df = pd.DataFrame()
self.table.redraw()
self.ax.clear()
self.graph.draw_idle()
def generatePlot(self):
self.ax.clear()
if not(self.table.model.df.empty):
df = self.table.model.df.copy()
self.ax.plot(pd.to_numeric(df["x"]), pd.to_numeric(df["y"]), color ='tab:blue', picker=True, pickradius=5)
self.graph.draw_idle()
root = tk.Tk()
window = Window(root)
root.mainloop()
Now you have two options. You can either go and install each library individually using “pip”, or you can create a requirements.txt file. We will go with the latter option, because its a recommended approach for helping to maintain the correct versions.
First create a new requirements.txt file in the parent folder of your virtual environment. File structure should look like this:
-- virtuals_envs_folder
-- tutorial
-- file.py
-- requirements.txtNow we will open a new command prompt (unrelated to our Virtual environment), and use it to check the versions of each our required libraries.
We can check the version of an installed library using pip show <library-name>. Running pip show matplotlib, gives us the following output.
Name: matplotlib
Version: 3.5.3We will now add this information to our requirements.txt file. For the sample code we provided above, our file will look like this:
matplotlib==3.5.3
sympy==1.10.1
pandastable==0.13.0
pandas==1.4.3
numpy==1.23.1
scipy==1.9.0
pyinstaller==5.6.2
auto-py-to-exe==2.24.1Now we will run the following command to have them all installed in one go.
pip install -r requirements.txtUsing Pyinstaller in our Venv Virtual Environment
And now, we are FINALLY done with all the setup!
But don’t forget why we are doing all of this!. As a test, try running pip show again, and see how many libraries get printed out this time. It should be alot less than what we had before.
All we need to do now is run the pyinstaller command like we would do normally.
pyinstaller --noconsole --onefile file.pyThis will generate our Pyinstaller EXE in our Venv Virtual environment (or in the parent folder)! Now observe the size difference and let us know down in the comments section how much of an improvement you got!
There should also be a slight speed bonus, due to the smaller size and lower number of modules to load.
If you are looking to further reduce the size of your Python module, the UPX packer is a great way to easily bring down the size of your EXE substantially. Check out our tutorial on it!
This marks the end of the “Setup Virtual Environment for Pyinstaller with Venv” Tutorial. Any suggestions or contributions for CodersLegacy are more than welcome. Questions regarding the tutorial content can be asked in the comments section below.
