Easy Proof Omitted

A sink for random thoughts & experiments

0%

At the time of writing, this "feature" cannot be disabled without a dedicated chrome extension1. And of course removing each item one by one is unreasonable.

Hence the following JS snippet iterate over each item in the list and clicks the delete button.

Usage

  1. Go to the search engine settings chrome://settings/searchEngines.
  2. Open the Developer Tools by clicking F12 or CTRL+SHIFT+I.
  3. Copy the script below and paste it in the Javascript console tab.
  4. Fire Away.

Snippet

A slight delay has been added between clicks since after each action the browser takes a moment to render the view. feel free to adjust the sleep at line 16 as you see fit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const otherEngines = document.querySelector("body > settings-ui")
.shadowRoot.querySelector("#main")
.shadowRoot.querySelector("settings-basic-page")
.shadowRoot.querySelector("#basicPage > settings-section.expanded > settings-search-page")
.shadowRoot.querySelector("#pages > settings-subpage > settings-search-engines-page")
.shadowRoot.querySelector("#otherEngines").shadowRoot

let n = otherEngines.querySelector('iron-list').childElementCount - 1;
let rmbtn = otherEngines.querySelector('#frb0')
.shadowRoot.querySelector('#delete')

const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

while(n--) {
rmbtn.click();
await sleep(2000);
}

  1. Don't add custom search engines by Greg Sadetsky,↩︎

Python has been always the ultimate glue language in various fields due to it's intuitive syntax and a huge community that support different high-level tools and libraries. But it's no secret that python is extremely slow compared to compiled languages like C.

Luckily, Python can easily interface with low-level languages to gain the best of both worlds. This post will show how to find and elevate some of the performance bottlenecks in your code.

In mathematics, the Fibonacci numbers, commonly denoted \(F_n\), form a sequence, called the Fibonacci sequence, such that each number is the sum of the two preceding ones, starting from 0 and 1. That is,

\[\begin{align*} &\quad F_0=0, F_1= 1,\\ &F_n=F_{n-1} + F_{n-2}, &&\textbf{n} > 1 \end{align*}\]

The beginning of the sequence is thus: \(0,\;1,\;1,\;2,\;3,\;5,\;8,\;13,\;21,\;34,\;55,\;89,\;144,\; \ldots\)

Fibonacci numberWikipedia

For the demonstration purposes , A grossly inefficient implementation of Fibonacci sequence will be used as a toy example:

fib.py
1
2
3
4
def f(x):
return f(x - 1) + f(x - 2) if x > 2 else 1

print(f(40))

Identify bottlenecks

Python has a couple of built-in profilers that can be used to benchmark the performance of your code: profile and cProfile. profile is written in pure python which in return adds a significant overhead and wastes more time than cProfile, which is a C extension with minimal overhead.

To run cProfile on your script, simply type:

python -m cProfile fib.py

102334155
204668313 function calls (5 primitive calls) in 76.602 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.011 0.011 76.602 76.602 fib.py:6(<module>)
204668309/1 76.591 0.000 76.591 76.591 fib.py:6(f)
1 0.000 0.000 76.602 76.602 {built-in method builtins.exec}
1 0.000 0.000 0.000 0.000 {built-in method builtins.print}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}

To calculate the 40th fibonacci number in the sequence, the script spent 76.602 seconds in f(x) strongly hinting that this causes a serious bottleneck. 1

What can we do about it?

Cython

Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex). It makes writing C extensions for Python as easy as Python itself.

In a nutshell, Cython will transpile our python code to C then compile it into a shared object.

First, Write the function in a new file fib.pyx providing as much type hints as possible

1
2
def f(int x): # type hint
return f(x - 1) + f(x - 2) if x > 2 else 1

Create a new file setup.py with the following:

1
2
3
4
5
6
from setuptools import setup
from Cython.Build import cythonize

setup(
ext_modules=cythonize("fib.pyx")
)

Finally run setup.py with the following arguments:

python setup.py build_ext --inplace

The previous step generated the required shared object and the necessary interfaces that integrates seamlessly with python using a regular import statement.

1
2
import fib
print(fib.f(40))

re-run cProfile

102334155
97 function calls in 4.156 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:103(release)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:143(__init__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:147(__enter__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:151(__exit__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:157(_get_module_lock)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:176(cb)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
4 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:222(_verbose_message)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:342(__init__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:376(cached)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:389(parent)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:397(has_location)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:477(_init_module_attrs)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:549(module_from_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:58(__init__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:650(_load_unlocked)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:725(find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:78(acquire)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:800(find_spec)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:863(__enter__)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:867(__exit__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:890(_find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:986(_find_and_load)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1088(__init__)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1099(create_module)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1107(exec_module)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1265(_path_importer_cache)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1302(_get_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1334(find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1426(_get_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1431(find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:40(_relax_case)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:424(_get_cached)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:62(_path_join)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:629(spec_from_file_location)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:64(<listcomp>)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:80(_path_stat)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:90(_path_is_mode_type)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:99(_path_isfile)
1 0.000 0.000 4.156 4.156 fib.py:10(<module>)
5 0.000 0.000 0.000 0.000 {built-in method _imp.acquire_lock}
1 0.000 0.000 0.000 0.000 {built-in method _imp.create_dynamic}
1 0.000 0.000 0.000 0.000 {built-in method _imp.exec_dynamic}
1 0.000 0.000 0.000 0.000 {built-in method _imp.is_builtin}
1 0.000 0.000 0.000 0.000 {built-in method _imp.is_frozen}
5 0.000 0.000 0.000 0.000 {built-in method _imp.release_lock}
2 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
2 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
1 0.000 0.000 4.156 4.156 {built-in method builtins.exec}
6 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
3 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
1 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
1 0.000 0.000 0.000 0.000 {built-in method builtins.print}
1 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
1 0.000 0.000 0.000 0.000 {built-in method posix.getcwd}
2 0.000 0.000 0.000 0.000 {built-in method posix.stat}
1 4.155 4.155 4.155 4.155 {fib.f}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {method 'rpartition' of 'str' objects}
2 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}

Went down from 76.602 seconds to 4.156 almost 18x speed-up.

Although this approach is much easier, but the generated C code is part of large generic boilerplate and results in a noticeable overhead. Therefore, it's highly recommended to provide as much type hints as possible before compilation process.

ctypes

ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

First, Rewrite the performance critical functions in C or C++.

1
2
3
extern "C" int f(int x) {
return x > 2 ? f(x - 1) + f(x - 2) : 1;
}

extern "C" tells the compiler to avoid mangling the function name. which will work with simple function prototypes, but one-to-one mapping between C++ classes and python classes is not possible without an interface library like pybind11.

Then compile it into a shared object:

clang++ -shared -o fib.so fib.cpp

Lastly we can call any function from the shared object inside python:

1
2
3
4
import ctypes
fib = ctypes.CDLL('./fib.so')

print(fib.f(40))

Running cProfile again yields a few extra system calls that loads the shared object:

102334155
1265 function calls (1242 primitive calls) in 0.617 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1017(_handle_fromlist)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:103(release)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:143(__init__)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:147(__enter__)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:151(__exit__)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:157(_get_module_lock)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:176(cb)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:194(_lock_unlock_module)
7/1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
75 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:222(_verbose_message)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:342(__init__)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:35(_new_module)
8 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:376(cached)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:389(parent)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:397(has_location)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:477(_init_module_attrs)
5 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap>:549(module_from_spec)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:58(__init__)
5/1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap>:650(_load_unlocked)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:725(find_spec)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:78(acquire)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:800(find_spec)
15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:863(__enter__)
15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:867(__exit__)
5 0.000 0.000 0.001 0.000 <frozen importlib._bootstrap>:890(_find_spec)
5/1 0.000 0.000 0.003 0.003 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
5/1 0.000 0.000 0.003 0.003 <frozen importlib._bootstrap>:986(_find_and_load)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1010(path_stats)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:104(_path_isdir)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1088(__init__)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1099(create_module)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1107(exec_module)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1252(_path_hooks)
19 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1265(_path_importer_cache)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1302(_get_spec)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1334(find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1394(__init__)
8 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1400(<genexpr>)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1426(_get_spec)
15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1431(find_spec)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1479(_fill_cache)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:1520(path_hook_for_FileFinder)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:294(cache_from_source)
15 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:40(_relax_case)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:424(_get_cached)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:456(_check_name_wrapper)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:493(_classify_pyc)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:51(_unpack_uint32)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:554(_validate_hash_pyc)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:578(_compile_bytecode)
71 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:62(_path_join)
5 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:629(spec_from_file_location)
71 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:64(<listcomp>)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:68(_path_split)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:774(create_module)
3/1 0.000 0.000 0.002 0.002 <frozen importlib._bootstrap_external>:777(exec_module)
28 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:80(_path_stat)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:849(get_code)
9 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:90(_path_is_mode_type)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:939(__init__)
3 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:964(get_filename)
6 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:969(get_data)
8 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap_external>:99(_path_isfile)
1 0.000 0.000 0.000 0.000 <frozen zipimport>:63(__init__)
1 0.000 0.000 0.002 0.002 __init__.py:1(<module>)
2 0.000 0.000 0.000 0.000 __init__.py:101(CFunctionType)
14 0.000 0.000 0.000 0.000 __init__.py:141(_check_size)
1 0.000 0.000 0.000 0.000 __init__.py:153(py_object)
1 0.000 0.000 0.000 0.000 __init__.py:162(c_short)
1 0.000 0.000 0.000 0.000 __init__.py:166(c_ushort)
1 0.000 0.000 0.000 0.000 __init__.py:170(c_long)
1 0.000 0.000 0.000 0.000 __init__.py:174(c_ulong)
1 0.000 0.000 0.000 0.000 __init__.py:183(c_int)
1 0.000 0.000 0.000 0.000 __init__.py:187(c_uint)
1 0.000 0.000 0.000 0.000 __init__.py:191(c_float)
1 0.000 0.000 0.000 0.000 __init__.py:195(c_double)
1 0.000 0.000 0.000 0.000 __init__.py:199(c_longdouble)
1 0.000 0.000 0.000 0.000 __init__.py:220(c_ubyte)
1 0.000 0.000 0.000 0.000 __init__.py:227(c_byte)
1 0.000 0.000 0.000 0.000 __init__.py:232(c_char)
1 0.000 0.000 0.000 0.000 __init__.py:237(c_char_p)
1 0.000 0.000 0.000 0.000 __init__.py:243(c_void_p)
1 0.000 0.000 0.000 0.000 __init__.py:248(c_bool)
1 0.000 0.000 0.000 0.000 __init__.py:253(c_wchar_p)
1 0.000 0.000 0.000 0.000 __init__.py:258(c_wchar)
1 0.000 0.000 0.000 0.000 __init__.py:261(_reset_cache)
1 0.000 0.000 0.000 0.000 __init__.py:318(CDLL)
2 0.000 0.000 0.001 0.001 __init__.py:339(__init__)
2 0.000 0.000 0.000 0.000 __init__.py:367(_FuncPtr)
1 0.000 0.000 0.000 0.000 __init__.py:383(__getattr__)
1 0.000 0.000 0.000 0.000 __init__.py:390(__getitem__)
1 0.000 0.000 0.000 0.000 __init__.py:396(PyDLL)
1 0.000 0.000 0.000 0.000 __init__.py:436(LibraryLoader)
2 0.000 0.000 0.000 0.000 __init__.py:437(__init__)
3 0.000 0.000 0.000 0.000 __init__.py:498(PYFUNCTYPE)
3 0.000 0.000 0.000 0.000 __init__.py:499(CFunctionType)
2 0.000 0.000 0.000 0.000 __init__.py:75(CFUNCTYPE)
1 0.000 0.000 0.000 0.000 _endian.py:1(<module>)
1 0.000 0.000 0.000 0.000 _endian.py:23(_swapped_meta)
1 0.000 0.000 0.000 0.000 _endian.py:46(BigEndianStructure)
1 0.613 0.613 0.617 0.617 fib.py:1(<module>)
1 0.000 0.000 0.000 0.000 struct.py:3(<module>)
2 0.000 0.000 0.000 0.000 {built-in method _ctypes.POINTER}
2 0.001 0.001 0.001 0.001 {built-in method _ctypes.dlopen}
38 0.000 0.000 0.000 0.000 {built-in method _ctypes.sizeof}
3 0.000 0.000 0.000 0.000 {built-in method _imp._fix_co_filename}
26 0.000 0.000 0.000 0.000 {built-in method _imp.acquire_lock}
2 0.000 0.000 0.000 0.000 {built-in method _imp.create_dynamic}
2 0.000 0.000 0.000 0.000 {built-in method _imp.exec_dynamic}
4 0.000 0.000 0.000 0.000 {built-in method _imp.is_builtin}
5 0.000 0.000 0.000 0.000 {built-in method _imp.is_frozen}
26 0.000 0.000 0.000 0.000 {built-in method _imp.release_lock}
3 0.000 0.000 0.000 0.000 {built-in method _imp.source_hash}
18 0.000 0.000 0.000 0.000 {built-in method _struct.calcsize}
10 0.000 0.000 0.000 0.000 {built-in method _thread.allocate_lock}
12 0.000 0.000 0.000 0.000 {built-in method _thread.get_ident}
30 0.001 0.000 0.001 0.000 {built-in method builtins.__build_class__}
4/1 0.000 0.000 0.617 0.617 {built-in method builtins.exec}
30 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}
26 0.000 0.000 0.000 0.000 {built-in method builtins.hasattr}
31 0.000 0.000 0.000 0.000 {built-in method builtins.isinstance}
12 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1 0.000 0.000 0.000 0.000 {built-in method builtins.print}
2 0.000 0.000 0.000 0.000 {built-in method builtins.setattr}
3 0.000 0.000 0.000 0.000 {built-in method from_bytes}
6 0.000 0.000 0.000 0.000 {built-in method io.open_code}
3 0.000 0.000 0.000 0.000 {built-in method marshal.loads}
11 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
4 0.000 0.000 0.000 0.000 {built-in method posix.getcwd}
1 0.000 0.000 0.000 0.000 {built-in method posix.listdir}
28 0.000 0.000 0.000 0.000 {built-in method posix.stat}
2 0.000 0.000 0.000 0.000 {method 'clear' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
7 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
3 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
10 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
77 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects}
9 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
6 0.000 0.000 0.000 0.000 {method 'read' of '_io.BufferedReader' objects}
37 0.000 0.000 0.000 0.000 {method 'rpartition' of 'str' objects}
148 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
5 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects}

We went from 76.602 seconds to 0.612 almost 125x speed-up!


  1. obviously since it's the only function.↩︎

Decorators allow you to attach new functionality to a method without modifying it's behavior. This can be easily implemented in python because functions are treated as first class citizens (i.e a can be passed as arguments or returned from functions).

To put in simple terms:

A Decorator is a function the takes a function and returns a function.

They are used extensively in some popular frameworks like Flask.

Dummy Decorator

This is a simple decorator that prints a statement before and after a function is called.

Declare

1
2
3
4
5
6
7
8
def dec(f: Callable) -> Callable:
def g(*args, **kwargs):
print('Function starts')
r = f(*args, **kwargs)
print(f) # print function name and location in memory
print('Function ends')
return r # return result
return g # returns a new "decorated" function
  • Line 1: We define a new function (which will soon be our decorator) called dec which takes a function f as a parameter. I have added few type hints using import typing but it's not necessary and not enforced by the interpreter.

  • Line 2: We define a new inner function g which takes all the ordinary arguments and keyword arguments that function f takes.

  • Line 3: Just an example of something that happens before executing function f.

  • Line 4: Call function f with all of it's arguments and store result in variable r.

  • Line 5: Just and example of something that happens after executing function f.

  • Line 6: Returns the result of function f in order to keep it working as intended.

  • Line 7: Returns the newly decorated function g.

Usage

Now let's decorate a simple add function with the new decorator:

1
2
3
4
5
6
@dec
def add(x: float, y: float):
return x + y

add(5, 6)
print(add)

Output

1
2
3
4
5
Function starts
<function add at 0x7f6735d528b0>
Function ends
11
<function dec.<locals>.g at 0x7f6735d52940>

The difference in function name and memory address (line 2 & line 5) will be discussed below.

Timer Decorator

Here is an applicable example of a decorator that enables you to easily determine how long does a function takes in order to finish execution.

Declare

1
2
3
4
5
6
7
8
def timer(f: Callable) -> Callable:
def g(*args, **kwargs):
t_start = time.monotonic()
r = f(*args, **kwargs)
t_end = time.monotonic()
print(f"It took {t_end - t_start} to execute \"{f.__name__}\" function")
return r
return g

Usage

You replace the @dec with @timer or even stack multiple decorators on top of each other:

1
2
3
4
5
6
7
@dec
@timer
def add(x: float, y: float):
return x + y

add(5, 6)
print(add)

Output

1
2
3
4
5
Function starts
It took 5.5779964895918965e-06 to execute "add" function
<function timer.<locals>.g at 0x7f1da16d2940>
Function ends
11
  • Line 1: Output of decorator @dec.
  • Line 2: Output of decorator @timer.
  • Line 3, 4: The rest of @dec's output.
  • Line 5: Output of function add.

Tracker Decorator

Let's assume you want to keep track of the output of different functions in some sort of a global set. This can be useful for some sort of a simple unit testing solution.

Declare

1
2
3
4
5
6
7
8
9
TEST = set()

def track(f: Callable) -> Callable:
def g(*args, **kwargs):
r = f(*args, **kwargs)
t = (f.__name__, args, kwargs, r)
TEST.add(t)
return r
return g

Not much different than the previous decorators, only is adding a tuple that contains the function name, arguments and result to a global set called TEST.

Usage

1
2
3
4
5
6
7
8
print(add(5, 6))
print(add(3, 7))
print(add(11, 17))
print(add)
print(STATE)

assert STATE == {('add', (11, 17), 28), ('add', (5, 6), 11), ('add', (3, 7), 10)}
print('All tests passed successfully')

If anything went wrong with the assert statement; the interpreter will halt execution and raise and assert exception. AssertionError

Output

1
2
3
4
5
6
11
10
28
<function track.<locals>.g at 0x7f01e1b399d0>
[('add', (5, 6), {}, 11), ('add', (3, 7), {}, 10), ('add', (11, 17), {}, 28)]
All tests passed successfully

functools.wraps

In order to add additional functionality to a method, the decorator generate a new function with a new name in a new memory address which might confuse some tools and debuggers.

To avoid renaming your function and keep it's original docstring, use wraps from functools:

1
2
3
4
5
6
7
8
9
10
11
from functools import wraps

def dec(f: Callable) -> Callable:
@wraps(f)
def g(*args, **kwargs):
print('Function starts')
r = f(*args, **kwargs)
print(f) # print function name and location in memory
print('Function ends')
return r # return result
return g

Every once and a while, I check my email's spam folder for fun phishing attempts and stumped upon one in particular that was rather interesting:

1
2
3
4
5
6
7
8
Title: [Some old password]

Actually, I placed a virus on the xXx vids (sex sites) site & guess what, you visited this web site to have fun. While you were viewing videos, your web browser started working as a Remote Desktop having a keylogger which gave me accessibility to your display and also cam recording.
Just after that, my software collected all your contacts from your Messenger, social networks, and email.
[Some old password] is one of your passwords.
if you send me $986 as a donation through Bitcoin, I will erase the recording immediately.
(search for in Google "how to buy bitcoin"). my BTC Address: [A brand new bitcoin wallet address with zero transactions]
If I don't get the BitCoins in 24hrs, I will definately send your video to all of your contacts, don't.reply to this email it's hacked. WxEQ

Of course it goes without saying, This is "definately"1 non-sense, yet you might ask how did he managed to get the old password?

This answer is from one of many data breaches that happens almost every few months. Almost every major company had a data breach in some point (Adobe, Dropbox, LinkedIn, ...) and you can check your email using one of the following services:

have i been pwned?

have i been pwned? checks if you have an account that has been compromised in a data breach and offer to notify you if your email appears in any public accounts dump or spam list.

Hacked Emails

Hacked Emails very similar to have i been pwned? but requires email verification before checking your email against it's database of public data breaches.

DeHashed

DeHashed is similar to the other solutions but it takes this process one step further by offers a cheap subscription plan that allows anyone to get the list of publicly plaintext password for any email address.

If you happen to receive a similar email, you can report the bitcoin address to:

Bitcoin Abuse Database

BitcoinAbuse.com is a public database of bitcoin addresses used by scammers, hackers, and criminals. Bitcoin is anonymous if used perfectly. Luckily, no one is perfect. Even hackers make mistakes. It only takes one slip to link stolen bitcoin to a hacker's their real identity. It is our hope that by making a public database of bitcoin addresses used by criminals it will be harder for criminals to convert the digital currency back into fiat money.


  1. The miss-spelling was intentional.↩︎

Since it's foundation in 2015, Jupyter notebooks became the De facto standard for data manipulation and visualization. You can write code and annotate it with Markdown, HTML or even LaTeX.

Jupyter notebooks are not only for python, you can install additional community supported kernels for other programming languages like JavaScript, Scala or even C++ (yes, even compiled languages).

Google Colab

Google offers limited CPU, GPU and TPU time on their infrastructure free of charge.

At the time of writing, they offer:

  • 12GB RAM
  • 1 CPU
  • NVIDIA Tesla K80
  • TPU

Open Google Drive and from right-click menu More -> Connect more apps.

Search for Google Colaboratory.

Now you can create new notebooks in your Google Drive.

You can also access your files in Google Drive from inside the notebook. but first you have to mount it.

On left sidebar click on Files -> Mount Drive.

Paperspace

Under their Gradient° option, Paperspace offers a more generous notebooks for less time (6 hours) but you create new one after the previous one expired if they have available resources to spare.

Make sure to keep all your work in the storage folder in order for the changes made to persist across different notebooks.

At the time of writing, they offer:

  • 30 GB RAM
  • 8 CPU
  • NVIDIA Quadro P5000

Your Own Server

You always have the option to rent your own server from any cloud provider and customize it to your taste (and budget).

Here is a step-by-step guide on how to setup a quick notebook:

After you install python on your server you can install Jupyter through pip

1
$ pip install jupyterlab

Or download and install Anaconda which already comes with jupter pre-installed.

1
$ wget https://repo.anaconda.com/archive/Anaconda3-YYYY.MM-Linux-x86_64.sh

Make the script executable

1
$ chmod +x Anaconda3-YYYY.MM-Linux-x86_64.sh
Run it
1
$ ./Anaconda3-YYYY.MM-Linux-x86_64.sh

A very helpful wizard will ask you to accept the license and choose a location for the install directory.

Careful not click ENTER mindlessly. Last option asks you if you want to add the previously mentioned location to $PATH environment variable. the default is no, but i recommend typing yes.

1
2
3
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>> yes

If you didn't, you can edit $PATH variable to add the anaconda's directory as the first path, in my case I installed it in /root/anaconda3.

1
2
 $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

1
$ export PATH=/root/anaconda3:$PATH

If you type jupyter in your terminal and the usage instructions appeared, then everything this far is working as intended.

Next step is to generate a jupyter config file.

1
$ jupyter notebook --generate-config

This will create a config file in the ~/.jupyter/ called jupyter_notebook_config.py. Now open it with your favorite text editor and edit and uncomment few stuff.

Use '*' to allow any origin to access your server not just localhost.

1
c.NotebookApp.allow_origin = '*'

Change the allow_remote_access from False to True, Unless you want access the notebook through an SSH Forwarding.

1
c.NotebookApp.allow_remote_access = True

You can open a SSH tunnel without restarting your session using this escape sequence (sometimes called SSH Konami Code)

  1. Press the SHIFT key. (and keep pressing)
  2. Press the tilde (~) key.
  3. Press the letter C.
  4. Un-Press the SHIFT key.

A special SSH prompt will appear

1
2
3
4
5
6
7
8
9
ssh> help

Commands:
-L[bind_address:]port:host:hostport Request local forward
-R[bind_address:]port:host:hostport Request remote forward
-D[bind_address:]port Request dynamic forward
-KL[bind_address:]port Cancel local forward
-KR[bind_address:]port Cancel remote forward
-KD[bind_address:]port Cancel dynamic forward
This will bind port 8888 on the server to port 8888 on your machine.
1
ssh> -L 8888:localhost:8888

Change the allow_remote_access from False to True, Unless you want access the notebook through an SSH Forwarding.

1
c.NotebookApp.allow_remote_access = True

This enables you to server your Notebook over HTTPS (highly recommended for security).

  1. You can generate your own self-signed SSL certificate but this will make your browser show a warning everytime you visit your notebook (since the browser doesn't recognize the issuer of the certificate). $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

  2. You can generate a free certificate from Let's Encrypt, they offer certbot a very handy tool that automates the process of creating the certificate.

  3. After that put the path of the certificate file, CA file and key file here

1
2
3
4
c.NotebookApp.certfile = ''
c.NotebookApp.client_ca = ''
...
c.NotebookApp.keyfile = ''

Let the server accept connections from any IP instead of just locahost.

1
c.NotebookApp.ip = '*'

The port the notebook server will listen on.

Make sure to use port 443 if and only if you have set a SSL certificate and instead to use HTTPS.

1
c.NotebookApp.port = 80 # (443)

Here you can specify a directory as a starting point for your notebook. I like to create a workspace folder and keep everything in it.

This makes backup process easy as you only have only directory to worry about.

1
c.NotebookApp.notebook_dir = '' # (/root/workspace)

We don't need a browser tab to open (on the server side) whenever we run jupyter.

1
c.NotebookApp.open_browser = False

Assuming multiple parties are going to use this notebook server, set quit button to False to avoid accidental shutdown of the server.:

1
c.NotebookApp.quit_button = False

Feel free to change anything else according to your own needs. Then save your changes and run the jupyter and copy token.

1
2
3
4
$ jupyter notebook
...
http(s)://[ip]:[port]/?token=[token]
...

Type in the IP of your server in your browser and you will be greeted with a window asking your a token to setup the password. paste the token under the Setup a Password bottom section and type new password.

Later on, the server will only ask you for the password.

Congratulations, your own jupyter server is ready.

Go create notebooks that will change the world.