Technically Personal - python

Testing multiple versions of Python in parallel

2025-07-21T00:00:00+00:00

I was reading my RSS feed when I came across Daniel Roy Greenfeld’s article</a> about using uv</a> to run unit tests on multiple versions of Python. Daniel writes:</p>

But what if I want to test a particular version of Python? Then I simple specify the version of Python to run the test:</p>

uv</span> run --python=3.13 --with pytest --with httpx pytest</span></span></code></pre>
Here’s where it gets fun. I can use a Makefile</code> (or a justfile</a>) to test on multiple Python
versions.</p>
testall</span>:</span>  ## Run all the tests for all the supported Python versions</span></span>
    uv run --python=3.10 --with pytest --with httpx pytest</span></span>
    uv run --python=3.11 --with pytest --with httpx pytest</span></span>
    uv run --python=3.12 --with pytest --with httpx pytest</span></span>
    uv run --python=3.13 --with pytest --with httpx pytest</span></span></code></pre></blockquote>
This is fantastic! I’ve used uv</code> for only a couple of projects, and already I cannot imagine ever
giving it up again.</p>
However, there’s one disadvantage to the technique Daniel describes above: every time uv switches
between Python versions, it will have to remove and replace the whole .venv</code> directory and
reinstall all dependencies.</p>
For example, this is what happens when I consecutively run the tests on a project with two different
python versions:</p>
$</span> uv run --python</span> 3.12</span> --extra test pytest</span></span>
Using</span> CPython</span> 3.12.11</span> interpreter at: ...</span></span>
Creating</span> virtual environment at: .venv</span></span>
Installed</span> 5</span> packages in 10ms</span></span>
...</span></span>
</span>
$</span> uv run --python</span> 3.13</span> --extra test pytest</span></span>
Using</span> CPython</span> 3.13.5</span> interpreter at: ...</span></span>
Removed</span> virtual environment at: .venv</span></span>
Creating</span> virtual environment at: .venv</span></span>
Installed</span> 5</span> packages in 8ms</span></span></code></pre>
This most likely isn’t much of a problem, because of how fast uv</code> is, but it’s still not ideal. In
addition, using Makefiles to run multiple commands sequentially is sooOOoOo boring, especially now
that we have computers with insane processor counts. Let’s run these commands in parallel!</p>
Caveat emptor</strong></em>: This presumes that your tests don’t have any real side-effects, and that they
can be run fully simultaneously without clashing with one another (shared databases like sqlite, or
creating test files could wreak havoc here).</p>
We need to deploy a few tricks so that uv</code> won’t stumble over itself and try to re-use the .venv</code>
directory. In the end, this ends up being slightly less elegant than I wanted, but I guess that’s a
lesson for me in believing slightly too much in Makefiles.</p>
.PHONY</span>:</span> testall</span></span>
.PRECIOUS</span>:</span> .venv-%/pyvenv.cfg</span></span>
</span>
PYTHON_VERSIONS</span> :=</span> 3.10 3.11 3.12 3.13</span></span>
TEST_TARGETS</span> :=</span> $(</span>addprefix</span> .tests-</span>,</span>$(PYTHON_VERSIONS))</span></span>
</span>
testall</span>:</span> $(TEST_TARGETS)</span></span>
</span>
.venv-%/pyvenv.cfg</span>:</span></span>
    uv venv \</span></span>
      --python $(</span>patsubst</span> .venv-%</span>,</span>%</span>,</span>$(@D)) \</span></span>
      $(@D)</span></span>
</span>
.tests-%</span>:</span> .venv-%/pyvenv.cfg</span></span>
    venv</span>=</span>$(</span>patsubst</span> .tests-%</span>,</span>.venv-%</span>,</span>$@) && </span>\</span></span>
    source $$venv/bin/activate && </span>\</span></span>
    uv run </span>\</span></span>
      --active </span>\</span></span>
      --python $$venv </span>\</span></span>
      --extra test </span>\</span></span>
      pytest</span></span>
    touch $@</span></span></code></pre>
Okay, there’s a lot to unpack here. Let’s dive in.</p>
The .PHONY</code> and .PRECIOUS</code> lines are easily explained: there will never be a file called
testall</code>, so no need to check if it exists or not, and don’t delete the intermediate files created
when we initiate a venv.</p>
Similarly easy to understand, the PYTHON_VERSIONS</code> just contains all the versions of Python we want
to test against. And the TEST_TARGETS</code> represents the actual targets in the file we want to
execute for (tests-3.13</code>, tests-3.12</code>, etc).</p>
The pyvenv.cfg</code> target just says “if anyone needs .venv-whatever/pyvenv.cfg</code>, here’s how to create
it”. It very basically calls uv venv</code> and initialises a virtualenv with the correct Python version.
The pyvenv.cfg</code> file is created by uv</code> when it sets up the virtualenv, satisfying the Make rule.</p>
The final target, .tests-%</code>, we create manually (touch $@</code>) after running uv run</code> inside the
proper virtualenv. This is most likely the part I’m the least happy with; I wish uv</code> had a way of
either prefixing venv dirs, or the ability to configure it to say “Python version 3.12 uses
.venv-3.12 as the venv dir” or something.</p>
But what does it do? Well, for starters, if you use it exactly like Daniel’s version, it behaves
exactly the same way, but it doesn’t replace .venvs all the time:</p>
$</span> make testall</span></span>
venv</span>=</span>.venv-3.10</span> &&</span> source</span> $venv</span>/bin/activate</span> &&</span> \</span></span>
  uv</span> run --active --python</span> $venv</span> --extra test pytest</span></span>
=============</span> 1</span> passed in 1.01s =============</span></span>
touch</span> .tests-3.10</span></span>
</span>
venv</span>=</span>.venv-3.11</span> &&</span> source</span> $venv</span>/bin/activate</span> &&</span> \</span></span>
  uv</span> run --active --python</span> $venv</span> --extra test pytest</span></span>
=============</span> 1</span> passed in 1.01s =============</span></span>
touch</span> .tests-3.11</span></span>
</span>
venv</span>=</span>.venv-3.12</span> &&</span> source</span> $venv</span>/bin/activate</span> &&</span> \</span></span>
  uv</span> run --active --python</span> $venv</span> --extra test pytest</span></span>
=============</span> 1</span> passed in 1.01s =============</span></span>
touch</span> .tests-3.12</span></span>
</span>
venv</span>=</span>.venv-3.13</span> &&</span> \</span> source</span> $venv</span>/bin/activate</span> &&</span> \</span></span>
  uv</span> run --active --python</span> $venv</span> --extra test pytest</span></span>
=============</span> 1</span> passed in 1.01s =============</span></span>
touch</span> .tests-3.13</span></span></code></pre>
But more interestingly, we can now run run all tests in parallel across versions of Python!</p>
$</span> time make testall</span></span>
make</span> testall  0.55s user 0.13s system 14% cpu</span> 4.725</span> total</span></span>
</span>
$</span> time make -j testall</span></span>
make</span> -j testall  0.63s user 0.17s system 64% cpu</span> 1.234</span> total</span></span></code></pre>
For real-world use, I would most likely also add a clean</code> target, and I would make the test targets
depend on the actual pyproject.toml</code> and .py</code> files, but that’s an exercise left for the reader.</p>


Freebox automatic ruTorrent download
2016-04-26T00:00:00+00:00
I love Free. I think it is one of the best ISPs in the world, and even more so in France. If I’m not
mistaken, they invented the ISP-provided set top box that seems so common place these days. Their
Wikipedia entry</a> explains some of the reasons I like them so much, but to list a few:</p>

unmetered bandwidth,</li>
fixed IPv4 address, /60 IPv6 block,</li>
they allowed and encouraged people to run their own servers (reverse-IP customisation, every port
available, etc),</li>
their router doesn’t suck,</li>
fixed price policy: ~30 euros, all inclusive,</li>
the IPTV streams are viewable on your PCs through VLC,</li>
the list could go on for a long time.</li>
</ul>
All this being said, one of the major reason I loved their product so much was because of their
modem, the Freebox</a>. I’ve been a long fan of all their hardware (since the V4), but
the Freebox Revolution takes the cake.</p>
Anyway, back on topic. The Freebox Revolution Server allows you to set an RSS feed, and all items in
this RSS feed will be automatically downloaded by the newsgroup/torrent/direct download client that
is built into the router (yeah, they have that built in).</p>
I used to use a seedbox, but because I had a really bad DSL connection (4 Mbits), I preferred for my
downloads to happen during the day—when nobody was home—rather than having to painstakingly wait
for the bits to transfer when I was home. The other reason was also that downloading anything would
immediately saturate the link, and ruin browsing/YouTubing for anyone home at that point. The other
main point that should be stressed is that the Freebox Player (the part connected to the TV) could
stream any file stored on the Server’s hard drive (it acted as a NAS, out of the box). This meant
that we were able to watch the new GoT episode faster than most of my colleagues who had FTTH
connections, simply because we just had to turn the telly on, and click “Play”.</p>
I came up with the following script to generate an RSS feed of downloaded</em> files that my Freebox
would then automatically pick up.</p>
#!/usr/bin/python</span></span>
</span>
import</span> cPickle</span> as</span> pickle</span></span>
import</span> xml</span>.</span>etree</span>.</span>cElementTree</span> as</span> et</span></span>
# You can find PyRSS2Gen at:</span></span>
# https://pypi.python.org/pypi/PyRSS2Gen</span></span>
import</span> PyRSS2Gen</span> as</span> RSS2</span></span>
import</span> urllib</span>,</span> mimetypes</span></span>
from</span> os</span> import</span> path</span>,</span> walk</span></span>
from</span> datetime</span> import</span> datetime</span></span>
</span>
DIRECTORY</span> =</span> '/home/rtorrent/downloaded'</span></span>
DATABASE</span> =</span> '/home/rtorrent/.rssdb'</span></span>
EXTENSIONS</span> =</span> [</span>'.avi'</span>,</span> '.mkv'</span>,</span> '.mp4'</span>,</span> '.mpeg'</span>,</span> '.mpg'</span>]</span></span>
HOSTNAME</span> =</span> 'myseedbox.example.com'</span></span>
PATH</span> =</span> 'downloads'</span></span>
HTTPS</span> =</span> True</span></span>
RSS_FILENAME</span> =</span> '/home/rtorrent/downloads.rss'</span></span>
</span>
class</span> FileList</span>:</span></span>
  def</span> __init__</span>(</span>self</span>,</span> database</span>):</span></span>
    self</span>.</span>database</span> =</span> database</span></span>
    self</span>.</span>load</span>()</span></span>
</span>
  def</span> append</span>(</span>self</span>,</span> file</span>,</span> size</span>,</span> type</span>):</span></span>
    if</span> file</span> not in</span> self</span>.</span>files</span>:</span></span>
      self</span>.</span>files</span>[</span>file</span>]</span> =</span> {</span></span>
        'date'</span>:</span> datetime</span>.</span>now</span>(),</span></span>
        'size'</span>:</span> size</span>,</span></span>
        'mimetype'</span>:</span> type</span></span>
      }</span></span>
</span>
  def</span> store</span>(</span>self</span>):</span></span>
    with</span> open</span>(</span>self</span>.</span>database</span>,</span> 'w'</span>)</span> as</span> fp</span>:</span></span>
      pickle</span>.</span>dump</span>(</span>self</span>.</span>files</span>,</span> fp</span>)</span></span>
</span>
  def</span> load</span>(</span>self</span>):</span></span>
    if</span> path</span>.</span>exists</span>(</span>self</span>.</span>database</span>):</span></span>
      with</span> open</span>(</span>self</span>.</span>database</span>,</span> 'r'</span>)</span> as</span> fp</span>:</span></span>
        self</span>.</span>files</span> =</span> pickle</span>.</span>load</span>(</span>fp</span>)</span></span>
    else</span>:</span></span>
      self</span>.</span>files</span> =</span> {}</span></span>
</span>
  def</span> get</span>(</span>self</span>):</span></span>
    """</span></span>
    Return the files in reverse order in which we stored them.</span></span>
    """</span></span>
    return</span> self</span>.</span>files</span></span>
</span>
class</span> DirectoryParser</span>:</span></span>
  def</span> __init__</span>(</span>self</span>,</span> directory</span>,</span> extensions</span> =</span> []):</span></span>
    self</span>.</span>directory</span> =</span> directory</span></span>
    self</span>.</span>extensions</span> =</span> extensions</span></span>
    mimetypes</span>.</span>init</span>()</span></span>
</span>
  def</span> parse</span>(</span>self</span>,</span> store</span>):</span></span>
    for</span> _path</span>,</span> directories</span>,</span> files</span> in</span> walk</span>(</span>self</span>.</span>directory</span>):</span></span>
      for</span> file</span> in</span> files</span>:</span></span>
        if</span> self</span>.</span>filter</span>(</span>file</span>):</span></span>
          type</span> =</span> self</span>.</span>get_filetype</span>(</span>file</span>)</span></span>
          full_path</span> =</span> path</span>.</span>join</span>(</span>_path</span>,</span> file</span>)</span></span>
          size</span> =</span> path</span>.</span>getsize</span>(</span>full_path</span>)</span></span>
          storeable_path</span> =</span> full_path</span>[</span>len</span>(</span>self</span>.</span>directory</span>)</span> +</span> 1</span>:]</span></span>
          store</span>.</span>append</span>(</span>storeable_path</span>,</span> size</span>,</span> type</span>)</span></span>
</span>
  def</span> filter</span>(</span>self</span>,</span> file</span>):</span></span>
    if</span> len</span>(</span>self</span>.</span>extensions</span>)</span> is</span> 0</span>:</span></span>
      return</span> True</span></span>
</span>
    elif</span> path</span>.</span>splitext</span>(</span>file</span>)[</span>1</span>]</span> in</span> self</span>.</span>extensions</span>:</span></span>
      return</span> True</span></span>
</span>
    else</span>:</span></span>
      return</span> False</span></span>
</span>
  def</span> get_filetype</span>(</span>self</span>,</span> file</span>):</span></span>
    return</span> mimetypes</span>.</span>types_map</span>[</span>path</span>.</span>splitext</span>(</span>file</span>)[</span>1</span>]]</span></span>
</span>
class</span> RSSGenerator</span>:</span></span>
  def</span> __init__</span>(</span>self</span>):</span></span>
</span>
    if</span> HTTPS</span>:</span></span>
      self</span>.</span>url</span> =</span> 'https://'</span></span>
    else</span>:</span></span>
      self</span>.</span>url</span> =</span> 'http://'</span></span>
    self</span>.</span>url</span> +=</span> '</span>%s</span>/</span>%s</span>/</span>%%</span>s'</span> %</span> (</span>HOSTNAME</span>,</span> PATH</span>)</span></span>
</span>
  def</span> generate</span>(</span>self</span>,</span> files</span>):</span></span>
    items</span> =</span> []</span></span>
</span>
    for</span> file</span> in</span> files</span>:</span></span>
      url</span> =</span> self</span>.</span>url</span> %</span> urllib</span>.</span>quote</span>(</span>file</span>)</span></span>
      item</span> =</span> RSS2</span>.</span>RSSItem</span>(</span></span>
        title</span> =</span> file</span>,</span></span>
        link</span> =</span> url</span>,</span></span>
        guid</span> =</span> RSS2</span>.</span>Guid</span>(</span>url</span>),</span></span>
        pubDate</span> =</span> files</span>[</span>file</span>][</span>'date'</span>],</span></span>
        enclosure</span> =</span> RSS2</span>.</span>Enclosure</span>(</span>url</span>,</span> files</span>[</span>file</span>][</span>'size'</span>],</span> files</span>[</span>file</span>][</span>'mimetype'</span>])</span></span>
      )</span></span>
      items</span>.</span>append</span>(</span>item</span>)</span></span>
</span>
    self</span>.</span>rss</span> =</span> RSS2</span>.</span>RSS2</span>(</span></span>
      title</span> =</span> '</span>%s</span> downloads RSS feed'</span> %</span> HOSTNAME</span>,</span></span>
      description</span> =</span> 'The latest finished downloads at </span>%s\'</span>s ruTorrent'</span> %</span> HOSTNAME</span>,</span></span>
      link</span> =</span> self</span>.</span>url</span> %</span> ''</span>,</span></span>
      items</span> =</span> items</span></span>
    )</span></span>
</span>
  def</span> store</span>(</span>self</span>,</span> filename</span>):</span></span>
    with</span> open</span>(</span>filename</span>,</span> 'w'</span>)</span> as</span> fp</span>:</span></span>
      self</span>.</span>rss</span>.</span>write_xml</span>(</span>fp</span>)</span></span>
</span>
if</span> __name__</span> ==</span> '__main__'</span>:</span></span>
  file_list</span> =</span> FileList</span>(</span>DATABASE</span>)</span></span>
  parser</span> =</span> DirectoryParser</span>(</span>DIRECTORY</span>,</span> extensions</span> =</span> EXTENSIONS</span>)</span></span>
  parser</span>.</span>parse</span>(</span>file_list</span>)</span></span>
</span>
  file_list</span>.</span>store</span>()</span></span>
</span>
  rss_generator</span> =</span> RSSGenerator</span>()</span></span>
  rss_generator</span>.</span>generate</span>(</span>file_list</span>.</span>get</span>())</span></span>
</span>
  rss_generator</span>.</span>store</span>(</span>RSS_FILENAME</span>)</span></span></code></pre>
This was called by a cron job every 5 minutes:</p>
# Generate the downloads.rss, every 5 minutes</span></span>
*</span>/5 </span>* * * *</span> /usr/local/bin/rssdirectory.py </span>&&</span> /bin/sed</span> -i 's!></enclosure!/!g' /home/rtorrent/downloads.rss</span></span></code></pre>
The reason that sed command is there is because the Freebox doesn’t understand the enclosure tag
that is generated by PyRss2Gen. Yeah, that one took a while to debug and figure out.</p>
While this setup worked very nicely, every couple of months, the Freebox would simply stop
downloading new items. In the router’s WebOS, I’d force updates to the RSS feed, but nothing would
show up. The feed would be empty. After some time, I figured out that it was due to the Freebox not
being able to cope with the size of the RSS file. So here’s the hack-ish remedy for that, you’ve
guessed it, another cron job:</p>
# Remove the .rssdb once a week (garbage collection)</span></span>
@weekly</span> /bin/rm /home/rtorrent/.rssdb</span></span></code></pre>
This setup worked very nicely for the better part of 5 years, while we lived in France.</p>
So thank you Free, and all the engineers behind the amazing ISP that you are, and all the engineers
who make the Freebox possible. It’s an amazing piece of kit, and the hacker spirit that you guys
show all over the place is awe-inspiring, to say the least.</p>


Hash anagram challenge, part 2
2015-02-03T00:00:00+00:00
This is part two of my attempt to solve the Hash Anagram challenge offered by TrustPilot. Full
disclosure: These articles are not written in chronological order; I finished the challenge a few
weeks ago. I’m merely documenting my mental process, and trying to get back into proper writing. In
this part, we’ll start writing a small script that crunches the words, and do some rough
benchmarking on it, to see whether our approach actually makes sense or not.</p>
In part 1</a>, we saw that we have a couple thousand words to work with. We also saw that
working from the letters in the seed anagram alone, there were a few quintillion possible
permutations.</p>
So what kind of algorithms could we apply to this problem, to try and solve it? Let’s try to restate
the problem in order to see things more clearly:</p>

Given a list of 1758 words, find a permutation of n</em> words whose md5 hash computes to
4624d200580677270a54ccff86b9610e</code>.</p>
</blockquote>
First, let’s lay down some observations:</p>

There are 18 letters in the seed anagram, hence the number of letters in the target anagram will
be 18 as well;</li>
We don’t know how many words the target anagram contains;</li>
We don’t know how many permutations of the words list are actual anagrams.</li>
</ol>
Technically, we could assume that the target anagram contains nothing more than three words, but I
don’t want to make that assumption.</p>
I see two main algorithmic paths:</p>

Generate all possible word permutations, and filter it down;</li>
Recursively append words to one another until an anagram is found.</li>
</ul>
So how do you check whether a string is an anagram of another string? I’ve found the following
algorithm in Python:</p>
def</span> isAnagram</span>(</span>first</span>,</span> second</span>):</span></span>
    # If the strings are identical, they are anagrams</span></span>
    if</span> first</span> ==</span> second</span>:</span></span>
        return</span> True</span></span>
</span>
    # Remove spaces, and convert `first` into a list</span></span>
    first</span> =</span> first</span>.</span>translate</span>({</span>ord</span>(</span>' '</span>):</span> None</span>})</span></span>
    second</span> =</span> second</span>.</span>translate</span>({</span>ord</span>(</span>' '</span>):</span> None</span>})</span></span>
</span>
    # If they don't have the same length, they can't be anagrams</span></span>
    if</span> len</span>(</span>first</span>)</span> !=</span> len</span>(</span>second</span>):</span></span>
        return</span> False</span></span>
</span>
    else</span>:</span></span>
        first</span> =</span> list</span>(</span>first</span>)</span></span>
</span>
        try</span>:</span></span>
            # Remove all characters found in second</span></span>
            map</span>(</span>first</span>.</span>remove</span>,</span> second</span>)</span></span>
        except</span> ValueError</span>:</span></span>
            return</span> False</span></span>
        else</span>:</span></span>
            return</span> True</span></span></code></pre>
And an equivalent implementation in C:</p>
// Return a copy of the letters a string (without spaces)</span></span>
char *</span> lttrdup</span>(</span>char const *</span> str</span>) {</span></span>
  // Technically, we're wasting a few bytes here, but that's OK.</span></span>
  char</span> *</span> copy </span>=</span> malloc</span>(</span>strlen</span>(</span>str</span>)</span> +</span> 1</span>);</span></span>
  char</span> *</span> tmp </span>=</span> copy</span>;</span></span>
</span>
  do</span> {</span></span>
    if</span> (</span>*</span>str </span>!=</span> ' '</span>)</span></span>
      *</span>tmp</span>++ = *</span>str</span>;</span></span>
  }</span> while</span> (</span>*</span>str</span>++ !=</span> '</span>\0</span>'</span>);</span></span>
</span>
  return</span> copy</span>;</span></span>
}</span></span>
</span>
bool</span> is_anagram</span>(</span>char const *</span> first</span>,</span> char const *</span> second</span>) {</span></span>
  // If the strings are equivalent, they are anagrams</span></span>
  if</span> (</span>strcmp</span>(</span>first</span>,</span> second</span>)</span> ==</span> 0</span>)</span></span>
    return</span> true</span>;</span></span>
</span>
  else</span> {</span></span>
    // Remove the spaces from our strings</span></span>
    char</span> *</span> cpy_first </span>=</span> lttrdup</span>(</span>first</span>);</span></span>
    char</span> *</span> cpy_second </span>=</span> lttrdup</span>(</span>second</span>);</span></span>
</span>
    // tmp_second is an iterator over cpy_second</span></span>
    char</span> *</span> tmp_second </span>=</span> cpy_second</span>;</span></span>
</span>
    // If both strings don't have the same length, they can't be anagrams</span></span>
    if</span> (</span>strlen</span>(</span>cpy_first</span>)</span> ==</span> strlen</span>(</span>cpy_second</span>)) {</span></span>
</span>
      // Iterate over cpy_second until the end is reached</span></span>
      while</span> (</span>*</span>tmp_second </span>!=</span> '</span>\0</span>'</span>) {</span></span>
        // find the character pointed by tmp_second in cpy_first</span></span>
        char</span> *</span> match </span>=</span> strchr</span>(</span>cpy_first</span>,</span> *</span>tmp_second</span>);</span></span>
</span>
        // if a match is found, overwrite it, and move on the to the</span></span>
        // next character</span></span>
        if</span> (</span>match </span>!=</span> NULL</span>) {</span></span>
          *</span>match </span>=</span> '.'</span>;</span></span>
          tmp_second</span>++</span>;</span></span>
        }</span></span>
</span>
        // no match found, skip to cleanup</span></span>
        else</span></span>
          break</span>;</span></span>
      }</span></span>
</span>
    }</span></span>
</span>
    // if tmp_second doesn't point to the end of the string, we found a</span></span>
    // character that doesn't exist in first, hence, no anagram</span></span>
    bool</span> result </span>= *</span>tmp_second </span>==</span> '</span>\0</span>'</span>;</span></span>
</span>
    // cleanup</span></span>
    free</span>(</span>cpy_first</span>);</span></span>
    free</span>(</span>cpy_second</span>);</span></span>
</span>
    return</span> result</span>;</span></span>
  }</span></span>
}</span></span></code></pre>
The main issue I see with both implementations is that we actually need to do allocations in order
to verify that two strings are anagrams of each other. Granted, the allocations in C are way smaller
and more manageable than the Python ones, but still, not great. If you know of an algorithm that
doesn’t require allocations, I’d be happy to hear about it.</p>
Talking about allocations, what could we do to reduce their numbers? At the moment, the only check
we do before undergoing any allocation, both in Python and C, is checking whether both strings are
the same. How could we cut that down? This is my train of thought:</p>
Characters are encoded into their ASCII value, and for lowercase letters in the alphabet, their
value will range between 97 and 122. What happens if we add the values of our seed anagram (letters
only, no spaces)?</p>
anagram</span> =</span> "poultry outwits ants"</span>.</span>translate</span>(</span>None</span>,</span> ' '</span>)</span></span>
sum</span>(</span>map</span>(</span>ord</span>,</span> anagram</span>))</span></span>
=></span> 2036</span></span></code></pre>
Regardless of the order of the letters, an anagram of “poultry outwits ants” will always have the
same order point sum. The reverse, however, isn’t true: a string that has an order point sum value
of 2036 isn’t necessarily an anagram of “poultry outwits ants”. Let’s throw that into our code:</p>
def</span> letterCount</span>(</span>word</span>):</span></span>
    return</span> sum</span>(</span>map</span>(</span>lambda</span> x</span>:</span> 1</span> if</span> x</span> !=</span> ' '</span> else</span> 0</span>,</span> word</span>))</span></span>
</span>
def</span> sumLetters</span>(</span>word</span>):</span></span>
    return</span> sum</span>(</span>map</span>(</span>lambda</span> x</span>:</span> ord</span>(</span>x</span>)</span> if</span> x</span> !=</span> ' '</span> else</span> 0</span>,</span> word</span>))</span></span>
</span>
def</span> isPlausibleAnagram</span>(</span>first</span>,</span> second</span>):</span></span>
    return</span> letterCount</span>(</span>first</span>)</span> ==</span> letterCount</span>(</span>second</span>)</span> and</span> \</span></span>
        sumLetters</span>(</span>first</span>)</span> ==</span> sumLetters</span>(</span>second</span>)</span></span></code></pre>
Excellent. Let’s try to do the same in C:</p>
bool</span> is_plausible_anagram</span>(</span>char const *</span> first</span>,</span> char const *</span> second</span>) {</span></span>
  unsigned int</span> first_length </span>=</span> 0</span>;</span></span>
  unsigned int</span> second_length </span>=</span> 0</span>;</span></span>
  unsigned int</span> first_ord </span>=</span> 0</span>;</span></span>
  unsigned int</span> second_ord </span>=</span> 0</span>;</span></span>
</span>
  do</span> {</span></span>
    if</span> (</span>*</span>first </span>!=</span> ' '</span>) {</span></span>
      first_ord </span>+= *</span>first</span>;</span></span>
      first_length</span>++</span>;</span></span>
    }</span></span>
  }</span> while</span> (</span>*</span>first</span>++ !=</span> '</span>\0</span>'</span>);</span></span>
</span>
  do</span> {</span></span>
    if</span> (</span>*</span>second </span>!=</span> ' '</span>) {</span></span>
      second_ord </span>+= *</span>second</span>;</span></span>
      second_length</span>++</span>;</span></span>
    }</span></span>
  }</span> while</span> (</span>*</span>second</span>++ !=</span> '</span>\0</span>'</span>);</span></span>
</span>
  return</span> first_length </span>==</span> second_length </span>&&</span> first_ord </span>==</span> second_ord</span>;</span></span>
}</span></span></code></pre>
I decided to write the C version as a single function, so as to only have to loop over the two
strings once. The Python version uses a more functional approach.</p>
Great, so we now have a way to approximate whether a string might be an anagram without doing any
allocations, and a way to know whether a string really is an anagram of another string. Let’s put
together a quick script that gives us all the permutations and starts pumping out anagrams, but
first, let’s use our newfound quicktester to spare some CPU cycles:</p>
def</span> isAnagram</span>(</span>first</span>,</span> second</span>):</span></span>
    if not</span> isPlausibleAnagram</span>(</span>first</span>,</span> second</span>):</span></span>
        return</span> False</span></span>
</span>
    # If the strings are identical, they are anagrams</span></span>
    if</span> first</span> ==</span> second</span>:</span></span>
        return</span> True</span></span>
</span>
    first</span> =</span> first</span>.</span>translate</span>({</span>ord</span>(</span>' '</span>):</span> None</span>})</span></span>
    second</span> =</span> second</span>.</span>translate</span>({</span>ord</span>(</span>' '</span>):</span> None</span>})</span></span>
</span>
    # If they don't have the same length, they can't be anagrams</span></span>
    if</span> len</span>(</span>first</span>)</span> !=</span> len</span>(</span>second</span>):</span></span>
        return</span> False</span></span>
</span>
    else</span>:</span></span>
        first</span> =</span> list</span>(</span>first</span>)</span></span>
</span>
        try</span>:</span></span>
            # Remove all characters found in second</span></span>
            map</span>(</span>first</span>.</span>remove</span>,</span> second</span>)</span></span>
        except</span> ValueError</span>:</span></span>
            return</span> False</span></span>
        else</span>:</span></span>
            return</span> True</span></span></code></pre>
And now for the main function:</p>
def</span> main</span>():</span></span>
    words_url</span> =</span> 'http://followthewhiterabbit.trustpilot.com/cs/wordlist'</span></span>
    anagram</span> =</span> 'poultryoutwitsants'</span></span>
</span>
    words</span> =</span> getEnglishWords</span>(</span>words_url</span>)</span></span>
    words</span> =</span> filterWords</span>(</span>words</span>,</span> anagram</span>)</span></span>
</span>
    length</span> =</span> 0</span></span>
    while</span> True</span>:</span></span>
        length</span> +=</span> 1</span></span>
        print</span>(</span>"Testing permutations of </span>%d</span> words"</span> %</span> length</span>)</span></span>
        generator</span> =</span> itertools</span>.</span>permutations</span>(</span>words</span>,</span> length</span>)</span></span>
</span>
        for</span> permutation</span> in</span> generator</span>:</span></span>
            permutation</span> =</span> ' '</span>.</span>join</span>(</span>permutation</span>)</span></span>
            if</span> isAnagram</span>(</span>permutation</span>,</span> anagram</span>):</span></span>
                print</span>(</span>"--> Found anagram:"</span>,</span> permutation</span>)</span></span></code></pre>
I let this run for a few minutes, and as luck would have it, it actually spat out the answer:</p>
python permutations.py</span></span>
Splitting file into list()</span></span>
Removing words that contain letters not found in anagram</span></span>
Original set of words length: 99174</span></span>
Filtered set of words length: 1658</span></span>
Testing permutations of 1 words</span></span>
Testing permutations of 2 words</span></span>
Testing permutations of 3 words</span></span>
--> Found anagram: want spit tortuously</span></span>
--> Found anagram: want pursuit tolstoy</span></span>
--> Found anagram: want tortuously spit</span></span>
--> Found anagram: want tortuously pits</span></span>
--> Found anagram: want tortuously tips</span></span>
--> Found anagram: want pits tortuously</span></span>
--> Found anagram: want tolstoy pursuit</span></span>
--> Found anagram: want lousy outstript</span></span>
--> Found anagram: want yous trustpilot</span></span>
--> Found anagram: want trustpilot yous</span></span></code></pre>
Wait, trustpilot</code>? Would the answer just be “trustpilot wants you</em>”? Nevermind, I haven’t tested
any md5 hashes or anything, so I won’t cheat. The major issue with using itertools.permutations</code> is
that it’s pretty difficult to run on a cluster, and looking at the speed, it’s not going to complete
quickly. What I mean by speed is that a quick benchmark shows that the Python script running on my
netbook is capable of rougly 50k permutations per second. Going back to the number crunching we did
in part 1, how many different permutations are there for n</em> ranging from 1 to 5?</p>
1658! / (1658-1)! + 1658! / (1658-2)! + 1658! / (1658-3)! + \</span></span>
    1658! / (1658-4)! + 1658! / (1658-5)!</span></span>
=> 12 461 304 888 660 100</span></span></code></pre>
I don’t even need to calculate how long that would take to know it would be a very, very long time
(hint: they should’ve started back in the Copper Age</a>). Now obviously, I wouldn’t
need to test every single</em> permutation; especially if I check the md5 hash every time I find an
anagram.</p>
So how can we split the work across multiple cores? I know just the library for the job:
ZeroMQ</a>. ZeroMQ (also written ØMQ) is a great tool to have on your belt. It doesn’t
require much, other than a good ol’ paradigm shift (POP!).</p>
</p>
ZeroMQ makes multithreading and multiprocessing easy. Well, it doesn’t, it just makes communicating
between multiple threads, processes or servers a piece of cake. The PyZMQ bindings</a> are
excellent, and they make everything an absolute pleasure to work with. ZeroMQ takes care of managing
the sockets, establishing or receiving connections, polling and whatnot. Instead of working with
dumb sockets, ZeroMQ offers pre-defined socket types that behave in certain ways (request/response,
router, publish/subscribe, etc.).</p>
Here’s the architecture we’ll deploy, simple and concise:</p>

The server will take care of downloading and reducing the wordlist;</li>
When a client asks for a word, the server will sequentially go through its words list, and send
the next word to a client;</li>
Upon exhausting all possibilities for a given word, the client will send back all the anagrams it
found, and request a new word.</li>
</ul>
Later on, we’ll teach clients to also switch to md5 computing tasks if there are any. This will
allow us to not spend the next 8000 years computing, but also completing the challenge.</p>
ZeroMQ gives us a socket we can talk on between the clients and the server, but what dialect should
we employ? These days, I usually default to JSON; it’s very well supported and can encode nearly all
the types of data one usually needs. It compresses fairly well, and isn’t too verbose, yet remains
human-readable. And hey look! PyZMQ offers convenience functions to read/write in JSON directly!</p>
The other question we should probably answer is “What kind of sockets are we going to use?” For the
sake of simplicity, we’re going to use one of ZeroMQ’s simplest socket types: the Request-Response
combo. Let’s take a moment to define the grammar of our dialect:</p>
--> {'type': 'initialise'}</span></span>
<-- {'seed': 'poultryoutwitsants', 'words': ['a', 'abaci', 'aback', ... ]}</span></span>
</span>
--> {'type': 'next_word'}</span></span>
<-- {'word': 'truants'}</span></span>
</span>
--> {'type': 'found_anagrams', 'result': ['truants outwit slopy', ...]}</span></span>
<-- {}</span></span></code></pre>
Let’s start by creating our server. Most of the stuff will remain the same, we’re just going to add
a small class that keeps track of the current word, and stuff like that:</p>
class</span> WordsList</span>(</span>object</span>):</span></span>
    """I download a list of words, clean it up and iterate through it."""</span></span>
</span>
    def</span> __init__</span>(</span>self</span>,</span> filter</span>,</span> url</span>):</span></span>
        self</span>.</span>_words</span> =</span> getEnglishWords</span>(</span>url</span>)</span></span>
        self</span>.</span>_words</span> =</span> filterWords</span>(</span>self</span>.</span>_words</span>,</span> filter</span>)</span></span>
    # A copy for future reference</span></span>
    self</span>.</span>_all_words</span> =</span> list</span>(</span>self</span>.</span>_words</span>)</span></span>
</span>
    def</span> nextWord</span>(</span>self</span>):</span></span>
        """I give the next word in the words list, or None if the end has been</span></span>
        reached"""</span></span>
</span>
        try</span>:</span></span>
            return</span> self</span>.</span>_words</span>.</span>pop</span>()</span></span>
        except</span> IndexError</span>:</span></span>
            return</span> None</span></span>
</span>
    def</span> getWords</span>(</span>self</span>):</span></span>
        """I give access to the whole words list."""</span></span>
        return</span> self</span>.</span>_all_words</span></span>
</span>
    def</span> endReached</span>(</span>self</span>):</span></span>
        """I'm a helper function that tells you whether the end of the list has</span></span>
        been reached or not"""</span></span>
        return</span> len</span>(</span>self</span>.</span>_words</span>)</span> ==</span> 0</span></span></code></pre>
And this is again where I’m going to call it a night ;) I’ll finish writing this article tomorrow or
something. Thanks for reading, anyway.</p>


Hash anagram challenge, part 1
2015-02-02T00:00:00+00:00
As I mentioned previously, I’ve been working on a technical challenge I found a few days ago. Just
to be clear: this is a low-priority challenge for me. I don’t expect anything out of this, and I
don’t consider it to be anything more than yet another toy-project to keep my mind busy whenever I
have a moment to spare.</p>
The challenge</a> is as follows:</p>

What is the anagram of “poultry outwits ants” whose md5 hash computes down to
4624d200580677270a54ccff86b9610e</code>?</p>
</blockquote>
Sounds simple enough, right? That’s what I thought as well. Let’s start by doing some math:</p>
There are 18 letters in the seed anagram, not including spaces. Let’s leave the spaces aside for the
moment and just focus on those 18 letters. How many permutations of letters are there? Well, that’s
fairly easy, let’s head up to WolframAlpha</a> and calculate some
factorials</a>:</p>
18!</span></span>
=> 6 402 373 705 728 000</span></span></code></pre>
Just shy of 6.5 quadrillion</a> possibilities? Not bad. Now, let’s see what throwing in
those spaces changes…</p>
21!</span></span>
=> 51 090 942 171 709 440 000</span></span>
</span>
23!</span></span>
=> 25 852 016 738 884 976 640 000</span></span></code></pre>
Between 51 quintillion</a> possibilities and 25 sixtillion possibilities</a>,
assuming there are respectively 3 to 5 words in the actual anagram. Wow. We now know how many
possible permutations there are. These are purely permutations (e.g.: theoretical anagrams), and not
actually matched to real, existing words. That’s another step. Let’s just go with these numbers and
see how long it would take to calculate all of these.</p>
How fast is my computer? I’m writing this on a small, cheap netbook that runs off battery power. I’m
not expecting much out of it. Let’s see what John the ripper</a> thinks of my CPU.</p>
$ john -test | grep -A md5</span></span>
Benchmarking: md5crypt [MD5 32/64 X2]... DONE</span></span>
Raw: 5383 c/s real, 5427 c/s virtual</span></span></code></pre>
I’m taking md5crypt as a low baseline, as I’m guesstimating it to be more computationally intensive
than just calculating the md5 sum of small strings. Let’s assume I get a thousand boxes like mine,
slightly more powerful (four cores instead of two):</p>
5000 hashes/s * 1 000 000 * 4 cores</span></span>
=> 20 000 000 000</span></span></code></pre>
20 billion hashes per second, that’s pretty good, and in our era of cloud computing, surely not that
impossible to achieve. How long would it take me to crack the challenge, by simple, honest to
goodness bruteforce?</p>
21! / (5000 * 4 * 1 000 000 per second)</span></span>
=> 709 596 hours</span></span>
=> 29 567 days</span></span></code></pre>
So, assuming perfect workload distribution, and assuming no overhead from the software, it would
take around 80 years</a> to calculate the md5 hash of every single possible anagram.
Now, I realise those are very, very low values; maybe more worthy of vapour that cloud computing.
According to Andrew Dunham</a>, it is possible to get up to 2.3 billion md5 hashes per second
per Amazon EC2 GPU cluster. So assuming an hourly rate of USD 0.7, it would only cost under USD 5
million</a> to calculate everything, but it would still take nearly as
long</a> as to put a human on this planet. Note that those numbers are for 1000 GPU
clusters; The reason I didn’t go for the million-figure in this estimate is because I somehow
strongly doubt the fact that Amazon has a spare million GPU clusters lying around.</p>
I love a good challenge as much as the next guy, but I don’t have that kind of time or money to
throw at just something to keep my brain idling. So where do we go from now? Let’s look at the
challenge again. Hey, they gave us a list of words</a>! That must</em> be useful, right? Of
course it is!</p>
How many words did they give us?</p>
$</span> wc -l wordlist</span></span>
99174</span> wordlist</span></span></code></pre>
Wait, how many unique words did they give us?</p>
$</span> sort -u wordlist</span> |</span> wc</span> -l</span></span>
96316</span></span></code></pre>
OK, so let’s remove any word that contains a letter that doesn’t exist in our seed anagram: poultry
outwits ants</em>. Time to let go of bash and friends, and call in the big guns: Python.</p>
#!/usr/bin/env python</span></span>
</span>
from</span> __future__</span> import</span> (</span>absolute_import</span>,</span> print_function</span>,</span></span>
                        unicode_literals</span>)</span></span>
</span>
import</span> os</span>.</span>path</span></span>
import</span> urllib2</span></span>
</span>
</span>
"""</span></span>
Unoptimised file downloader with minimal cache management. It expects a</span></span>
newline-delimited file and removes any line that contains letters not found in</span></span>
an arbritrary string</span></span>
"""</span></span>
</span>
</span>
def</span> fileFromHTTP</span>(</span>url</span>,</span> filename</span>):</span></span>
    """I download a file and save it to the provided filename"""</span></span>
</span>
    print</span>(</span>"Downloading file:"</span>,</span> url</span>)</span></span>
    file</span> =</span> urllib2</span>.</span>urlopen</span>(</span>url</span>).</span>read</span>()</span></span>
</span>
    with</span> open</span>(</span>filename</span>,</span> 'wb'</span>)</span> as</span> fp</span>:</span></span>
        fp</span>.</span>write</span>(</span>file</span>)</span></span>
</span>
</span>
def</span> fileToList</span>(</span>file</span>):</span></span>
    """I know how split a file into a list(). Yay me."""</span></span>
</span>
    print</span>(</span>"Splitting file into list()"</span>)</span></span>
    lines</span> =</span> [</span>x</span>.</span>decode</span>(</span>'utf-8'</span>)</span> for</span> x</span> in</span> file</span>.</span>splitlines</span>()]</span></span>
    return</span> lines</span></span>
</span>
</span>
def</span> getEnglishWords</span>(</span>url</span>):</span></span>
    """I check if the cache is warm, and download the file if it can't be found</span></span>
    locally"""</span></span>
</span>
    filename</span> =</span> os</span>.</span>path</span>.</span>basename</span>(</span>url</span>)</span></span>
</span>
    if not</span> os</span>.</span>path</span>.</span>isfile</span>(</span>filename</span>):</span></span>
        fileFromHTTP</span>(</span>url</span>,</span> filename</span>)</span></span>
</span>
    with</span> open</span>(</span>filename</span>,</span> 'rb'</span>)</span> as</span> fp</span>:</span></span>
        file</span> =</span> fp</span>.</span>read</span>()</span></span>
</span>
    return</span> fileToList</span>(</span>file</span>)</span></span>
</span>
</span>
def</span> filterWords</span>(</span>haystack</span>,</span> needles</span>):</span></span>
    """I take a a list of words, and remove any that can't be found in the</span></span>
    needles string"""</span></span>
</span>
    print</span>(</span>"Removing words that contain letters not found in anagram"</span>)</span></span>
    print</span>(</span>"Original set of words length:"</span>,</span> len</span>(</span>haystack</span>))</span></span>
</span>
    # The list(set()) operation is done to remove dupes</span></span>
    haystack</span> =</span> list</span>(</span>set</span>(</span>haystack</span>))</span></span>
</span>
    for</span> word</span> in</span> list</span>(</span>haystack</span>):</span></span>
        if not</span> existsInPool</span>(</span>needles</span>,</span> word</span>):</span></span>
            haystack</span>.</span>remove</span>(</span>word</span>)</span></span>
</span>
    print</span>(</span>"Filtered set of words length:"</span>,</span> len</span>(</span>haystack</span>))</span></span>
    return</span> haystack</span></span>
</span>
</span>
def</span> existsInPool</span>(</span>pool</span>,</span> word</span>):</span></span>
    """I check whether `word` could possibly exist in `pool`"""</span></span>
    pool</span> =</span> list</span>(</span>pool</span>)</span></span>
</span>
    try</span>:</span></span>
        for</span> c</span> in</span> word</span>:</span></span>
            pool</span>.</span>remove</span>(</span>c</span>)</span></span>
    except</span> ValueError</span>:</span></span>
        return</span> False</span></span>
</span>
    return</span> True</span></span>
</span>
</span>
def</span> main</span>():</span></span>
    words_url</span> =</span> 'http://followthewhiterabbit.trustpilot.com/cs/wordlist'</span></span>
    anagram</span> =</span> 'poultryoutwitsants'</span></span>
</span>
    words</span> =</span> getEnglishWords</span>(</span>words_url</span>)</span></span>
    words</span> =</span> filterWords</span>(</span>words</span>,</span> anagram</span>)</span></span>
</span>
</span>
if</span> __name__</span> ==</span> '__main__'</span>:</span></span>
    main</span>()</span></span></code></pre>
Let’s give it a run and see how many words we managed to squash out:</p>
$</span> python foo.py</span></span>
Splitting</span> file into list</span>()</span></span>
Removing</span> words that contain letters not found in anagram</span></span>
Original</span> set of words length:</span> 99174</span></span>
Filtered</span> set of words length:</span> 1658</span></span></code></pre>
OK, we managed to remove 98.32% of the words list provided by Trustpilot. This definitely reduces
the amount of calculations we’re going to have to perform, but it doesn’t really make things that
much easier. Let’s look at why.</p>
The usual way to calculate the number of permutations will again be based around factorials, as we
used previously. As before, let’s assume the target anagram is made of 3 words, just like the seed
anagram. What would be the formula to calculate that? Well actually, the Python
documentation</a> gives it right to us:</p>
# for n the pool length, and r the number of permutations:</span></span>
n! / (n-r)!</span></span></code></pre>
So applied to our problem:</p>
1658! / (1658 - 3)!</span></span>
=> 4 549 538 736</span></span>
</span>
# 18 bytes per anagram, 3 bytes for spaces, 32 bytes for the md5 hash</span></span>
4549538736 * (18 bytes + 3 bytes + 32 bytes)</span></span>
=> 241 125 553 008</span></span></code></pre>
I guess it would be possible to load up a Cassandra cluster with 240GB of data and use Apache Spark
to carve through it, but it doesn’t really seem like the right way to do it.</p>
In the next article (part 2</a>), we’ll write a quick python script that starts crunching
through the list of words and find a way to discard impossible permutations.</p>


RPM versioning with Jenkins
2014-02-01T00:00:00+00:00
I’m currently deploying Jenkins at work, in order to simplify the build process. At the moment, we
have a fairly complicated process that involves a couple dozen steps, which, even though each step
is fairly braindead, does require a good amount of focus and planning in order to get everything
just right.</p>
Jenkins</a> provides a build number for all the different projects you might configure. Each build
number is unique in a given project; this ensures that you can refer to specific builds without too
much of a hassle. That’s great, but knowing that I’m trying to achieve fully automated RPM
packaging, it doesn’t help me much.</p>
The versioning system in our company (defined by moi, pretty much), is heavily inspired by the usual
RPM rules: release numbers are reset to 1 whenever the version number changes. There’s no real way,
as far as I know, to use Jenkins’ build number for this, as there is no programmatic way to reset
the build number in Jenkins.</p>
Multi-instance exclusive lock in Python</h1>
The first issue I tried to tackle was the fact that the Jenkins server will run multiple builds at
the same time, possibly for the same project. What this means is that simply having stuff in a file
isn’t really sensible due to concurrency issues. I know that portalocker</a> roughly does the same
as what I tried to do, but not completely. portalocker provides a lock to a file; my implementation
provides a transaction-kinda lock, regardless of whether you want to access files or databases or
whatever.</p>
Without further ado, here’s the class:</p>
import</span> os</span>,</span> time</span></span>
</span>
class</span> Lock</span>(</span>object</span>):</span></span>
    """</span></span>
    Manage a system-wide lock used to inform other processes they should wait.</span></span>
    The lockfile argument is the token that should be shared amongst all the</span></span>
    processes wanting to acquire exclusive locks to the same resource.</span></span>
    """</span></span>
</span>
    polling</span> =</span> 0.1</span></span>
</span>
    def</span> __init__</span>(</span>self</span>,</span> lockfile</span>):</span></span>
        self</span>.</span>_lockfile</span> =</span> '</span>%s</span>.lock'</span> %</span> lockfile</span></span>
        # I only target linux systems... Sorry.</span></span>
        self</span>.</span>_path</span> =</span> os</span>.</span>path</span>.</span>join</span>(</span>'/tmp'</span>,</span> self</span>.</span>_lockfile</span>)</span></span>
        self</span>.</span>locked</span> =</span> False</span></span>
</span>
    def</span> __enter__</span>(</span>self</span>):</span></span>
        self</span>.</span>acquire</span>()</span></span>
        return</span> self</span></span>
</span>
    def</span> __exit__</span>(</span>self</span>,</span> _</span>,</span> __</span>,</span> ___</span>):</span></span>
        self</span>.</span>release</span>()</span></span>
</span>
    def</span> acquire</span>(</span>self</span>):</span></span>
        """</span></span>
        Try to acquire a lock. This is a blocking call, until the lock is</span></span>
        acquired. If you want a non-blocking call, try nb_acquire().</span></span>
        """</span></span>
        if</span> self</span>.</span>locked</span>:</span></span>
            """</span> TODO</span>: Raise """</span></span>
            pass</span></span>
</span>
        while not</span> self</span>.</span>_acquire</span>():</span></span>
            time</span>.</span>sleep</span>(</span>self</span>.</span>polling</span>)</span></span>
</span>
        self</span>.</span>locked</span> =</span> True</span></span>
</span>
    def</span> nb_acquire</span>(</span>self</span>):</span></span>
        if</span> self</span>.</span>locked</span>:</span></span>
            """</span> TODO</span>: Raise """</span></span>
            pass</span></span>
</span>
        self</span>.</span>locked</span> =</span> self</span>.</span>_acquire</span>()</span></span>
        return</span> self</span>.</span>locked</span></span>
</span>
    def</span> _acquire</span>(</span>self</span>):</span></span>
        try</span>:</span></span>
            os</span>.</span>mkdir</span>(</span>self</span>.</span>_path</span>)</span></span>
        except</span> OSError</span>:</span></span>
            return</span> False</span></span>
        return</span> True</span></span>
</span>
    def</span> release</span>(</span>self</span>):</span></span>
        if not</span> self</span>.</span>locked</span>:</span></span>
            """</span> TODO</span>: Raise """</span></span>
            pass</span></span>
</span>
        os</span>.</span>rmdir</span>(</span>self</span>.</span>_path</span>)</span></span>
</span>
        self</span>.</span>locked</span> =</span> False</span></span></code></pre>
There’s a couple of TODOs, and I might push this stuff to my github account at some point or
another, but it gives a general idea on what I tried to do. The jist of it is that I use mkdir as an
atomic locking operation. What you lock against is basically just an identifier, which means that
whether you want to read/write to files, access a database, fiddle your toes doesn’t realy matter.
If the lock acquisition fails, the acquire()</code> method will block until it is able to acquire the
lock. If you don’t want it to block, there’s always nb_acquire()</code> which will just return a boolean
indicating the lock status.</p>
How to use it</h2>
Here’s how it could be used:</p>
lock_name</span> =</span> 'my-lock'</span></span>
</span>
# Primary usage</span></span>
with</span> Lock</span>(</span>lock_name</span>):</span></span>
    print</span> "We should be locked!"</span></span></code></pre>
And that’s really just it. The locking magic happens in the with</code> statement. As soon you enter the
with</code> statement, you are guaranteed to be the sole executor of whatever code you have running. As
soon as you leave the scope of the with</code> statement, the lock is released. Below are a few other
scenarios of how you could use the above class:</p>
lock_name</span> =</span> 'my-lock'</span></span>
</span>
# Another kind of usage</span></span>
lock</span> =</span> Lock</span>(</span>lock_name</span>)</span></span>
lock</span>.</span>acquire</span>()</span></span>
print</span> "We should be locked!"</span></span>
</span>
# If you do not call the following function, you're going to have bugs.</span></span>
lock</span>.</span>release</span>()</span></span>
</span>
# Yet another kind of usage (non-blocking, this time)</span></span>
if</span> lock</span>.</span>nb_acquire</span>():</span></span>
    print</span> "We should be locked!"</span></span>
    lock</span>.</span>release</span>()</span></span>
else</span>:</span></span>
    print</span> "We're most definitely not locked!"</span></span></code></pre>
So, now we can execute a block of code without having to worry about other instances of the script
mucking about our stuff. Let’s work on the RPM versioning now.</p>
Efficient RPM release numbers</h1>
RPM versioning works like this: 1.2.3-4</code> where 1.2.3</code> is the version number, and 4</code> is the
release. For a more in-depth overview of RPM versioning, check out Fedora’s wiki, or something. I
usually put the version number of my software in a file somewhere, so getting that isn’t usually a
problem. However, the release part of the RPM version is a bit more tricky. The way we use it is
that every time we release something to the QA team (or directly to production if it’s a very urgent
bugfix), we increment the release number if the version number hasn’t changed. This is the case, for
example, if you have to fix something in the packaging process, or if you are applying successive
fixes to the release branch (according to nvie’s branching model</a>).</p>
This also means that as soon as the version number changes, the release number should be reset to 1.
I wrote the following tool, which uses the Lock</code> class above, to keep track of all the different
releases that have been built so far, based on the package name and version number:</p>
import</span> argparse</span>,</span> json</span></span>
from</span> lock</span> import</span> Lock</span></span>
from</span> os</span>.</span>path</span> import</span> expanduser</span>,</span> join</span></span>
</span>
class</span> RPMReleases</span>(</span>object</span>):</span></span>
</span>
    def</span> __init__</span>(</span>self</span>):</span></span>
        self</span>.</span>_path</span> =</span> join</span>(</span>expanduser</span>(</span>'~'</span>),</span> 'iv-rpm-releases'</span>)</span></span>
        self</span>.</span>_releases</span> =</span> {}</span></span>
</span>
    def</span> __enter__</span>(</span>self</span>):</span></span>
        self</span>.</span>load</span>()</span></span>
        return</span> self</span></span>
</span>
    def</span> __exit__</span>(</span>self</span>,</span> _</span>,</span> __</span>,</span> ___</span>):</span></span>
        self</span>.</span>dump</span>()</span></span>
</span>
    def</span> load</span>(</span>self</span>):</span></span>
        try</span>:</span></span>
            with</span> open</span>(</span>self</span>.</span>_path</span>,</span> 'r'</span>)</span> as</span> f</span>:</span></span>
                self</span>.</span>_releases</span> =</span> json</span>.</span>load</span>(</span>f</span>)</span></span>
        except</span> IOError</span>:</span></span>
            pass</span></span>
</span>
    def</span> dump</span>(</span>self</span>):</span></span>
        with</span> open</span>(</span>self</span>.</span>_path</span>,</span> 'w+'</span>)</span> as</span> f</span>:</span></span>
            json</span>.</span>dump</span>(</span>self</span>.</span>_releases</span>,</span> f</span>)</span></span>
</span>
    def</span> get_next_release</span>(</span>self</span>,</span> package</span>,</span> version</span>,</span> increment</span> =</span> True</span>):</span></span>
        if</span> package</span> not in</span> self</span>.</span>_releases</span>:</span></span>
            self</span>.</span>_releases</span>[</span>package</span>]</span> =</span> {}</span></span>
</span>
        if</span> version</span> not in</span> self</span>.</span>_releases</span>[</span>package</span>]:</span></span>
            self</span>.</span>_releases</span>[</span>package</span>][</span>version</span>]</span> =</span> 0</span></span>
</span>
        if</span> increment</span>:</span></span>
            self</span>.</span>_releases</span>[</span>package</span>][</span>version</span>]</span> +=</span> 1</span></span>
</span>
        return</span> self</span>.</span>_releases</span>[</span>package</span>][</span>version</span>]</span></span>
</span>
    def</span> set_release</span>(</span>self</span>,</span> package</span>,</span> version</span>,</span> release</span>):</span></span>
        if</span> package</span> not in</span> self</span>.</span>_releases</span>:</span></span>
            self</span>.</span>_releases</span>[</span>package</span>]</span> =</span> {}</span></span>
</span>
        self</span>.</span>_releases</span>[</span>package</span>][</span>version</span>]</span> =</span> release</span></span>
</span>
if</span> __name__</span> ==</span> '__main__'</span>:</span></span>
    parser</span> =</span> argparse</span>.</span>ArgumentParser</span>()</span></span>
    parser</span>.</span>add_argument</span>(</span>"package"</span>,</span></span>
        help</span> =</span> "The software package for which you want a release number"</span>)</span></span>
    parser</span>.</span>add_argument</span>(</span>"version"</span>,</span></span>
        help</span> =</span> "The version of the package for which you want a release number"</span>)</span></span>
</span>
    parser</span>.</span>add_argument</span>(</span>"--current"</span>,</span> action</span> =</span> "store_true"</span>,</span></span>
        help</span> =</span> "Don't increment any counters, just print the current value"</span>)</span></span>
    parser</span>.</span>add_argument</span>(</span>"--full"</span>,</span> action</span> =</span> "store_true"</span>,</span></span>
        help</span> =</span> "Print the full package name"</span>)</span></span>
</span>
    args</span> =</span> parser</span>.</span>parse_args</span>()</span></span>
</span>
    with</span> Lock</span>(</span>'iv-rpm-releases'</span>):</span></span>
        with</span> RPMReleases</span>()</span> as</span> releases</span>:</span></span>
            inc</span> =</span> not</span> args</span>.</span>current</span></span>
            release</span> =</span> releases</span>.</span>get_next_release</span>(</span>args</span>.</span>package</span>,</span> args</span>.</span>version</span>,</span> increment</span> =</span> inc</span>)</span></span>
</span>
            if not</span> args</span>.</span>full</span>:</span></span>
                print</span> release</span></span>
</span>
            else</span>:</span></span>
                print</span> '</span>%s</span>-</span>%s</span>-</span>%s</span>'</span> %</span> (</span>args</span>.</span>package</span>,</span> args</span>.</span>version</span>,</span> release</span>)</span></span></code></pre>How to use it</h2>
The tool is used as follows:</p>
$</span> python rpm-release-number.py -h</span></span>
usage:</span> rpm-release-number.py</span> [-h] </span>[</span>--current</span>] [</span>--full</span>]</span> package version</span></span>
</span>
positional</span> arguments:</span></span>
  package</span>     The software package for which you want a release number</span></span>
  version</span>     The version of the package for which you want a release number</span></span>
</span>
optional</span> arguments:</span></span>
  -h,</span> --help  show this help message and exit</span></span>
  --current</span>   Don't increment any counters, just print the current value</span></span>
  --full      Print the full package name</span></span></code></pre>
For example:</p>
$</span> python rpm-release-number.py foo</span> 1.1</span></span>
1</span></span>
$</span> python rpm-release-number.py foo</span> 1.1</span></span>
2</span></span>
$</span> python rpm-release-number.py foo</span> 1.1</span></span>
3</span></span>
$</span> python rpm-release-number.py foo</span> 1.1</span> --current</span></span>
3</span></span>
$</span> python rpm-release-number.py foo</span> 1.1</span> --current</span></span>
3</span></span>
$</span> python rpm-release-number.py foo</span> 1.1</span> --current --full</span></span>
foo-1.1-3</span></span></code></pre>
So, there we have it. A quick tool that keeps track of what release every package is supposed to
have, and stores it in an easy-to-maintain location (a JSON file in the user’s home directory). This
tool is easily usable in Jenkins scripts, but that’ll be for a different blog entry.</p>