Albert O'Connor Web Developerhttp://albertoconnor.ca/2017-07-07T19:00:00-04:00f-strings For the Win2017-07-07T19:00:00-04:00Albert O'Connortag:albertoconnor.ca,2017-07-07:f-strings-for-the-win.html<p>It has been a long time coming, but I am now actively migrating
existing projects to Python 3. <a href="https://docs.python.org/3/whatsnew/3.6.html">Python
3.6</a> specifically,
because when I am done I will be able to take advantage of my new
favorite feature everywhere!</p>
<p>That feature is <code>f-strings</code>, you can <a href="https://www.python.org/dev/peps/pep-0498/">Read
PEP-0498</a> for all the details
but here is a basic example:</p>
<div class="codehilite"><pre><span></span><span class="n">Python</span> <span class="mf">3.6</span><span class="o">.</span><span class="mi">1</span> <span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="n">Type</span> <span class="s2">"help"</span><span class="p">,</span> <span class="s2">"copyright"</span><span class="p">,</span> <span class="s2">"credits"</span> <span class="ow">or</span> <span class="s2">"license"</span> <span class="k">for</span> <span class="n">more</span><span class="o">...</span>
<span class="o">>>></span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Albert'</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'Hello, {name}!'</span>
<span class="s1">'Hello, Albert!'</span>
</pre></div>
<p>Yes, <code>f-strings</code> or "format strings" are yet another way to format
strings. I know how you might feel on first glance, I didn't like
<code>f-strings</code> at first myself. Surely the explicitness of <code>string.format()</code> would be better than the syntactic magic
of <code>f-strings</code>, but once you use them it is hard not to love them.</p>
<p>Another complaint is that there are too many ways to format strings in
Python. That is true, but having one obvious way to do things shouldn't
cause a lack of real progress in the language. None of the previous string formatting approaches have felt as natural or as
teachable as <code>f-strings</code>.</p>
<h3>How do they work?</h3>
<p>Initially, you might think about <code>f-strings</code> as having an implicit call to
<code>.format()</code> with all of the local and global scope passed in. This isn't the correct mental model and that kind of implementation was specifically
rejected in the design process. Instead, you can think about <code>f-strings</code>
like this:</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">f</span><span class="s1">'Hello, {name}!'</span>
<span class="c1"># is the same as</span>
<span class="o">>>></span> <span class="s1">'Hello, '</span> <span class="o">+</span> <span class="n">format</span><span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="o">+</span> <span class="s1">'!'</span>
</pre></div>
<p>The built-in <code>format</code> function calls the underlying <code>__format__</code>
interface which allows types to control how they are formatted. An
additional spec can be passed into <code>format</code> as well.</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">datetime</span>
<span class="o">>>></span> <span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'The year {today:%Y}'</span>
<span class="s1">'The year 2017'</span>
<span class="c1"># is the same as</span>
<span class="o">>>></span> <span class="s1">'The year '</span> <span class="o">+</span> <span class="n">format</span><span class="p">(</span><span class="n">today</span><span class="p">,</span> <span class="s1">'%Y'</span><span class="p">)</span>
</pre></div>
<p>Since what is in the <code>{}</code> is an expression which is evaluated in the same context as
the <code>f-string</code> literal appears, what you might feel should work, just does.</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">foo</span><span class="o">=</span><span class="s1">'bar'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'The answer is {data["foo"]}'</span>
<span class="s1">'The answer is bar'</span>
<span class="o">>>></span> <span class="k">class</span> <span class="nc">Circle</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="o">...</span> <span class="n">color</span> <span class="o">=</span> <span class="s1">'red'</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">circle</span> <span class="o">=</span> <span class="n">Circle</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'The circle is {circle.color}'</span>
<span class="s1">'The circle is red'</span>
</pre></div>
<p>The expressions can become quite complicated including literals,
operations, and function calls:</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">i</span> <span class="o">=</span> <span class="mi">10</span>
<span class="o">>>></span> <span class="n">word</span> <span class="o">=</span> <span class="s1">'grease'</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'Math: {i + 1}'</span>
<span class="s1">'Math: 11'</span>
<span class="o">>>></span> <span class="n">f</span><span class="s1">'Upper case: {word.upper()}'</span>
<span class="s1">'Upper case: GREASE'</span>
</pre></div>
<p>This offers a huge amount of flexibility with code which just works the
way one might expect.</p>
<h3>Even more in Python 3.6</h3>
<p>First, there is everything in Python 3 including unicode everywhere, all
the new async support, and an interactive shell which remembers the commands in your last
session!</p>
<p>Second, 3.6 features dictionaries which are even faster and perserve key insertion order. Some claim this is an <a href="https://www.youtube.com/watch?v=npw4s1QTmPg">implementation
detail</a> which can change, but as Brandon Rhodes and others have observed <a href="https://www.youtube.com/watch?v=66P5FMkWoVU">there probably isn't
any going back</a>!</p>
<p>Python 3.6 isn't available on Ubuntu 16.04 by default. Installing isn't
<a href="https://askubuntu.com/a/865569">too hard</a>, and you can add it to your
provisioning automation in a few simple steps.</p>
<p>Start migrating your projects to Python 3.6, and <a href="https://www.youtube.com/watch?v=2DkfPzWWC2Q">deprecate support</a> for
2.7 so we can all use <code>f-strings</code> everywhere sooner.</p>PyCon Canada 2016 Talk2016-11-11T11:00:00-05:00Albert O'Connortag:albertoconnor.ca,2016-11-11:pycon-canada-2016-talk.html<p>I am very excited to be giving a talk on <a href="https://2016.pycon.ca/en/schedule/016-albert-oconnor/">Async Talks with Django
Channels</a> at <a href="https://2016.pycon.ca/">PyCon Canada 2016</a>.</p>
<p>I am no expert, but I have enjoyed using the pre 1.0
versions of <a href="https://channels.readthedocs.io/">Channels</a> and integrating it into my client work and side projects.
I believe Channels is hugely promising and an elegant way to work new
technologies such as WebSocket into Django while making it possible to
still write clean readable Django code.</p>
<p>Huge debt of gratitude to Andrew Godwin and everyone who has contributed
to Channels to make it possible.</p>
<p>Here are the resources for my talk:</p>
<ul>
<li><a href="https://github.com/albertoconnor/asyncdemo">Demo Repo</a></li>
<li><a href="https://speakerdeck.com/albertoconnor/async-tasks-with-django-channels">Speaker Deck Slides</a></li>
<li><a href="/files/AsyncChannels.pdf">PDF Slides</a></li>
<li><a href="/django-channels-background-tasks.html">Previous Blog Post on Channels</a></li>
</ul>
<p>And some even better Channel resources:</p>
<ul>
<li><a href="https://channels.readthedocs.io/">The Documentation</a></li>
<li><a href="https://youtu.be/rsEkQbMLCH4?t=53m30s">Andrew's Recent Talk at Under the Hood</a></li>
<li><a href="https://github.com/django/channels">Channels on GitHub</a></li>
</ul>
<p>If you are thinking you want to use Channels in one of your projects and
are looking for some help, you can hire me.</p>
<p>WatPy can also help you with personal projects via the <a href="http://wapy.ca">WatPy Slack</a>—Enter your
email on our website to get an invite.</p>Django Channels for Background Tasks2016-05-18T04:00:00-04:00Albert O'Connortag:albertoconnor.ca,2016-05-18:django-channels-background-tasks.html<p><a href="https://github.com/andrewgodwin/channels">Django Channels</a> is
the most exciting thing to happen to Django since well Django :).</p>
<p>This little tutorial is what you need to add a background task processor
to Django using channels. Our task for this example will just be
outputting "Hello, Channels!", but you could imagine running a
subprocess on some data or sending an email.</p>
<p>NOTE: channels works on an at-most-once delivery model, so it is
possible a message in a channel could be lost, delivery isn't
guaranteed. That also means consumers don't have to worry about
duplicates.
<a href="http://channels.readthedocs.io/en/latest/concepts.html#what-is-a-channel">Read more in the docs</a>.</p>
<p>This example will be stripped down to the basic code without much error
checking. There are detailed examples one
<a href="https://github.com/andrewgodwin/channels-examples">here</a>
and <a href="https://github.com/jacobian/channels-example">here</a>.</p>
<p>We will start with a simple Django 1.9 app without channels</p>
<div class="codehilite"><pre><span></span><span class="c1"># urls.py</span>
<span class="kn">from</span> <span class="nn">django.conf.urls</span> <span class="kn">import</span> <span class="n">url</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">views</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">url</span><span class="p">(</span><span class="s1">r'^$'</span><span class="p">,</span> <span class="n">views</span><span class="o">.</span><span class="n">home</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">'home'</span><span class="p">),</span>
<span class="p">]</span>
<span class="c1"># views.py</span>
<span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span>
<span class="k">def</span> <span class="nf">home</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">template</span><span class="o">=</span><span class="s2">"home.html"</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"Hello, Channels!"</span><span class="p">)</span> <span class="c1"># long running task of printing.</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span>
<span class="n">request</span><span class="p">,</span>
<span class="n">template</span><span class="p">,</span>
<span class="nb">dict</span><span class="p">(),</span>
<span class="p">)</span>
</pre></div>
<p>You will need to define a home.html template where Django can find it,
but besides that this simple site should work and synchronously render
what is in your home.html and output "Hello, Channels!" on your
terminal.</p>
<h2>Now for channelification!</h2>
<p>First, you will need redis, you can use a built-in broker but redis is
pretty easy to install on Linux or Mac (using homebrew at least).</p>
<h4>MacOS</h4>
<div class="codehilite"><pre><span></span>homebrew install redis
</pre></div>
<h4>Debian</h4>
<div class="codehilite"><pre><span></span>apt-get install redis-server
</pre></div>
<p>Once that is done you will need to install channels and asgi-redis with
pip. This apparently also installs Twisted and zope.interface. The past
returns :)</p>
<div class="codehilite"><pre><span></span> pip install channels asgi-redis
</pre></div>
<p>Make sure redis is running in another terminal</p>
<div class="codehilite"><pre><span></span>redis-server
</pre></div>
<p>Add <code>channels</code> to your INSTALLED_APPS and add the following to your
settings.py to tell channel how to run:</p>
<div class="codehilite"><pre><span></span><span class="n">CHANNEL_LAYERS</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"default"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"BACKEND"</span><span class="p">:</span> <span class="s2">"asgi_redis.RedisChannelLayer"</span><span class="p">,</span>
<span class="s2">"CONFIG"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"hosts"</span><span class="p">:</span> <span class="p">[</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'REDIS_URL'</span><span class="p">,</span> <span class="s1">'redis://localhost:6379'</span><span class="p">)],</span>
<span class="p">},</span>
<span class="s2">"ROUTING"</span><span class="p">:</span> <span class="s2">"myproject.routing.channel_routing"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">}</span>
</pre></div>
<p>Where you replace "myproject" with the name of your project.</p>
<p>You need to create the routing.py file—beside your settings.py and root
url.py file. In it add the channel_routing list, but for now we will
leave it empty.</p>
<div class="codehilite"><pre><span></span><span class="c1"># routing.py</span>
<span class="n">channel_routing</span> <span class="o">=</span> <span class="p">[]</span>
</pre></div>
<p>This is actually enough to get what we already had to run.</p>
<div class="codehilite"><pre><span></span>python manage.py runserver
</pre></div>
<p>This runs workers and the front end server
<a href="https://pypi.python.org/pypi/daphne/">daphne</a> and it will handle
Django requests and responses as you are used to.</p>
<h2>Workers</h2>
<p>Stop the server for now with <code>Control-C</code>. Django has grown a new
command: <code>runworker</code></p>
<div class="codehilite"><pre><span></span>python manage.py runworker
</pre></div>
<p>This won't run on port 8000, it will just run a worker listening on the
default channel layer so you can't connect to it directly. Kill it for
now with Control-C.</p>
<p>Channels has a front end server you can run in a separate process called
daphne. To use it you need to create a asgi.py file—beside the wsgi.py
file Django created for you, or your settings.py file.</p>
<div class="codehilite"><pre><span></span><span class="c1"># asgi.py</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">channels.asgi</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="s2">"DJANGO_SETTINGS_MODULE"</span><span class="p">,</span> <span class="s2">"myproject.settings"</span><span class="p">)</span>
<span class="n">channel_layer</span> <span class="o">=</span> <span class="n">channels</span><span class="o">.</span><span class="n">asgi</span><span class="o">.</span><span class="n">get_channel_layer</span><span class="p">()</span>
</pre></div>
<p>Here is how you run daphne, should be familiar if you have used gunicorn
or similar.</p>
<div class="codehilite"><pre><span></span>daphne myproject.asgi:channel_layer --port 8000
</pre></div>
<p>Now you can connect to http://127.0.0.1:8000/ but it just times out
because there aren't any workers. In a separate terminal window run use
the runworker command to start a worker and try connecting again.</p>
<p>Boom! You are back to where you were, but now there are two processes
running. Now for the interesting bit, doing something asynchronously.</p>
<h2>Background Task</h2>
<p>Websockets are the killer feature of channels and other tutorials cover
how to use them. Here we are going to make our own channel for our
background task of saying hello.</p>
<p>It starts in routing.py where we will connect our channel to our new
consumer. A consumer is a function which takes a message much like how a
view takes a request.</p>
<div class="codehilite"><pre><span></span><span class="c1"># routing.py again</span>
<span class="kn">from</span> <span class="nn">channels.routing</span> <span class="kn">import</span> <span class="n">route</span>
<span class="kn">from</span> <span class="nn">.consumers</span> <span class="kn">import</span> <span class="n">hello</span>
<span class="n">channel_routing</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">route</span><span class="p">(</span><span class="s1">'background-hello'</span><span class="p">,</span> <span class="n">hello</span><span class="p">),</span>
<span class="p">]</span>
</pre></div>
<p>This means messages put into the background-hello channel will be
handled by the hello callable found in the local consumers.py file—place
it beside your views.py file in your app folder. We need to write our
first consumer:</p>
<div class="codehilite"><pre><span></span><span class="c1"># consumers.py</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">message</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s2">"Hello, Channels!"</span><span class="p">)</span> <span class="c1"># long running task or printing</span>
</pre></div>
<p>Unlike views, consumers don't have to return a response, but they can
send any number of messages down any number of channels they like.</p>
<p>Finally, we need to use our new background channel in our view, so let's
edit views.py</p>
<div class="codehilite"><pre><span></span><span class="c1"># views.py</span>
<span class="kn">from</span> <span class="nn">django.shortcuts</span> <span class="kn">import</span> <span class="n">render</span>
<span class="kn">from</span> <span class="nn">channels</span> <span class="kn">import</span> <span class="n">Channel</span>
<span class="k">def</span> <span class="nf">home</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">template</span><span class="o">=</span><span class="s2">"home.html"</span><span class="p">):</span>
<span class="n">message</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">Channel</span><span class="p">(</span><span class="s1">'background-hello'</span><span class="p">)</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">return</span> <span class="n">render</span><span class="p">(</span>
<span class="n">request</span><span class="p">,</span>
<span class="n">template</span><span class="p">,</span>
<span class="nb">dict</span><span class="p">(),</span>
<span class="p">)</span>
</pre></div>
<p>That should be enough to get "Hello, Channels!" to be printed
asynchronously to your terminal.</p>
<p>What is interesting is you didn't have to do anything to get the HTTP
request and response channel to work. Including and initializing the
channels library is enough for it to create those channels and route
based on url.py. Everything Django works as it always did.</p>
<p>This just scratches the surface of what you can do with channels, so
start using it today!</p>
<p>You can view the code from this blog post on
<a href="https://github.com/albertoconnor/channelsbackground">GitHub</a>.</p>Disabling Migrations While Testing2016-01-06T14:46:00-05:00Albert O'Connortag:albertoconnor.ca,2016-01-06:disabling-migrations-while-testing.html<p>If you have a large Django 1.7+ project with a lot of migrations running test even with <code>--keepdb</code> can be slow just because the new migration framework has to order the migrations even if there is nothing to do.</p>
<p>After a few attempts I have found something which works pretty well for me. In your testing setting you can include the following:</p>
<pre>
class DisableMigrations(object):
def __contains__(self, item):
return True
def __getitem__(self, item):
return "notmigrations"
MIGRATION_MODULES = DisableMigrations()
</pre>
<p>I have found related suggestions on the <a href="https://groups.google.com/forum/#!msg/django-developers/PWPj3etj3-U/kCl6pMsQYYoJ">Django mailing list</a>. Both ideas are presented on <a href="http://stackoverflow.com/questions/25161425/disable-migrations-when-running-unit-tests-in-django-1-7">Stackoverflow</a>.</p>
<p><strong>Update:</strong> Thanks <a href="https://github.com/NotSqrt" target="_blank">NotSqrt</a> for linking me to original <a href="https://gist.github.com/NotSqrt/5f3c76cd15e40ef62d09" target="_blank">Gist</a> I found when I was searching but wasn't able to easily find again :).</p>Default VPCs and T2 Instnaces2014-07-28T14:45:00-04:00Albert O'Connortag:albertoconnor.ca,2014-07-28:default-vpcs-and-t2-instnaces.html<p><a href="http://aws.amazon.com/">Amazon Web Services</a> grows increasingly complex as they provide new an compelling features while supporting legacy users. To some extent this is unavoidable, but the upgrade path is not always clear and keeping documentation up to date is challenging. I did some digging into the new T2 instances and this is some of what I found.</p>
<h2>New T2 Instances</h2>
<p>My interest was peaked by the T2 instances. The t2.micro is less expensive while nearly doubling the available memory. There are other trade offs to be aware of. T2 instances must be HVM (Hardware) instead of PV (Paravirtualization), which for Linux could mean reduced performance. The machines they will be on will be more powerful so it may end up being a net gain.
</p>
<p>T2 instances must also be 64-bit, which means your memory foot print may double versus 32-bit. A t2.mirco has twice the memory but if you are used to using 32-bit systems you may not get much actual benefit.</p>
<h2>VPC and Default VPC</h2>
<p>Virtual Private Cloud (VPC) was originally a way for large companies with existing data centres to extend them into AWS. Over time it has evolved into EC2 networking done better to the point that new AWS accounts
get a "default VPC" automatically. If you go into a new Region you haven't been in before you also get one automatically.</p>
<p>VPC have some nice advantages. They can be more secure. You can change security groups on running instances. They also feel more like real world data centre networking if you are into that kind of thing.</p>
<p>The downside is complexity. Lots of complexity. The default VPC helps by doing the right thing for you, though it may not be the most secure setup. Another downside is the NATs, but more on that soon.</p>
<p>If you don't have a default VPC because your AWS account is more than a year old you are stuck in EC2-Classic with the option of building your own VPC without the goodness of the default VPC, or are you?</p>
<h2>T2 and EC2-Classic</h2>
<p>During my first attempt to launch a t2.micro with the <a href="http://aws.amazon.com/cli/" target="_blank">aws cli</a> I received an error saying a VPC was required. This started my most recent dive into the world VPC and the discovery that I was stuck in EC2-Classic.</p>
<p>Next I tried to lunch a t2.micro via the management console and something fascinating happened. The process took a bit longer during one of the steps, and all of a sudden I had a VPC in us-east-1 even though I have had an account there for years.</p>
<p>The VPC appears to be like a "Default VPC", but the default subnet flag is set to "no". I think a key feature of default subnets is that instance launched in them get public IPs by default, but that is now an option when you launch an instance.</p>
<p>It would appear that T2 instances can help continue to bridge the gap between EC2-Classic and default VPC.</p>
<h2>NATs</h2>
<p>When you read the official docs and the unofficial blog posts you hear a lot about creating private subnets and NAT instances. The NAT are EC2 instance which allow outgoing traffic for instance in private subnets which are entirely isolated from the internet.</p>
<p>The downside if you are trying to keep costs low is you have to paid for an extra instance just to have some private subnets. Private subnets though seem to be required for certain managed services like RDS, but if NATs are only required for outbound internet traffic, than that would only be needed for servers I need to install updates on myself.</p>
<p>
</p>
<p>Since RDS is managed by Amazon, I image they have there own NAT setup to connect to them and update packages. So my theory is I can use RDS in a private subnet without the extra costs of NATs. Though I want one of my RDS instance to be available across accounts, so I will be left with using security groups to secure them for now.</p>PyCon 2014 and Python on Rails2014-06-16T14:17:00-04:00Albert O'Connortag:albertoconnor.ca,2014-06-16:pycon-2014-and-python-rails.html<p>The organizers of <a href="https://us.pycon.org/2014/" target="_blank">PyCon 2014</a> did a great job making a large conference feel like a regional one. The diversity among the attendees was great, though there is always room for improvement. I will definitely be going back in 2015.</p>
<p>Here are some lessons learned and a recap of Python on Rails which proved that trains still are the best way to travel.</p>
<p><a href="http://pyvideo.org/video/2566/pickles-are-for-delis-not-software" target="_blank">Stop using pickle</a>. Did you know pickle is a stack based language pretending to be a data interchange format? Even celery, which is basically instantaneously pickling and unpickling (which avoids some of the major pitfalls), is moving away from it.</p>
<p>You can use the environment variable <code>PYTHONDONTWRITEBYTECODE</code> to prevent Python from writing .pyc files.</p>
<pre>
$ export PYTHONDONTWRITEBYTECODE=1
</pre>
<p>In development when you don’t really care about the slight speed optimization of precompiling byte code this can help avoid other pitfalls. This tip comes from <a href="http://rhodesmill.org/brandon/slides/2014-04-pycon/day-of-the-exe/" target="_blank">
"Day of the EXE"</a>. All of <a href="http://pyvideo.org/speaker/337/brandon-rhodes" target="_blank">Brandon Rhodes</a> talks are worth watching. They do a beautiful job of expressing complex concepts in a way which is accessible to people of all skill levels.</p>
<p>If you use pdb—and especially if you don’t, you should watch <a href="http://pyvideo.org/video/2673/in-depth-pdb" target="_blank">
In Depth PDB by Nathan Yergler</a> which covers it from the basics to more advanced usage. I leaned about the <a href="https://docs.python.org/2/library/pdb.html#pdb.post_mortem" target="_blank">
postmortem debugger</a> <code>import pdb; pdb.pm()</code> and the potential to have Django enter it automatically using something like <a href="https://pypi.python.org/pypi/django-pdb" target="_blank">
django-pdb</a>.</p>
<p>I should really start using postgres in development to spare <a href="http://pyvideo.org/video/2630/designing-djangos-migrations" target="_blank">
Andrew Godwin</a> from shedding too many tears.</p>
<p>Finally next time you want to upload a package to PyPI instead of crying softly to yourself, you should try <a href="https://pypi.python.org/pypi/twine/" target="_blank">
twine</a>, I know I will.</p>
<p>Nearly all the videos are up on <a href="http://pyvideo.org/category/50/pycon-us-2014" target="_blank">pyvideo.org</a>. One notable exception, one of the most talked about talks of the conference <a href="https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript" target="_blank">The Birth and Death of Javascript by Gary Bernhardt</a></p>
<p>You should go next year, it will be in Montreal again. It will sell out months before the conference start date so buy your tickets early.</p>
<h2>Python On Rails</h2>
<p><a href="http://watpy.ca/" target="_blank">WatPy</a> worked with Brydon and Anastisa at <a href="http://threefortynine.com/" target="_blank">349 Coworking</a> to organize a chartered train trip from Toronto to Montreal. For 6 hours we had lunch, had few drinks, did a coding challenge and heard from Brandon Rodes about how to get the most out of PyCon.</p>
<p>It was a great stress free way to get to the conference and it meant when you got there you already knew enough people to get some friendly smiles in the hallways between sessions. That and we arrived right in the heart of Montreal and a short walk away from the hotels and conference centre.</p>
<p>We are planning on taking the train again next year. The website hasn’t been updated, but if you <a href="http://pythononrails.ca/" target="_blank">sign up</a> with your email you will be among the first know when details come out this fall.</p>
<p>If you are involved in a startup consider joining them on the <a href="https://trips.threefortynine.com/" target="_blank">Startup Train</a> to the Montreal Startup Festival.</p>
<p><center>
<iframe src="http://www.slideshare.net/slideshow/embed_code/35502942" width="476" height="400" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe>
</center></p>Django i18n2013-08-19T11:43:00-04:00Albert O'Connortag:albertoconnor.ca,2013-08-19:django-i18n.html<p>What does i18n mean?</p>
<p class="text-center">i18n = i followed by 18 letters followed by n = internationalization</p>
<p>
That is it, just a short hand. It isn't the name of standard governmental or otherwise.
</p>
<p>
When I watched the first <a href="http://www.youtube.com/watch?v=f3Y-QoEkPtw" target="_blank">videos about Django</a> many years ago, they declared that Django had great internationalization support including a Welsh translation. It sounded great and I wanted to try to use it, but I was never involved in a project that really needed it, until now!
</p>
<p>
At <a href="http://www.thalmic.com/" target="_blank>Thalmic Labs</a> we were able to translate the company website into German and Korean in a matter of weeks, the most time was devoted to getting the translations done, very little was figuring out i18n in Django. It was only a few hours of hitting my head against the wall.
</p>
<p>
Here is a bit of what I learned along the way. Hopefully it helps you.
</p>
<h2>Resources</h2>
<p>
First watch this <a href=">Thalmic Labs</a> we were able to translate the company website into German and Korean in a matter of weeks, the most time was devoted to getting the translations done, very little was figuring out i18n in Django. It was only a few hours of hitting my head against the wall.
</p>
<p>
Here is a bit of what I learned along the way. Hopefully it helps you.
</p>
<h2>Resources</h2>
<p>
First watch this <a href="http://pyvideo.org/video/1379/a-gringos-guide-to-internationalization" target="_blank">excellent talk by Jacob Burch</a>. He helps to fill in some of the gaps from the <a href="https://docs.djangoproject.com/en/1.5/topics/i18n/translation/" target="_blank">i18n documentation</a> which you should also have a look at.
</p>
<iframe width="640" height="360" src="//www.youtube.com/embed/j2ZHZWfx60Y" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
<p>
Definitely grab Jacob's version of poxx, you won't need it in production or on staging so you can just install it locally:
</p>
<pre>
pip install -e git+https://github.com/jacobb/poxx.git#egg=poxx
</pre>
<p>
By the end of this post you will have poxx generated fake translations appearing on your website. The first step is to mark your strings in your templates, code and javascript for translation. The docs cover that pretty well so I am going to pick up after that.
</p>
<h2>The Locale Directory</h2>
<p>
I was using a Django 1.4 directory layout so when I ran
</p>
<p><pre>
python manage.py makemessages
</pre>
</p><p>
And got the somewhat unhelpful error message:
</p>
<blockquote>
"This script should be run from the Django SVN tree or your project or app tree. If you did indeed run it from the SVN checkout or your project or application, maybe you are just missing the conf/locale (in the django tree) or locale (for project and application) directory? It is not created automatically, you have to create it by hand if you want to enable i18n for your project or application."
</blockquote>
<p>
What it means is Django can't find ./locale, so it will give up trying to do anything. I think the error message has been improved in newer versions of Django.
</p>
<p></p><p>
There are two options of where to place locale directories. One locale directory for the entire project or one locale directory per app. I think one per app is the right answer especially for large projects. However to get things working without spending a bunch of extra time fighting with makemessages and compilemessages, one locale directory for your entire project is easier.
</p>
<p>
It is easier to run makemessages and compilemessages, but you need to tell Django explicitly where your translations are, because by default it only looks for locale directories under each app in INSTALLED_APPS. You can use the LOCALE_PATHS setting for this:
</p>
<div class="highlight"><pre><span class="n">LOCALE_PATHS</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">BASE_DIR</span><span class="p">,</span> <span class="s">"locale"</span><span class="p">),</span> <span class="c"># Assuming BASE_DIR is where your manage.py file is</span>
<span class="p">)</span>
</pre></div>
<p></p><p>
Make a locale directory beside your manage.py file, then you can run:
</p>
<pre>
python manage.py makemessages -l de
</pre>
<p>
This will create a django.po file from the german locale under locale/de/LC_MESSAGES. The file will have message id, but the values (the german translations) will be empty. Great first step done! Now for the poxx.
</p>
<h2>Poxx.py</h2>
<p>
Jacob is right about poxx.py, it is very useful for making sure translations are working, and making sure you are translating everything you want to.
</p>
<p></p><p>
You can run it on the german .po file you created above:
</p>
<pre>
poxx.py locale/de/LC_MESSAGES/django.po
</pre>
<p>
Now run compilemessages:
</p>
<pre>
python manage.py compilemessages
</pre>
<p>
Now you are almost ready to see fake poxxified translations.
</p>
<h2>i18n_patterns</h2>
<p>
The fastest way to control what language you are viewing is to follow the <a href="https://docs.djangoproject.com/en/1.5/topics/i18n/translation/#internationalization-in-url-patterns" target="_blank">documentation to hook up the i18n_patterns</a> in your root url.py. One tradeoff is it will prepend your default locale (say 'en') to all your urls under the i18n_patterns, but all the redirection will happen automatically.
</p>
<p>
In the end we used the cookie based approached with a language chooser, but for getting it done in one sitting, stick to i18n_patterns.
</p>
<p>
Now you should see your poxxified translations by running your server: http://127.0.0.1:8000/de/your/translated/view/
</p>
<p>
The poxx translations replace normal characters with weird unicode characters which means your text is still readable, but odd looking. Normal looking text that means those strings weren't marked for translation. Fix any you find and then re-run makemessages, poxx.py and compilemessage to make sure your fix worked.
</p>
<p>If you want to test this out but don't have project handy to work on you can try an example project I built: <a href="https://bitbucket.org/amjoconn/django-i18n-example" target="_blank">https://bitbucket.org/amjoconn/django-i18n-example</a>. It already has most strings marked for translation so you can run through the messages and poxx.py steps to see how they work.
</p>
<p>
i18n is a complex topic, but a worthy one to sink your teeth into. The more multi-lingual the web is the better place it will be.
</p>
<p>
By the way, Thalmic Labs is great company which is <a href="https://www.thalmic.com/careers/web-developer/" target="_blank">looking for a full time web developer</a>.
</p>Going Responsive2013-07-22T13:50:00-04:00Albert O'Connortag:albertoconnor.ca,2013-07-22:going-responsive.html<p>I was somewhat skeptical of responsive design. Grids add some additional HTML which is essentially for layout and I thought responsive design generally relied on Javascript, but after transforming my own website I found things are getting better.</p>
<p>Let's start with what my website used to look like.</p>
<p><img style="border: medium solid white; box-shadow: 0px 0px 4px #333; margin: 0.5em 0em" src="http://s3.amazonaws.com:80/albertoconnor/image_manager%2FAlbert_OConnor_._ca.png" alt="Previous design of my website where most section were surrounded by a border." class="img-fluid" /></p>
<p>Boxy and segmented, it was built for a specific width and not on a grid. It looked alright on mobile because I used <a href="http://albertoconnor.ca/blog/2012/Jan/19/viewport-making-webpages-look-great-ios-devices">the meta viewport element</a> correctly.</p>
<p>For the redesign I used <a href="http://twitter.github.io/bootstrap/">bootstrap</a> with its responsive grid and <a href="http://fortawesome.github.io/Font-Awesome/">font awesome</a>, which is as awesome as the name implies. Gone were the boxes, replaced with infinitely wide horizontal stripes and content suspended on an invisible grid. A grid which can resize its cells and stack them all vertically when down at a phone resolution — thanks to everything floating left.</p>
<p>My cat-based banner image previously was precisely cropped, but that wasn't going to work in a world where everything is infinitely wide. I went with the common approach of making the picture of my cat full width and centred, keeping the most interesting part of the picture in the middle of the viewport. I had to tweak the media queries in the CSS for smaller screens to prevent vertical repeating.</p>
<p>
</p>
<p>Overall the process was straightforward. Bootstrap lets you do a bunch of <a href="http://twitter.github.io/bootstrap/customize.html">customization</a> and avoid <a href="http://lesscss.org/">LESS</a>. I removed the grid adjustment for the jumbo screen size which looked too weird to me. It would be nice if you could more easily disable the navigation menu on the tablet size; my navigation would look fine on a tablet. The Bootstrap grid is better because it requires less additional HTML; merely nest elements with the CSS classes container, row and span. No more clearing divs everywhere.</p>
<p>Media queries are totally awesome — even better they are very well supported including on all mobile browsers. IE 8.0 and below doesn't support them, but if you default to the desktop CSS rules it doesn't have to. There is probably also a javascript shim out there. Another thing I like about responsive design and media queries is, on Chrome at least, it makes every page more readable when zoomed in; a nice accessibility win.</p>
<p>I am happy with the result and the fact it only took a day or two of effort to complete. Hopefully it looks better, no matter what device you are using to view it.</p>
<p></p>Go Open Data2013-04-08T13:10:00-04:00Albert O'Connortag:albertoconnor.ca,2013-04-08:go-open-data.html<p>What started as a brief conversation between myself and Michael Druker almost a year ago has come to life. On Saturday, May 11th people from Windsor to Ottawa are going to come together at the University of Waterloo School of Pharmacy for GO Open Data, a one day conference.</p>
<p><center>
<img src="http://s3.amazonaws.com:80/albertoconnor/image_mananger%2Fgo_open_data.png" alt="GO Open Data Conference" style="margin: 5px" />
</center></p>
<p>The conference aims to bring together people from across Ontario and across the various interests related to open data to enjoy 8 talks, 2 panels, and an exhibition called Open Data Alley. Citizens, journalists, developers, administrators, CIOs, librarians, different people from all the areas connected to this fusion of technology and policy will be in attendance. That alone will make for an exciting day.</p>
<p>
</p>
<p>It is thanks to the many partners from the Region of Waterloo, the City of Waterloo and the University of Waterloo that such a small idea has become so successful. We have amazing speakers like Ontario Information and Privacy Commissioner Dr. Ann Cavoukian, and up and coming speakers like James McKinney from <a href="http://opennorth.ca/">Open North</a> in Montreal.</p>
<p>
</p>
<p>You can hear an interview with Dr. Cavoukian on the lawful access episode of Jessie Brown's <a href="http://searchengine.tvo.org/blog/search-engine-blog/audio-podcast-123-privacy-commissioner-ann-cavoukian-lawful-access">Search Engine podcast</a>.</p>
<p>The rest of the schedule can be found online: <a href="http://2013.go-opendata.ca/">http://2013.go-opendata.ca/</a></p>
<p>The conference website itself is a source of open data about the conference. Built with Django using Daniel Lindsley's excellent library, <a href="http://django-tastypie.readthedocs.org/en/latest/">Tastypie</a>, to easily transform models into API endpoints. Snow Conrad did an amazing job creating a professional looking logo and helping with the overall look of the website.</p>
<p>
</p>
<p>Events like this can't be successful unless there is a community behind them. I am grateful there is a strong open data community in Ontario such that when we struck the match a little under a year ago, there was a community to catch fire.</p>
<p>You should come and see what the community is up to. I think it will be a great conference.</p>Pjax vs JSON2013-01-23T17:11:00-05:00Albert O'Connortag:albertoconnor.ca,2013-01-23:pjax-vs-json.html<p>At DjangoCon US 2012 I asked a question about performance after the BDFL keynote on Pjax. Asking about performance is never a good idea since it is never the most important thing. I was attempting to evaluate Pjax versus the plethora of Javascript based rendering that was being discussed.</p>
<p>A common idea was to use the same template language in Django as you use in Javascript. The Meteor keynote explained that just sharing a template language doesn't solve the hard problems of passing the data and the logic about pre-rendering the data to the client. The Django template language is very simple, but it still lets you indirectly run arbitrarily complex code. All of that logic either has to be reimplemented on the client or the results magically exposed there.</p>
<p>Pjax keeps rendering of templates on the server side and uses push state and ajax to optimize the user experience by avoiding page refreshes. Blocks of HTML are send from the server and installed on the existing page.</p>
<p>I was looking for a comparison of the performance of Pjax vs Javascript rendering, which in the end isn't a simple question to answer. Even so I think it is worth having a mental model of the performance trade offs. Here is my attempt.</p>
<p>Passing everything off to the client could be viewed as parallel computing. Have 1,000 simultaneous users? Send them small bits of JSON and have each of their CPUs do the template rendering for you. You also have to send the template, but on the whole you send fewer bytes and consume less of your server cpu. Massively parallelized! Awesome right?</p>
<p>It is parallelized, but you have to rely on the client's browser to create a good user experience. Browsers largely are great and are getting better, but what if the user is on a low powered netbook and you are taking Javascript template rending to the extreme. What about the old school way of making things performant through caching?</p>
<p>I remember reading a blog post once about how you can't make programs faster, no matter what you do the CPU does the same number of instructions per second. The only possible gains come from figuring out how to do fewer things. The best way on the web to do less is cache!</p>
<p>Caching correctly is hard, most users are looking at pages with different combinations of content, but 37 Signals focused on doing it right and have proved with Basecamp that Pjax and caching can be very fast and responsive.</p>
<p>Not being able to cache rendering and losing some control over the clients experience is a disadvantage to Javascript heavy web frameworks. One could argue I just really like Python, but I think there are other interesting reasons to pursue a Pjax path.</p>Function Based Views Are My First Love2012-09-01T16:14:00-04:00Albert O'Connortag:albertoconnor.ca,2012-09-01:django-class-based-views-and-domain-specific-langu.html<p>There is something special about your first love. It is possible be too attached and not embrace the new hotness of say Class Based View (CBV). But after having a fling and then dating CBV for a while, I now know that Function Based Views will always have a special place in my heart, and I hope in Django.</p>
<p>Let's look at a FBV:</p>
<pre>
def foo(request, arg, template="foo.html"):
# Do something
return render(request,
template,
dict(bar="bar"))
</pre>
<p>Simple. Beautiful. Easy to follow.</p>
<p>An explicit context filled with exactly what is available in the template and what names those values have. No extra docs needed. Explicit request and response, and the path to get from one to the other is outlined in one place, in one file.</p>
<p>This straight forward approach of FBVs also makes it easier to learn and debug. Most University Computer Science programs are moving away from teaching Object Oriented Programming first and instead focusing on functional and procedural programming because it is easier to learn. There is a time and a place to use the power of classes and in Django CBVs. Consider carefully if your problem is really the kind where you need to use the CBV shaped hammer.</p>
<p><strong>Flat is Beautiful</strong></p>
<p>If you decide to reach for the CBV hammer, I think the flatter and the simpler a hierarchy is, the easier it will be to use. The deeper it goes and the more mixins that are created, the more semantic combinations you can end up with. You may find yourself solving problems which have nothing to do with your original one, but more on that later.</p>
<p>If you look at the Django admin you see it is really one giant flat CBV which is somewhat hard to understand, but easy to extend in specific ways. A better example is the syndication framework, a class which can be specialized in many ways with semantics clearly documented in one place.</p>
<p>Let's say you are going to write some kind of registration app. Instead of passing a bunch of arguments into a FBV, which does get kind of clunky, using CBV enables you to let your client specialize the behaviour to suit their needs. This works well because there is a single flow which could be specialized and because the docs for it could be found in one place. The same kind of thing could also be done by passing a behaviour object to a FBV, but I digress.</p>
<p>A flat CBV used to implement a utility view where there is a single common flow that will be specialized by each user in a subtle different way seems like a good use case. It follows that there are likely apps in contrib which could benefit from providing CBVs. Making CBVs the solution to every problem is a much taller order.</p>
<p><strong>CBVs vs DSLs</strong></p>
<p>Deep hierarchies of CBVs with various mixins can get out of hand. One can look at the new generic views to see an example.</p>
<p>I am reminded of a problem with Domain Specific Languages (DSL). A good DSL can create a powerful platform from which to solve problems, but if you set out to write a new language every time you want to solve a new problem, you will spend your time learning how to write languages and not focusing on your original problems. CBV are similar. You will spend time learning how to get the semantics of your hierarchy right instead of solving whatever problem you originally set out to solve. The deeper and more complex the harder you may find reusing the classes becomes.</p>
<p>It turns out semantics are hard. Hard to write and hard to maintain. Some problems would benefit from a solution built with the power of a reusable class hierarchy, but are you sure that is the first hammer you should reach for? Often using CBVs is akin to prematurely optimizing.</p>
<p>CBVs have a purpose to serve. If they can be made flatter I think it will make them easier to use. Either way FBV are stil my first love and the right way to introduce people to the simplicity of Django. If people advocating Flask can point to Django using FBV and say it is too complex, just imagine what they could say if CBVs became the default.</p>DjangoCon and PyCon Canada2012-08-20T11:11:00-04:00Albert O'Connortag:albertoconnor.ca,2012-08-20:djangocon-and-pycon-canada.html<p>This fall I am venturing into the conference circuit starting with <a href="http://www.djangocon.us/">DjangoCon US</a> in Washington DC, in September. I am definitely looking forward to seeing what it like to attend a major open source conference. I would also like to try to meet up with other Canadians making there way down to DjangoCon this year. Comment here or on find me on Twitter.</p>
<p>Then in November in Toronto an excellent group of Python developers are launching <a href="http://pycon.ca/">PyCon Canada</a> to help build community for 2014 when the official North American PyCon comes to Montreal. From the 9th the 11th of November in Toronto there will be a full fledged regional PyCon conference with talks, tutorials and sprints. Registration is open and they are accepting speaker applications.</p>
<p>The next obvious step will be to plan a conference of my own.</p>UnicodeEncodeError when uploading files in Django using Supervisor and Gunicorn2012-07-21T20:45:00-04:00Albert O'Connortag:albertoconnor.ca,2012-07-21:unicodeencodeerror-when-uploading-files-django-usi.html<p>Issues related uploading files with unicode filenames have long history with Django and Linux.</p>
<p>There are many stack overflow <a href="http://stackoverflow.com/questions/3715865/unicodeencodeerror-ascii-codec-cant-encode-character">questions</a> and <a href="http://jj.isgeek.net/2011/06/unicodeencodeerror-on-django-uploaded-files/">blog posts</a> on the subject already, but very few deal with solving the problem when using Supervisord and Gunicorn directly.</p>
<p>If you are using apache2 and mod_python you can configure your as per the docs: </p>
<p><a href="https://code.djangoproject.com/wiki/django_apache_and_mod_wsgi#AdditionalTweaking">https://code.djangoproject.com/wiki/django_apache_and_mod_wsgi#AdditionalTweaking</a></p>
<p>Fundamentally all fixes have the same goal: configure the environment of the server running Django to use the correct locale, so sys.getfilesystemencoding() will return 'UTF-8' instead of 'ANSI_X3.4-1968'. What makes solving this problem especially hard is there are many possible solutions and picking the right one for you situation can be difficult.</p>
<p><strong>Ignore Nginx Related Advice</strong></p>
<p>Since you are using Gunicorn, I am going to guess you are using Nginx and you will come across posts saying you need to add <code>charset utf-8</code> to the http section in the root Nginx config file. Though doing this is probably generally a good idea, it will not solve the problem. Nginx knows nothing about Python, Django or Gunicorn, except that when it makes requests down a unix or TCP socket it gets back bytes. No setting in Nginx will affect the environment Python is running in.</p>
<p><strong>Environment and Supervisord</strong></p>
<p>The most helpful <a href="http://stackoverflow.com/a/10986010/91243">advice</a> is to add a line similar to this:</p>
<pre>
environment=LANG=en_CA.UTF-8,LC_ALL=en_CA.UTF-8,LC_LANG=en_CA.UTF-8
</pre>
<p>To your supervisord.conf file in the [supervisord] section. Your locale maybe different.</p>
<p>This is good advice, but make sure that supervisor has actually read and is using this configuration directive is difficult. Supervisor tries hard not to completely restart since that means all of its processes/services will be forced to restart, but this is one situation where as far as I can tell it is necessary.</p>
<p>Unfortunately you cannot just do <code>/etc/init.d/supervisord restart</code>, since this doesn't really restart anything. You must issue a stop first, then issue a start.</p>
<p>You could potentially add the environment line to a specific supervisord .conf file, but that isn't supported in all versions.</p>
<p>You can use the following test supervisor program to help debug if your locale settings are working or not:</p>
<pre>
[program:locale]
command=python -c "import sys; print sys.getfilesystemencoding();"
directory=/
user=www-data
autostart=true
autorestart=true
redirect_stderr=True
</pre>
<p>You can adjust the settings to suit your purpose.</p>
<p>Also see this very handy blog post on the matter: <a href="http://tech.barszcz.info/2012/05/17/django-unicodeencodeerror-when-uploading-files/">http://tech.barszcz.info/2012/05/17/django-unicodeencodeerror-when-uploading-files/</a>.</p>
<p>If you discover more about how to solve this problem please contact me and I will update this post.</p>Rate your ISP2012-07-09T13:04:00-04:00Albert O'Connortag:albertoconnor.ca,2012-07-09:rate-your-isp.html<p><center>
<a href="http://ratemyisp.wildernesslabs.ca">
<img src="http://s3.amazonaws.com:80/albertoconnor/image_mananger%2Fratemyisp_banner.png" alt="ratemyisp" style="margin-bottom: 5px" />
</a>
</center></p>
<p>In June a <a href="http://ratemyisp.wildernesslabs.ca/about/">bunch</a> of great people went with me to a cottage for the weekend to create something. It was our 3rd <a href="http://wildernesslabs.ca/">Wilderness Labs</a> adventure. The outcome is pretty awesome.</p>
<p>We built a service to collect simple ISP reviews to help Canadians find their best local ISP. We called it <a href="http://ratemyisp.wildernesslabs.ca/">RateMyISP</a>. (Coming up with names might not be our strongest suit.)</p>
<p>
</p>
<p>The idea came from <a href="http://openmedia.ca/">Open Media</a>'s <a href="http://openmedia.ca/meter">Stop the Meter</a> campaign when more Canadians signed a petition protecting indy ISPs than use them. People who use the larger players still wanted the indy ISPs to exist so they would have an option when they become dissatisfied. Trouble is, finding the right local or indy ISP is kind of difficult. I found mine in London, ON thanks to a <a href="http://reddit.com">Reddit</a> comment which is no doubt now buried never to be found again. The comment listed the local ISP options per city in southern Ontario. There had to be a better way and now there is.</p>
<p>The service is ready to start absorbing reviews. As we gather reviews over the next few months we will use that information to vastly improve the search. We plan to highlight what areas indy ISPs provide special services for, and we hope that your reviews will help the great indy ISPs stand out from the rest.</p>
<p>I also wanted to acknowledge the many open source tools which made the project possible including <a href="http://djangoproject.com">Django</a>, <a href="http://jquery.com/">jQuery</a> and twitter's <a href="http://twitter.github.com/bootstrap/">Bootstrap</a>. These projects, among others, enabled us to go from concept to working service in a very small amount of time.</p>WatPy: Waterloo Region Python Group2012-05-30T16:35:00-04:00Albert O'Connortag:albertoconnor.ca,2012-05-30:watpy-waterloo-region-python-group.html<p><img src="http://s3.amazonaws.com:80/albertoconnor/image_mananger%2FWatPy.png" alt="WatPy" width="180" height="75" style="float: right; margin: 5px" /></p>
<p>Come hear what the newest tech group in Kitchener-Waterloo has to say at our first Peer 2 Peer event.</p>
<p>Thursday June 7th, 6:30pm at the Communitech Hub.</p>
<p>Featuring Brydon Gilliss and a handfull of lightning speakers.</p>
<p><a href="http://watpy.ca/blog/post/peer-2-peer-talks/">More Information</a> | <a href="http://events.r20.constantcontact.com/register/event?oeidk=a07e5x48s9ab46a25cf&llr=auyxwfdab">Register</a></p>
<p><strong>WatPy</strong>
</p>
<p>We started meeting 4 months ago, and so far the response has been awesome. Our focus is building community, teaching Python and talking about the language we all like a heck of a lot.</p>
<p>I am proud to be part of the beginnings of the community group. Python is a diverse language with uses for web development, scientific computing, psychology and wrapping C libraries. All of these activities are already represented by our group. Diversity complements a changing and innovative Region.</p>
<p>Beyond reaching out and building the community itself, I want to work with other community groups to help built tools which make running community groups easier. If you are interested contact me.</p>Export to CSV using the Django ORM2012-05-17T16:25:00-04:00Albert O'Connortag:albertoconnor.ca,2012-05-17:export-csv-using-django-orm.html<p>Django apps tend to be pretty data heavy. One common task is exporting sets of data to csv (comma separated values), a plain text file which can be loaded up in a spreadsheet and manipulated further. The first thing to do is make sure you know about the Python <a href="http://docs.python.org/library/csv.html">csv</a> library.</p>
<p>If you are exporting all of you data with respect to one model, here is a quick way to do it and keep your code cleaner using values_list and the Django ORM query shorthand. Start by defining your export as a data structure.</p>
<div class="highlight"><pre><span class="n">export_info</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">(</span><span class="s">"Role"</span><span class="p">,</span> <span class="s">"role__name"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"Department"</span><span class="p">,</span> <span class="s">"department"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"Last Name"</span><span class="p">,</span> <span class="s">"person__last_name"</span><span class="p">),</span>
<span class="p">(</span><span class="s">"First Name"</span><span class="p">,</span> <span class="s">"person__first_name"</span><span class="p">),</span>
<span class="p">]</span>
</pre></div>
<p>The first item in the tuple is the row header and the second item is an ORM path to the value you want. It will be passed in to values_list. One quirk to note is if you have a relationship and you don't specify the field on the relationship are you interested in you will just get the id for the object. The model's unicode method isn't called. For a better idea here is what the models would look like in this example.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Position</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">role</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="s">"Role"</span><span class="p">)</span>
<span class="n">department</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">person</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="s">"Person"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Role</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Person</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">first_name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">last_name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
</pre></div>
<p>The core logic looks like this.</p>
<div class="highlight"><pre><span class="k">global</span> <span class="n">export_info</span>
<span class="n">positions</span> <span class="o">=</span> <span class="n">Position</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">all</span><span class="p">()</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="s">"role__name"</span><span class="p">,</span>
<span class="s">"person__last_name"</span><span class="p">)</span>
<span class="c"># The inverse of zip is zip</span>
<span class="n">headers</span><span class="p">,</span> <span class="n">fields</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">export_info</span><span class="p">)</span>
<span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">headers</span><span class="p">]</span>
<span class="k">for</span> <span class="n">position</span> <span class="ow">in</span> <span class="n">positions</span><span class="p">:</span>
<span class="n">qs</span> <span class="o">=</span> <span class="n">Position</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">pk</span><span class="o">=</span><span class="n">position</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
<span class="c"># convert values like datetimes into unicode objects</span>
<span class="n">rows</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="nb">unicode</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">qs</span><span class="o">.</span><span class="n">values_list</span><span class="p">(</span><span class="o">*</span><span class="n">fields</span><span class="p">)[</span><span class="mi">0</span><span class="p">]])</span>
</pre></div>
<p>The last line is where the magic happens. I suggest going through it in your interpreter if you are confused about what it does. When this code is done you will have the list of rows which are your csv file. All that is left is to write the result to a proxy file and return it in a response.</p>
<div class="highlight"><pre><span class="n">f</span> <span class="o">=</span> <span class="n">StringIO</span><span class="o">.</span><span class="n">StringIO</span><span class="p">()</span>
<span class="n">writer</span> <span class="o">=</span> <span class="n">UnicodeWriter</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
<span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">(</span><span class="n">row</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">getvalue</span><span class="p">(),</span> <span class="n">mimetype</span><span class="o">=</span><span class="s">"text/csv"</span><span class="p">)</span>
<span class="n">response</span><span class="p">[</span><span class="s">'Content-Disposition'</span><span class="p">]</span> <span class="o">=</span> <span class="s">'attachment; filename=export.csv'</span>
<span class="k">return</span> <span class="n">response</span>
</pre></div>
<p>To get StringIO do "import cStringIO as StringIO". The UnicodeWriter for csv is a bit of custom code inspired by this <a href="http://docs.python.org/library/csv.html#examples">section</a> of the Python docs. You can use the normal csv library writer if all your data is ASCII and different file proxy object if you like.</p>
<p>
</p>
<p>Is there an even better way?</p>Using jQuery to sum the values in a table column2012-05-09T14:45:00-04:00Albert O'Connortag:albertoconnor.ca,2012-05-09:using-jquery-sum-values-table-column.html<p>Here is some neat jQuery based off of this <a href="http://naspinski.net/post/Use-jQuery-to-add-all-the-values-in-a-table-column_.aspx">post</a> [naspinski.net] that lets you sum up the values in a HTML table.</p>
<p>
</p>
<p>In general I would definitely use a different, usually database centric, approach to summing values. On the page I am currently working on though, there are going to be over 10 tables which are only useful with sums. As it is I am going to be doing many queries for this page, and it will be alright that the summing is a bit delayed.</p>
<p>
</p>
<p>First the HTML:</p>
<div class="highlight"><pre><span class="nt"><table</span> <span class="na">class=</span><span class="s">"tally"</span><span class="nt">></span>
<span class="nt"><th></span>balance<span class="nt"></th></span>
<span class="nt"><td></span>100<span class="nt"></td></span>
<span class="nt"><tr></span>
<span class="nt"><th></span>gains<span class="nt"></th></span>
<span class="nt"><td></span>20<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr></span>
<span class="nt"><th></span>loses<span class="nt"></th></span>
<span class="nt"><td></span>-36<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"><tr</span> <span class="na">class=</span><span class="s">"sum"</span><span class="nt">></span>
<span class="nt"><th></span>Total<span class="nt"></th></span>
<span class="nt"><td</span> <span class="na">class=</span><span class="s">"sum"</span><span class="nt">></span>#<span class="nt"></td></span>
<span class="nt"></tr></span>
<span class="nt"></table></span>
</pre></div>
<p>This will render a table with some values and "#" where the total should be. With some jQuery powered javascript we can do the summing:</p>
<div class="highlight"><pre><span class="kd">function</span> <span class="nx">sumOfColumns</span><span class="p">(</span><span class="nx">table</span><span class="p">,</span> <span class="nx">columnIndex</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">tot</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">table</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="s2">"tr"</span><span class="p">).</span><span class="nx">children</span><span class="p">(</span><span class="s2">"td:nth-child("</span> <span class="o">+</span> <span class="nx">columnIndex</span> <span class="o">+</span> <span class="s2">")"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$this</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">$this</span><span class="p">.</span><span class="nx">hasClass</span><span class="p">(</span><span class="s2">"sum"</span><span class="p">)</span> <span class="o">&&</span> <span class="nx">$this</span><span class="p">.</span><span class="nx">html</span><span class="p">()</span> <span class="o">!=</span> <span class="s2">""</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">tot</span> <span class="o">+=</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">$this</span><span class="p">.</span><span class="nx">html</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="k">return</span> <span class="nx">tot</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">do_sums</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"tr.sum"</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="nx">tr</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$tr</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="nx">tr</span><span class="p">);</span>
<span class="nx">$tr</span><span class="p">.</span><span class="nx">children</span><span class="p">().</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span> <span class="nx">td</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$td</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="nx">td</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">table</span> <span class="o">=</span> <span class="nx">$td</span><span class="p">.</span><span class="nx">parent</span><span class="p">().</span><span class="nx">parent</span><span class="p">().</span><span class="nx">parent</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">$td</span><span class="p">.</span><span class="nx">hasClass</span><span class="p">(</span><span class="s2">"sum"</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">$td</span><span class="p">.</span><span class="nx">html</span><span class="p">(</span><span class="nx">sumOfColumns</span><span class="p">(</span><span class="nx">table</span><span class="p">,</span> <span class="nx">i</span><span class="o">+</span><span class="mi">1</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="p">}</span>
</pre></div>
<p>You can use some css with the class "sum" to highlight your sums to make your tables easier to read. This should work with different types of tables, including summing multiple columns at once. Unfortunately, it won't work with tables which use colspans or rowspans.</p>Django 1.4 admin image issues with S3 backed static files2012-04-23T13:17:00-04:00Albert O'Connortag:albertoconnor.ca,2012-04-23:django-14-admin-image-issues-s3-backed-staticfiles.html<p>This is going to be kind of obscure, but hopefully it helps someone.</p>
<p>My current default setup for using S3 backed static files for Django broke when I upgraded to 1.4. The affect was that the admin media paths had been URL encoded so "/" was replaced with "%2F".</p>
<pre>
https://s3.amazonaws.com/bucket_name/admin%2Fcss%2Fbase.css
</pre>
<p>This caused images referenced in the css to fail because relative paths were interpreted (in Chrome at least) as being relative to the last unescaped slash instead of to /css. This also caused horizontal filters to fail as well.</p>
<h4>The Source</h4>
<p>The problem was exposed by a change in Django 1.4. Since Django no longer uses ADMIN_MEDIA_PREFIX, the storage system gets involved in creating the urls for the admin site media. I had been using a combination of the S3 Python library and django-storages. Django storages is basically doing the right thing, though it does have an opportunity to unescape the paths it gets from S3. The escaping itself is contained S3 library for Python, but I elected to take a different route to work around it.</p>
<h4>Solution</h4>
<p>My solution was to bypass the problem and switch S3Boto backend for django-storages. I have been using Boto for SES based email for a while, and though I found issues using S3Boto back in the day, I figured the different versions have hopefully have stabilized by now.</p>
<p>Switching to the S3Boto backend was painless, except for my AWS creds being included on every admin media url. Fortunately there is a setting to repress this behaviour when all your media is public.</p>
<pre class='"Python'>
STATIC_URL = 'https://BUCKET_NAME.s3.amazonaws.com/'
#DEFAULT_FILE_STORAGE = 'storages.backends.s3.S3Storage'
#STATICFILES_STORAGE = 'storages.backends.s3.S3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_QUERYSTRING_AUTH = False # Don't include auth in every url
AWS_STORAGE_BUCKET_NAME = BUCKET_NAME
</pre>
<p>The django-storages docs claim that Boto requires subdomain style access to S3. I am not convinced that is entirely true, but it does seem like a clean way to do it, so I made that switch at the same time.</p>Python Library for University of Waterloo Open Data API2012-04-06T14:28:00-04:00Albert O'Connortag:albertoconnor.ca,2012-04-06:python-library-university-waterloo-open-data-api.html<p><a href="http://en.wikipedia.org/wiki/Open_data">Open Data</a> is awesome.</p>
<p>I think there is real value in information made available through flexible standards such as JSON for build the next generation of useful web experiences.</p>
<p>The Open Data initiative at the University of Waterloo has gathered their api at <a href="http://api.uwaterloo.ca/">api.uwaterloo.ca</a> which gives you access to courses, parking, and the weather.</p>
<p>I created a Python library to wrap the API to make it even easier to use:</p>
<p><a href="https://bitbucket.org/amjoconn/uwaterlooapi">https://bitbucket.org/amjoconn/uwaterlooapi</a>*</p>
<p><small>*github version to come soon</small></p>
<pre class="Python">
>>> from uwaterlooapi import UWaterlooAPI
>>> uw = UWaterlooAPI(api_key="YOUR API KEY")
>>> uw.weather()
{u'Date': u'04-01-2012', u'Current': {u'Windchill': u'NA', u'Temp': u'4.1', ...
</pre>
<p>I was bit surprised not to find a Python library designed to make building REST API wrappers easier. My approach, like most API wrappers, is a good start, but lacks the elegance of a solution applied to many different instances of the problem.</p>
<p>Let me know if you create something using this library.</p>Viewport: Making webpages look great on iOS devices2012-01-19T13:52:00-05:00Albert O'Connortag:albertoconnor.ca,2012-01-19:viewport-making-webpages-look-great-ios-devices.html<pre>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
</pre>
<p>This is the default viewport meta tag found in the very useful <a href="http://html5boilerplate.com/">html5 boilerplate</a>.</p>
<p>If you are like me, and don't initially understand exactly what it does, this default <strong>will</strong> get you into trouble.</p>
<p>The main issue is your website will look great on your computer but when you look at it on an iPhone or iPad it will look small, squished even depending on your CSS. If you happen be making a website which is pixel perfect for the iPhone resolution, then everything will look good, but most websites I develop are meant to be viewed on computer browser and on mobile devices, so knowing what the viewport meta tag does helps you look great everywhere.</p>
<p><strong>What is the viewport</strong>
<p>The viewport is the space your webpage will rendered into. Think of it like the size of the browser window. When you zoom, you are zooming into the viewport, but the size of the viewport doesn't change. You can zoom out of the viewport either, though if the page is wider than the viewport you can scroll to see more.</p>
<p>The above viewport meta tag says make the viewport the same size as the device's screen width and a 1:1 scale, pixel perfect. In the case of an iPhone that means 320px wide. If your CSS is designed for 800px of minimum width your page is guaranteed to look bad.</p>
<p>You can specify any default width in pixels, and default scale or let the scale be determined automatically.</p>
<p>For more detail you can consult the in-depth Apple documentation on the <a href="http://developer.apple.com/library/IOs/#documentation/AppleApplications/Reference/SafariWebContent/UsingtheViewport/UsingtheViewport.html">viewport meta tag</a>.</p>
<strong>Looking good everywhere</strong>
<p>If you have designed your website to be rendered at a minimum width of say 1000 pixels, you can make it look great on a computer and on a mobile device with the following meta tag:</p>
<pre>
<meta name="viewport" content="width=1000">
</pre>
<p>This will make the viewport, the space the website renders into, 1000 pixels wide, and default the scaling to show the entire width.</p>
<p>By default if no viewport meta tag is provided a default viewport width 985 pixel is used, which will actually works well for most sites. If you want your site to look really good and you have an optimal width of 600 pixels, explicitly setting the viewport width to 600 pixels ("width=600") will make it look just as you intend. If your sites optimal width is smaller than 985 pixel it is good to be explicit, otherwise your pages might appear kind of far away and the user will have to manually zoom in.</p></p>Good Night Patel2011-12-04T17:21:00-05:00Albert O'Connortag:albertoconnor.ca,2011-12-04:good-night-patel.html<p>Today is the day I decommission the first server I created for this venture.</p>
<p>Over a year ago I started investigating how modern web hosting works and discovered the awesome price points you can get for small servers designed to serve small amounts of traffic. Not only were these cloud based virtual machines inexpensive, they weren't tied to hardware allowing the providers to provide excellent up time.</p>
<p>I named my first server Patel, and launched him on Rackspace's Cloud. He used to serve this very site until a few months ago. Overall I was happy with Rackspace. They provided a very simplistic introduction to cloud computing which readied me for my next infrastructure evolution.</p>
<p>Amazon Web Service's free tier did its job on me, opening the door to exploring what Amazon had to offer. Everything was much more complicated, but also far more powerful and more secure. The key power I appreciate the most is the API controlled firewall which you can adjust through the web interface for AWS. In general I also felt like my instances were better protected by default. It is in Amazon's interest to protect every instance on their cloud including mine.</p>
<p>I have moved up from having one server on AWS for myself and select clients to planning a scalable infrastructure both in terms of resource allocation and security. The core idea is to heavily secure and monitor key front end instances while the app server and database servers are kept inaccessible from the outside world. This means more resources can be provisioned without needing to increase the amount of security information to review.</p>
<p>The end of Patel is the end of an era for me, but what comes next is even more exciting. Contact me if you want help getting into this new way of creating awesome websites and web services.</p>Hosting Django under different locations with Nginx and gunicorn2011-09-15T16:40:00-04:00Albert O'Connortag:albertoconnor.ca,2011-09-15:hosting-django-under-different-locations.html<p>Do you know how Django uses SCRIPT_NAME?</p>
<p>It's not often I host different instances of Django under different locations on the same domain. One legacy install has been setup that way for over a year with Apache2 and mod_wsgi (with Nginx infront). Almost all of my new deployments use virtual hosts with different subdomains for each service.</p>
<p>If you are using virtual hosts with Nginx it looks like this:</p>
<pre>
server {
listen 80;
server_name service.albertoconnor.ca;
# other settings
location / {
try_files $uri/index.html $uri.html $uri @cluster;
}
location @cluster {
proxy_pass http://service.albertoconnor.ca;
}
}
</pre>
<p>Assuming you have defined an upstream server cluster called "service.albertoconnor.ca".</p>
<p>If you are doing something by location on the same domain it might look something more like this:</p>
<pre>
server {
listen 80;
server_name _;
# other settings...
location /a/ {
proxy_pass http://127.0.0.1:8001/;
}
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
}
}
</pre>
<p>The idea is that something else, let's say Apache2, is running on 127.0.0.1:8080, and Django being served by gunicorn is running on 127.0.0.1:8001. So under /a/ the pages will be generated by Django; anywhere else Apache2 takes over.</p>
<p>If you try this though, things will go horribly wrong. Besides redirects not working because you need some proxy_redirect mojo, your generated links in your templates won't work. They will generate absolute urls like "/foo/bar" instead of "/a/foo/bar". But really how should {% url %} know any better.</p>
<p>Then I got that feeling. You know when you figure out why something doesn't work, but because of that reason you can't explain why other things ever worked in the first place. I have been successfully serving different Django services based on location with Apache2 and mod_wsgi for over a year. How, when I generate urls, could it know to prepend the location?</p>
<p>The answer is that mod_wsgi WSGIScriptAlias command was helping me. Django will actually do the magic prepend for url generation if the SCRIPT_NAME variable is set in the wsgi environment. mod_python also does this by the django.root PythonOption. The SCRIPT_NAME variable is mentioned in the gunicorn FAQ very <a href="http://gunicorn.org/faq.html">briefly</a> and without context.</p>
<p>From the FAQ and other hints around the internet I gathered that I really just need to set the header SCRIPT_NAME HTTP header myself and things will start working. Since the SCRIPT_NAME is also taken off of incoming urls I had to alter my proxy_pass a bit. Here is what I ended up with:</p>
<pre>
# ...
location /a/ {
proxy_pass http://127.0.0.1:8001/a/;
proxy_redirect http://127.0.0.1:8001/a/ http://$host/a/;
proxy_set_header SCRIPT_NAME /a;
}
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
}
# ...
</pre>
<p>I also needed a slightly different proxy_redirect directive than the default. This may not be ideal so suggestions are welcome. I will apply them to this post. You can comment or contact me though this website.</p>
<p><strong>Update Oct 26th, 2011</strong></p>
<p>When using CAS with services I discovered the redirection url has the internal host IP rather than the proxy host name. One way to fix it is to use the X_Forwarded_Host header. In the /a/ block add:</p>
<pre>
proxy_set_header X-Forwarded-Host your.host.name.com;
</pre>
<p>Internally Django has a HttpRequest.get_host method that if the setting USE_X_FORWARDED_HOST is True will use the value of the HTTP-X-FORWARDED-HOST header as the current request's host.</p>
<pre>
USE_X_FORWARDED_HOST = True
</pre>
<p><strong>NOTE:</strong> The setting is new as of Django 1.3.1 and it is worth reading <a href="https://www.djangoproject.com/weblog/2011/sep/09/security-releases-issued/">security advisory</a>, see the section headed "Host header cache poisoning". The up shot of which is in this case seems to be to filter on your specific host and not "server _;" as I did in my example above to avoid a security hole.</p>jQuery fallback for HTML5 placeholder attribute2011-08-21T19:56:00-04:00Albert O'Connortag:albertoconnor.ca,2011-08-21:jquery-fallback-html5-placeholder-attribute.html<p>If you google placeholder and jQuery you quickly find some handy code for a javascript fallback for the HTML5 placeholder functionality:</p>
<p><a href="https://gist.github.com/379601">https://gist.github.com/379601</a></p>
<p>If you want to read more check out the <a href="http://www.hagenburger.net/BLOG/HTML5-Input-Placeholder-Fix-With-jQuery.html">blog post</a>.</p>
<p>As written this will always override the built in browser behaviour, and it isn't as good as the built in behaviour. We only want this code to be registered if the current browser doesn't natively support placeholder, fortunately detection is easy thanks to Dive Into HTML5</p>
<p><a href="http://diveintohtml5.org/everything.html#placeholder">http://diveintohtml5.org/everything.html#placeholder</a></p>
<p>This leads to this combination of code:</p>
<div class="highlight"><pre><span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="s1">'placeholder'</span> <span class="k">in</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s1">'input'</span><span class="p">))</span> <span class="o">||</span> <span class="o">!</span><span class="p">(</span><span class="s1">'placeholder'</span> <span class="k">in</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s1">'textarea'</span><span class="p">)))</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="nb">document</span><span class="p">).</span><span class="nx">ready</span><span class="p">(</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s1">'[placeholder]'</span><span class="p">).</span><span class="nx">focus</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">input</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">()</span> <span class="o">==</span> <span class="nx">input</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">(</span><span class="s1">''</span><span class="p">);</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">removeClass</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}).</span><span class="nx">blur</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">input</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">()</span> <span class="o">==</span> <span class="s1">''</span> <span class="o">||</span> <span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">()</span> <span class="o">==</span> <span class="nx">input</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">addClass</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">);</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}).</span><span class="nx">blur</span><span class="p">().</span><span class="nx">parents</span><span class="p">(</span><span class="s1">'form'</span><span class="p">).</span><span class="nx">submit</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">find</span><span class="p">(</span><span class="s1">'[placeholder]'</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">input</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">()</span> <span class="o">==</span> <span class="nx">input</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'placeholder'</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">input</span><span class="p">.</span><span class="nx">val</span><span class="p">(</span><span class="s1">''</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="p">});</span>
<span class="p">}</span>
</pre></div>
<p>The code makes the somewhat broad assumption that no browser supports placeholder for an input but not for a textarea, or if it did, the behaviour shouldn't be overridden. Part of the appeal for HTML5 for me is the ability to provide great user experience for those on modern browser and good experiences on those forced not to upgrade.</p>Wilderness Labs2011-06-28T16:10:00-04:00Albert O'Connortag:albertoconnor.ca,2011-06-28:wilderness-labs.html<p><a href="http://wildernesslabs.ca">
<div style="float:right; margin: 0.5em;">
<img src="https://s3.amazonaws.com/wildernesslabs/img/logo2.png" width="250" />
</div>
</a>
</p>
<p>In December 2010, inspired by several interesting events I had heard about, I started a unique project with a few of my good friends. The idea is to find out what happens when you go away for the weekend with some talented people to a beautiful cottage with the intention to create something.</p>
<p>We decided to call it <a href="http://wildernesslabs.ca">Wilderness Labs</a>.</p>
<p>Inspiration came from the likes of <a href="http://devfort.com/">/dev/fort</a> and the <a href="http://7cubedproject.com/">7 Cubed</a> project. Since there aren't as many rent-able forts in Southern Ontario, cottages seemed like the logical replacement.</p>
<p>Some 8 months on we have already run our first event and are planning to go back on the weekend of Nov 11th to 13th. We are excited to share the experience with some new people, so if it sounds like the kind of thing you would be interested in please let us know on our <a href="http://wildernesslabs.ca/get_involved/">get involved</a> page.</p>
<p>If the first event is any indication we will be doing this for a long time. Going out to a cottage, enjoying a late night camp fire and being able to walk down to the beach made for a weekend which was both memorable and productive.</p>Best Practices: Off-Site Backups2011-05-31T23:52:00-04:00Albert O'Connortag:albertoconnor.ca,2011-05-31:best-practices-site-backups.html<p>Having a backup strategy is key. What works for me won't necessary work for you. I am going to outline my method for my own reference, to get feedback, and in the hope it might help someone else out there.</p>
<p>I am using <a href="http://aws.amazon.com/s3/">Amazon S3</a> as my back-end, taking advantage of 10 GB for a year for free. I am abusing <a href="http://fabfile.org/">Fabric</a> as my glue. Fabric is meant for remote execution but I am using its local feature as a kind of Pythonic shell script.</p>
<p>The first major issue is security. S3 is private by default but you are implicitly trusting Amazon. The only solution is to encrypt your data, which is easier said then done. The core of this post is documenting my method for encryption.</p>
<p><strong>Encrypting Your Bits</strong>
<p>I use OpenSSL to do the encryption instead of GPG or PGP because it is available on my Mac and widely on Linux without apt-getting.</p>
<p>Conceptually a backup is like a message sent from present day me to future me using asymmetric, or public/private key, encryption. I want to encrypt the backup with my public key and decrypt it later with my private key.</p>
<p>The trouble is that with OpenSSL asymmetric encryption can only be used for small files. The solution is to generate a unique symmetric key for every backup and encrypt that small file with my public key. I save the symmetrically encrypted backup and the asymmetrically encrypted symmetric key together and send them both to S3. Without your private key the two files get you nothing since they are both encrypted.</p>
<p>The first thing you have to once and only once is generate the public/private keys for encrypting the symmetric encryption keys (passwords):</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">create_keys</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Only do this once, if you overwrite your keys you won't be able to decrypt your backups!</span>
<span class="sd"> """</span>
<span class="n">key_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="s">"keys/backup.pem"</span><span class="p">)</span>
<span class="n">local</span><span class="p">(</span><span class="s">"openssl genrsa -des3 -out </span><span class="si">%s</span><span class="s"> 1024"</span> <span class="o">%</span> <span class="n">key_file_name</span><span class="p">)</span> <span class="c"># Generate encrypted private key</span>
<span class="n">local</span><span class="p">(</span><span class="s">"openssl rsa -in </span><span class="si">%s</span><span class="s"> -pubout > </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">key_file_name</span><span class="p">,</span> <span class="n">key_file_name</span> <span class="o">+</span> <span class="s">".pub"</span><span class="p">))</span> <span class="c">#Output the public part</span>
</pre></div></p>
<p>Here is the meat of a backup:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">backup_to_s3</span><span class="p">():</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="nb">id</span> <span class="o">=</span> <span class="n">today</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"</span><span class="si">%d</span><span class="s">-%m-%Y"</span><span class="p">)</span>
<span class="c"># Tar everything you want to backup. Assume a list of directories in backup_dirs.</span>
<span class="n">backup_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">relative_store</span><span class="p">,</span> <span class="s">"backup-</span><span class="si">%s</span><span class="s">.tar.gz"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">)</span>
<span class="n">local</span><span class="p">(</span><span class="s">"tar -zcf </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">backup_file_name</span><span class="p">,</span><span class="s">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">backup_dirs</span><span class="p">)))</span>
<span class="c"># Generate a 64 character symmetric key</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="mi">64</span><span class="p">)</span>
<span class="n">key_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">relative_store</span><span class="p">,</span> <span class="s">"backup-</span><span class="si">%s</span><span class="s">.key"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">)</span>
<span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">local_store</span><span class="p">,</span> <span class="s">"backup-</span><span class="si">%s</span><span class="s">.key"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">),</span> <span class="s">"w"</span><span class="p">)</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="c"># Encrypt the backup with the symmetric key</span>
<span class="n">local</span><span class="p">(</span><span class="s">"openssl enc -e -aes128 -pass file:</span><span class="si">%s</span><span class="s"> < </span><span class="si">%s</span><span class="s"> > </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">key_file_name</span><span class="p">,</span>
<span class="n">backup_file_name</span><span class="p">,</span>
<span class="n">backup_file_name</span> <span class="o">+</span> <span class="s">".enc"</span><span class="p">))</span>
<span class="c"># Encrypt the symmetric key with the public key generated before</span>
<span class="n">public_key_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="s">"keys/backup.pem.pub"</span><span class="p">)</span>
<span class="n">local</span><span class="p">(</span><span class="s">"openssl rsautl -encrypt -inkey </span><span class="si">%s</span><span class="s"> -pubin -in </span><span class="si">%s</span><span class="s"> -out </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">public_key_name</span><span class="p">,</span>
<span class="n">key_file_name</span><span class="p">,</span>
<span class="n">key_file_name</span> <span class="o">+</span> <span class="s">".pubenc"</span><span class="p">))</span>
<span class="c"># Securely remove the unencrypted symmetric key file</span>
<span class="n">local</span><span class="p">(</span><span class="s">"srm </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">key_file_name</span><span class="p">)</span>
<span class="c"># Tar the encrypted backup and encrypted key together</span>
<span class="n">unfied_backup_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">relative_store</span><span class="p">,</span> <span class="s">"unified_backup-</span><span class="si">%s</span><span class="s">.tar.gz"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">)</span>
<span class="n">local</span><span class="p">(</span><span class="s">"tar -zcf </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">unfied_backup_file_name</span><span class="p">,</span>
<span class="n">backup_file_name</span> <span class="o">+</span> <span class="s">".enc"</span><span class="p">,</span>
<span class="n">key_file_name</span> <span class="o">+</span> <span class="s">".pubenc"</span><span class="p">))</span>
<span class="c"># Push the result to s3</span>
<span class="n">local</span><span class="p">(</span><span class="s">"s3put -a </span><span class="si">%s</span><span class="s"> -s </span><span class="si">%s</span><span class="s"> -b amjoconn-backups -p </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">ACCESS_KEY</span><span class="p">,</span>
<span class="n">SECRET_KEY</span><span class="p">,</span>
<span class="n">local_store</span><span class="p">,</span>
<span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">unfied_backup_file_name</span><span class="p">)))</span>
</pre></div>
<p>This script has several global configuration variables which aren't addressed. It also won't clean up your local store except to delete the unencrypted key.</p>
<p><strong>Recovery</strong>
<p>Like an claiming an insurance policy, recovery is the most important part of any backup system. Like insurance people tend not to talk too much about how it will work nor do they test it often. Given the complexity of the backup process, I figured outlining the recovery process even if the automation ultimately fails me would be handy for reference.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">recover_backup</span><span class="p">(</span><span class="nb">id</span><span class="p">):</span>
<span class="c"># Try to get the file</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">boto</span><span class="o">.</span><span class="n">connect_s3</span><span class="p">(</span><span class="n">ACCESS_KEY</span><span class="p">,</span> <span class="n">SECRET_KEY</span><span class="p">)</span>
<span class="n">bucket</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="s">"amjoconn-backups"</span><span class="p">)</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="s">"unified_backup-</span><span class="si">%s</span><span class="s">.tar.gz"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">)</span>
<span class="k">if</span> <span class="n">key</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Unable to find backup with id </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="nb">id</span>
<span class="k">return</span>
<span class="n">recover_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">recovery_location</span><span class="p">,</span> <span class="s">"recover-</span><span class="si">%s</span><span class="s">.tar.gz"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"Writing"</span><span class="p">,</span> <span class="s">"unified_backup-</span><span class="si">%s</span><span class="s">.tar.gz"</span> <span class="o">%</span> <span class="nb">id</span><span class="p">,</span> <span class="s">"to"</span><span class="p">,</span> <span class="n">recover_name</span>
<span class="k">print</span> <span class="s">"This might take a bit of time..."</span>
<span class="n">key</span><span class="o">.</span><span class="n">get_contents_to_filename</span><span class="p">(</span><span class="n">recover_name</span><span class="p">)</span>
<span class="n">local</span><span class="p">(</span><span class="s">"tar -xzf </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">recover_name</span><span class="p">)</span></p>
<div class="codehilite"><pre><span></span><span class="nt"><span</span> <span class="na">class=</span><span class="s">"c"</span><span class="nt">></span># Decrypt the symmetric key file with the private key.<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>key_file_name<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>=<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>os<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>path<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>join<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>relative_store<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>backup-<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span>.key<span class="ni">&quot;</span><span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>%<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"nb"</span><span class="nt">></span>id<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>)<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>private_key_name<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>=<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>os<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>path<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>join<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>root<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>keys/backup.pem<span class="ni">&quot;</span><span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>)<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>local<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>openssl rsautl -decrypt -inkey <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span> -in <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span> -out <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span><span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>%<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>private_key_name<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>key_file_name<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>+<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>.pubenc<span class="ni">&quot;</span><span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>key_file_name<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>))<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"c"</span><span class="nt">></span># Decrypt the backup with the recently decrypted symmetric key.<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>backup_file_name<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>=<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>os<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>path<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>.<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>join<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>relative_store<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>backup-<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span>.tar.gz<span class="ni">&quot;</span><span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>%<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"nb"</span><span class="nt">></span>id<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>)<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>local<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>openssl enc -d -aes128 -pass file:<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span> <span class="ni">&lt;</span> <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span> <span class="ni">&gt;</span> <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span><span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>%<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>key_file_name<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>backup_file_name<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>+<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>.enc<span class="ni">&quot;</span><span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>,<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>backup_file_name<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>))<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"c"</span><span class="nt">></span># Untar the data here. Depending how you used tar, there will be some interesting directories created.<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>local<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>(<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span>tar -zxf <span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"si"</span><span class="nt">></span>%s<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"s"</span><span class="nt">></span><span class="ni">&quot;</span><span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"o"</span><span class="nt">></span>%<span class="nt"></span></span> <span class="nt"><span</span> <span class="na">class=</span><span class="s">"n"</span><span class="nt">></span>backup_file_name<span class="nt"></span><span</span> <span class="na">class=</span><span class="s">"p"</span><span class="nt">></span>)<span class="nt"></span></span>
</pre></div>
<p></pre></div></p>
<p>Note, this doesn't clean up the decrypted symmetric key.</p>
<p><strong>Wrap Up</strong>
<p>There it is. How I am keeping my bits safe. Such a problem is never really solved though. I hope to be able to update this post over time as I discover better ways to backup because this solution has much room for improvement.</p></p>Best Practices: Backups2011-04-02T22:23:00-04:00Albert O'Connortag:albertoconnor.ca,2011-04-02:best-practices-backups.html<p>My <a href="http://albertoconnor.ca/blog/2010/Dec/3/best-practices-passwords">last</a> post was on the importance of password management. Another critical best practice when you deal with other peoples data is backups.</p>
<p>Since the beginning of the digital age retaining data for both the short and long term has become an open problem or sorts. Fortunately fragile bits are easy to copy but every type of media has a limited lifetime, especially hard disks. Coming up with the right balance to ensure retention is tricky. Much like claim is the most important part of insurance, recovery is the most important aspect of any backup system and testing it can be tricky.</p>
<p>I will outline my approach to backing up data, but the real key is to have a strategy and to think about the problem. What works for me won't necessarily work for you. Don't be reliant a old beige computer in an office somewhere which you hope won't die on you.</p>
<p><strong>Time Machine and Time Capsule</strong>
<p>If you are running Mac OS X definitely take advantage of Time Machine. It can do automated hourly backups with a well tested recovery mechanism. A feature was added under System Preferences, Time Machine, Options... to enable or disable doing backups on battery power. I find disabling help keep performance up. My backups may not be hourly but when they happen they don't drag my system down.</p>
<p>Apple's Time Capsule sounds like a destination for your backups, but in my experience it is some what questionable. Apple has had some <a href="http://timecapsuledead.org/">quality issues</a> with Time Capsules. My first Time Capsule was newer than the generation which has officially be designated as flawed, but it still died after about 18 months. It died in such a way that the hard disk seemed functional, but I wasn't allowed to remove it and get warranty replace of the Time Capsule unless I paid an Apple certified tech to do it. The removal wasn't covered under warranty. For a data backup device this doesn't seem like a sensible policy.</p>
<p>There are likely better options which allow the removal of your disks if the device itself fails. I am currently still using my replacement Time Capsule, but when it dies in 18 months I will definitely look into replacing it with something else. Likely some kind of network hard disk since Gigabit Ethernet can move bytes faster than USB 2.0 which is important when moving around large 100 GB backup files when dealing with failure.</p>
<strong>Two Separate Devices and Off-site Backups</strong>
<p>The lesson I have learned is data really isn't backed up unless it is on two separate devices, not just two separate drives. The more critical data is the more places it should be stored. Often the most critical things are also things which need to be kept secure. Having many copies and security seem contradictory, but with a bit of encryption built on the passwords system I outline <a href="http://albertoconnor.ca/blog/2010/Dec/3/best-practices-passwords">last post</a>, both can be achieved.</p>
<p>The best backup strategies include off-site backups. The risk to your data are beyond just device failure. There is also external issues such as fire or other disaster. With data hosting services like Amazon's S3, Rackspace's Cloudfiles and others, you can take advantage of off-site backups while hooking into major players redundant storage infrastructure at a great price.</p>
<p>Unfortunately universal tools to make this easy and accessible to everyone aren't quite mature yet, but make no mistake they are on their way.</p>
<p>In my next post I will outline how I backup my most critical information to Amazon S3 in a secure way using Python, Fabric and OpenSSL.</p></p>Best Practices: Passwords2010-12-03T13:39:00-05:00Albert O'Connortag:albertoconnor.ca,2010-12-03:best-practices-passwords.html<p>When I decided to start working for myself, I felt I needed to establish best practices as soon as possible, while I still had time. My clients would be trusting me to with their information and access to their systems and I wanted to make sure I was being responsible.</p>
<p>Password management is one area I targeted. Even if you aren't in business for yourself, being secure online basically requires a good password management scheme.</p>
<p><strong>Existing Options</strong>
<p>There are a lot of solutions and partial solutions out there. For example browsers will remember arbitrary passwords for you, but that is only most convenient when you are using that browser. Many of my passwords are for ssh accounts and I planned on using multiple users on my laptop.</p>
<p>Other 3rd party solutions exists like <a href="http://www.splashdata.com/">Splash Data</a> and <a href="http://agilewebsolutions.com/onepassword">1Password</a>. My problem with their approach was not being clear how to backup and recover the data cross computers. As I will write about soon, backups is one of the other best practices I focused on.</p>
<strong>Rolling Your Own</strong>
<p>I talked to my resident cryptography expert <a href="http://www.douglas.stebila.ca/">Douglas Stebila</a> about how he was using a simple system which seem to fit my requirements. It was easy to use, backup, and completely transparent.</p>
<p>This approach is best for those who understand the command line, shell scripts and grep and who are using Unix compatible systems like Linux or Mac OS X. Cygwin would also work.</p>
<p>The core idea is to store all your password in an OpenSSL encrypted file. You decrypt the file and pass it through grep to look up passwords. I use a format of "description: password", but everything is customizable. I implemented a series of shell scripts and one Python script to make it all work.</p>
<p>First lets look at the passwords script.</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
openssl enc -d -aes128 < /somewhere/accessible/passwords.aes128 | grep -i <span class="nv">$1</span></p>
<p><span class="c"># Usage: passwords blah</span>
<span class="c"># password: <Enter master password></span>
<span class="c"># Finds all descripts or passwords with "blah" in them</span>
</pre></div></p>
<p>I make sure that the passwords script is always on the path and the passwords.aes128 file is accessible. It is important you make sure your master password is a strong password since it protects everything.</p>
<p>The master password is really just the password you encrypt the passwords.aes128 file with, so when you are editing it and re-encrypting it, make sure you don't mistype it or you could loose your password archive. Again a backup strategy is important.</p>
<p>I manage the file with a set of scripts. First there is passwords-decompress and passwords-edit:</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
openssl enc -d -aes128 < /somewhere/accessible/passwords.aes128 >/somewhere/accessible/passwords.aes128.decrypted
</pre></div>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
vi /somewhere/accessible/passwords.aes128.decrypted
</pre></div>
<p>Most of these scripts are just so I need to remember paths and arguments. I want adding passwords to be as easy as possible. To generate a new password I use something like the following Python script.</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">random</span>
<span class="n">chars</span> <span class="o">=</span> <span class="s">"1234567890qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM"</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Password Helper. Albert O'Connor. 2010"</span>
<span class="n">desc</span> <span class="o">=</span> <span class="nb">raw_input</span><span class="p">(</span><span class="s">"Enter password description: "</span><span class="p">)</span>
<span class="n">password</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">8</span><span class="p">):</span>
<span class="n">password</span> <span class="o">+=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span>
<span class="k">print</span> <span class="s">"</span><span class="si">%s</span><span class="s">: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">desc</span><span class="p">,</span> <span class="n">password</span><span class="p">)</span>
</pre></div>
<p>The last line this script prints out is what I copy and paste into the passwords file. If you want to be more secure about it you can use <a href="http://www.dlitz.net/software/pycrypto/doc/#crypto-random">Python Cryptography's Random</a>. Once I have edited the decrypted passwords my satisfaction I run passwords-compress.</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
openssl enc -e -aes128 < /somewhere/accessible/passwords.aes128.decrypted > /somewhere/accessible/passwords.aes128
</pre></div>
<p>This is where you have to be careful, since consistently mistyping your master password can lead to inaccessible data. I always test out the new password I added with passwords before backing up with passwords-backup.</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
cp /somewhere/accessible/passwords.aes128 /somewhere/accessible/passwords.aes128.backup
scp /somewhere/accessible/passwords.aes128 user@server.org:~/backup/
</pre></div>
<p>This makes a local and off site backup of the encrypted file. Finally I clean up with passwords-cleanup:</p>
<div class="highlight"><pre><span class="c">#!/bin/bash</span>
srm /somewhere/accessible/passwords.aes128.decrypted
</pre></div>
<p>Because leaving decrypted versions of your passwords file around totally defeats the purpose. Now whenever you need a password, open a prompt, type passwords blah, enter your master password, and you will get all your passwords for blah. Close the terminal when you are done though!</p>
<p>No matter what, if you will be freelancing in a digital world, keeping track of passwords is something you have to do. This method is simple, the encryption is secure enough for the <a href="http://en.wikipedia.org/wiki/Advanced_Encryption_Standard">NSA</a>, and highly usable. If it doesn't suit your taste employ another one, just make sure you are using one.</p>Autocomplete for Foreign Keys in the Django Admin Explained2010-09-03T21:26:00-04:00Albert O'Connortag:albertoconnor.ca,2010-09-03:autocomplete-foreign-keys-in-django.html<p>When the select drop-down for foreign keys in the Django Admin gets big it can become a usability issue. One solution is to create a custom UI for the the model, but stretching the Django Admin to the limit is one of the fun parts of working with Django.</p>
<h4>Prior Art</h4>
<p>The top hit on Google is <a href="http://code.google.com/p/django-autocomplete/">django-autocomplete</a>. This project is the basis of my approach and therefore much thanks goes to the author. There hasn't been an update since early 2009 and I wasn't able to get it working without a lot of changes. The changes evolved into what I consider a better approach.</p>
<p>Another interesting project is <a href="http://code.google.com/p/django-ajax-selects/">django-ajax-selects</a> which is very fancy, complicated looking and actively maintained. It allows for very pretty autocompletes. My solution does it without any AJAX so it is a bit easier to use and makes for an easier case study. An AJAX based solution solves another critical problem: the drop down being so large it takes a long time for the admin page to render. Raw id fields also help to solve this problem albeit in a less elegant way.</p>
<p>There are a few other projects, some using YUI, some focusing on many to many fields. The solution presented here is jQuery based and for foreign keys in the admin only. This post attempts to explain the solution end to end and is based on Django 1.2.x.</p>
<h4>Progressive Enhancement</h4>
<p><a href="http://en.wikipedia.org/wiki/Progressive_enhancement">Progressive enhancement</a> is one of the key benefits of my approach. Instead of having a hidden input element to hold the real id of the foreign object, I actually keep the select drop-down and merely hide it and show the autocomplete field if Javascript is enabled.
<h4>Usage</h4></p>
<p>Once complete, using it will look something like this inside an admin.py file:</p>
<div class="highlight"><pre><span class="n">admin</span><span class="o">.</span><span class="n">site</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Foreign</span><span class="p">,</span> <span class="n">AutocompleteTargetModelAdmin</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">MainAdmin</span><span class="p">(</span><span class="n">AutocompleteModelAdmin</span><span class="p">):</span>
<span class="c">#...</span>
<span class="n">autocomplete_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s">'foreign'</span><span class="p">,)</span>
<span class="c">#...</span>
<span class="n">admin</span><span class="o">.</span><span class="n">site</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Main</span><span class="p">,</span> <span class="n">MainAdmin</span><span class="p">)</span>
</pre></div>
<p>Where the Main model has a ForeignKey, foreign, to the model named Foreign.</p>
<h4>What You Will Need</h4>
<p>In addition to Django you will need:</p>
<ul>
<li><a href="http://www.jquery.com">jquery</a></li>
<li><a href="http://docs.jquery.com/Plugins/Autocomplete">jquery.autocomplete</a></li>
</ul>
<h4>Getting The Autocomplete Data</h4>
<p>Normally the hard part is providing an AJAX response from some url, for which you have to define a view which provides the possible values. Requiring the client to do this every time is undesirable. There is a simpler way. Autocomplete can also work on static arrays, and since the drop-down is static it should work just as well. Actually, the select drop-down we are keeping hidden <strong>is</strong> a static list of possible values! We will use the select to generate a static array of values to populate the autocomplete.</p>
<h4>The New Hard Part</h4>
<p>The only really hard part left is keeping the "+" adding of items in the Django Admin working. This will force us to do one zany thing: require that our foreign model's admin class is a subclass of a special class we provide, the so called AutocompleteTargetAdminModel. More on this shortly.</p>
<h4>Let's Do It</h4>
<p>Now that you have some idea of what is coming let's get started. Define a separate app called something clever like "autocomplete". It will contain an admin.py file (abuse of naming: we will be subclassing classes in django.contrib.admin rather than defining new admin classes like in a regular app admin.py file) and a widget.py file where a custom widget is defined.</p>
<p>Starting with widget.py. The ForeignKeyAutocompleteInput is based the ForeignKeySearchInput in the source of <a href="http://code.google.com/p/django-autocomplete/source/browse/trunk/widgets.py">django-autocomplete</a>. The major difference is I hide the original select widget with Javascript instead of using a hidden widget.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">django</span> <span class="kn">import</span> <span class="n">forms</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">django.utils.safestring</span> <span class="kn">import</span> <span class="n">mark_safe</span>
<span class="k">class</span> <span class="nc">ForeignKeyAutocompleteInput</span><span class="p">(</span><span class="n">forms</span><span class="o">.</span><span class="n">widgets</span><span class="o">.</span><span class="n">Select</span><span class="p">):</span>
<span class="k">class</span> <span class="nc">Media</span><span class="p">:</span>
<span class="n">css</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'all'</span><span class="p">:</span> <span class="p">(</span><span class="s">'</span><span class="si">%s</span><span class="s">/css/jquery.autocomplete.css'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,)</span>
<span class="p">}</span>
<span class="n">js</span> <span class="o">=</span> <span class="p">(</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/jquery.js'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/jquery.autocomplete.js'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/autocomplete.popup.js '</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">text_field_value</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">get_related_field</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_default_manager</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="o">**</span><span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="n">value</span><span class="p">})</span>
<span class="k">return</span> <span class="nb">unicode</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rel</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> rel - the relation for the foreign key.</span>
<span class="sd"> """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rel</span> <span class="o">=</span> <span class="n">rel</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ForeignKeyAutocompleteInput</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">attrs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">attrs</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">attrs</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">rendered</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">ForeignKeyAutocompleteInput</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">attrs</span><span class="p">)</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span>
<span class="n">text_field_value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">text_field_value</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">text_field_value</span> <span class="o">=</span> <span class="s">u''</span>
<span class="k">return</span> <span class="n">rendered</span> <span class="o">+</span> <span class="n">mark_safe</span><span class="p">(</span><span class="s">u'''</span>
<span class="s">#</span>
<span class="s"># Insert javascript wrapped in HTML as listed below here</span>
<span class="s">#</span>
<span class="s"> '''</span><span class="p">)</span> <span class="o">%</span> <span class="p">{</span>
<span class="s">'MEDIA_URL'</span><span class="p">:</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'model_name'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">module_name</span><span class="p">,</span>
<span class="s">'app_label'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">app_label</span><span class="p">,</span>
<span class="s">'text_field_value'</span><span class="p">:</span> <span class="n">text_field_value</span><span class="p">,</span>
<span class="s">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s">'value'</span><span class="p">:</span> <span class="n">value</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<p>Let us focus on the key parts one at a time. First we define an internal class called Media which defines the external resources required to render.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Media</span><span class="p">:</span>
<span class="n">css</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'all'</span><span class="p">:</span> <span class="p">(</span><span class="s">'</span><span class="si">%s</span><span class="s">css/jquery.autocomplete.css'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,)</span>
<span class="p">}</span>
<span class="n">js</span> <span class="o">=</span> <span class="p">(</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/jquery.js'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/jquery.autocomplete.js'</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'</span><span class="si">%s</span><span class="s">/js/autocomplete.popup.js '</span> <span class="o">%</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span>
<span class="p">)</span>
</pre></div>
<p>We require jQuery and jquery.autocomplete which provides the jquery.autocomplete.css. We will need to write a bit of Javascript which I called autocomplete.popup.js to support adding new values.</p>
<p>Next we write some boiler plate the widget class needs to work.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">text_field_value</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="n">key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">get_related_field</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_default_manager</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="o">**</span><span class="p">{</span><span class="n">key</span><span class="p">:</span> <span class="n">value</span><span class="p">})</span>
<span class="k">return</span> <span class="nb">unicode</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rel</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> rel - the relation for the foreign key.</span>
<span class="sd"> """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rel</span> <span class="o">=</span> <span class="n">rel</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ForeignKeyAutocompleteInput</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">attrs</span><span class="p">)</span>
</pre></div>
<p>The interesting bit is render.</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">attrs</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">attrs</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">attrs</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">rendered</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">ForeignKeyAutocompleteInput</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">attrs</span><span class="p">)</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span>
<span class="n">text_field_value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">text_field_value</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">text_field_value</span> <span class="o">=</span> <span class="s">u''</span>
<span class="k">return</span> <span class="n">rendered</span> <span class="o">+</span> <span class="n">mark_safe</span><span class="p">(</span><span class="s">u'''</span>
</pre></div>
<p>This is followed by a huge Javascript filled string which is the real meat of what we are doing here. We have rendered the select normally and in the very last line we append a bunch of additional HTML which includes the autocomplete widget and the Javascript to create the array of possible values and pass it into an .autocomplete function.</p>
<div class="highlight"><pre><span class="o"><</span><span class="nx">input</span> <span class="nx">type</span><span class="o">=</span><span class="s2">"text"</span> <span class="nx">id</span><span class="o">=</span><span class="s2">"lookup_%(name)s"</span> <span class="nx">value</span><span class="o">=</span><span class="s2">"%(text_field_value)s"</span> <span class="nx">size</span><span class="o">=</span><span class="s2">"40"</span> <span class="nx">style</span><span class="o">=</span><span class="s2">"display: none;"</span><span class="o">/></span>
<span class="o"><</span><span class="nx">script</span> <span class="nx">type</span><span class="o">=</span><span class="s2">"text/javascript"</span><span class="o">></span>
<span class="nx">$</span><span class="p">(</span><span class="nb">document</span><span class="p">).</span><span class="nx">ready</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span>
<span class="c1">// Javascript is required to show the autocomplete field and hide the select field.</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#id_%(name)s"</span><span class="p">).</span><span class="nx">hide</span><span class="p">();</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#lookup_%(name)s"</span><span class="p">).</span><span class="nx">show</span><span class="p">();</span>
<span class="kd">function</span> <span class="nx">liFormat_</span><span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s</span> <span class="p">(</span><span class="nx">row</span><span class="p">,</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">num</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="p">;</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_data</span> <span class="o">=</span> <span class="nb">Array</span><span class="p">();</span>
<span class="kd">var</span> <span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_id_map</span> <span class="o">=</span> <span class="p">{};</span>
<span class="kd">function</span> <span class="nx">load_autocomplete_data_from_select</span><span class="p">()</span> <span class="p">{</span>
<span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_data</span> <span class="o">=</span> <span class="nb">Array</span><span class="p">();</span>
<span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_id_map</span> <span class="o">=</span> <span class="p">{};</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#id_%(name)s option"</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span> <span class="p">{</span>
<span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_data</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">html</span><span class="p">());</span>
<span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_id_map</span><span class="p">[</span><span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">html</span><span class="p">()]</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">val</span><span class="p">();</span>
<span class="p">})</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#lookup_%(name)s"</span><span class="p">).</span><span class="nx">autocomplete</span><span class="p">(</span><span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_data</span><span class="p">,</span> <span class="p">{</span>
<span class="nx">delay</span><span class="o">:</span><span class="mi">10</span><span class="p">,</span>
<span class="nx">minChars</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span>
<span class="nx">matchSubset</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span>
<span class="nx">autoFill</span><span class="o">:</span><span class="kc">false</span><span class="p">,</span>
<span class="nx">matchContains</span><span class="o">:</span><span class="mi">1</span><span class="p">,</span>
<span class="nx">cacheLength</span><span class="o">:</span><span class="mi">10</span><span class="p">,</span>
<span class="nx">selectFirst</span><span class="o">:</span><span class="kc">true</span><span class="p">,</span>
<span class="nx">formatItem</span><span class="o">:</span><span class="nx">liFormat_</span><span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s</span><span class="p">,</span>
<span class="nx">maxItemsToShow</span><span class="o">:</span><span class="mi">10</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="nx">load_autocomplete_data_from_select</span><span class="p">();</span> <span class="c1">// Inital load</span>
<span class="c1">// Changing the autocomplete field needs to change the hidden select field</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#lookup_%(name)s"</span><span class="p">).</span><span class="nx">change</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">new_value</span> <span class="o">=</span> <span class="o">%</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="nx">s_id_map</span><span class="p">[</span><span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">val</span><span class="p">()];</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">new_value</span> <span class="o">==</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">new_value</span> <span class="o">=</span> <span class="s2">""</span>
<span class="p">}</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#id_%(name)s"</span><span class="p">).</span><span class="nx">val</span><span class="p">(</span><span class="nx">new_value</span><span class="p">);</span>
<span class="p">})</span>
<span class="c1">// It is possible to "change" the autocomplete text field and have the change</span>
<span class="c1">// event not happen. This double checks right before we submit.</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"form"</span><span class="p">).</span><span class="nx">submit</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#lookup_%(name)s"</span><span class="p">).</span><span class="nx">change</span><span class="p">();</span> <span class="c1">// Just to make sure</span>
<span class="p">})</span>
<span class="c1">// When the add feature is used, it only knows how to change the select field</span>
<span class="c1">// so the auto complete field needs to be updated too.</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#id_%(name)s"</span><span class="p">).</span><span class="nx">change</span><span class="p">(</span><span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#lookup_%(name)s"</span><span class="p">).</span><span class="nx">val</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">find</span><span class="p">(</span><span class="s2">"option:selected"</span><span class="p">).</span><span class="nx">html</span><span class="p">());</span>
<span class="nx">load_autocomplete_data_from_select</span><span class="p">();</span> <span class="c1">// Could be a new value from an add</span>
<span class="p">})</span>
<span class="p">});</span>
<span class="o"><</span><span class="err">/script></span>
</pre></div>
<p>We pass some values into the Javascript block allowing us to do some % substitution. For example the name of the field will be unique on the page so we can use for ids.</p>
<div class="highlight"><pre><span class="p">)</span> <span class="o">%</span> <span class="p">{</span>
<span class="s">'MEDIA_URL'</span><span class="p">:</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEDIA_URL</span><span class="p">,</span>
<span class="s">'model_name'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">module_name</span><span class="p">,</span>
<span class="s">'app_label'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">app_label</span><span class="p">,</span>
<span class="s">'text_field_value'</span><span class="p">:</span> <span class="n">text_field_value</span><span class="p">,</span>
<span class="s">'name'</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
<span class="s">'value'</span><span class="p">:</span> <span class="n">value</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<h4>Admin Integration</h4>
<p>The widget is handy, and could probably be used alone pretty effectively. But to support adding new values, and to make it much easier to use, some deep admin integration is handy. This approach might not be extremely future-proof, but I have tried to cut down on the zany-ness.</p>
<p>admin.py is also based on <a href="http://code.google.com/p/django-autocomplete/source/browse/trunk/widgets.py">django-autocomplete</a>. The big change is splitting up the AdminModel subclasses into a target and main variants.</p>
<p>Let's look at it in four parts. First import a bunch of stuff. We are going deep into Django so we will need a lot of utils functions.</p>
<div class="highlight"><pre><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">django.contrib</span> <span class="kn">import</span> <span class="n">admin</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">django.utils.safestring</span> <span class="kn">import</span> <span class="n">mark_safe</span>
<span class="kn">from</span> <span class="nn">django.utils.translation</span> <span class="kn">import</span> <span class="n">ugettext</span> <span class="k">as</span> <span class="n">_</span>
<span class="kn">from</span> <span class="nn">django.utils.html</span> <span class="kn">import</span> <span class="n">escape</span>
<span class="kn">from</span> <span class="nn">django.utils.encoding</span> <span class="kn">import</span> <span class="n">force_unicode</span>
<span class="kn">from</span> <span class="nn">django.contrib.auth.models</span> <span class="kn">import</span> <span class="n">Message</span>
<span class="kn">from</span> <span class="nn">django.http</span> <span class="kn">import</span> <span class="n">HttpResponse</span><span class="p">,</span> <span class="n">HttpResponseNotFound</span><span class="p">,</span> <span class="n">HttpResponseRedirect</span>
<span class="kn">from</span> <span class="nn">widgets</span> <span class="kn">import</span> <span class="n">ForeignKeyAutocompleteInput</span>
</pre></div>
<p>Moving quickly on we have the part which makes</p>
<div class="highlight"><pre><span class="n">autocomplete_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s">'foreign'</span><span class="p">,)</span>
</pre></div>
<p>work.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">AutocompleteModelAdmin</span><span class="p">(</span><span class="n">admin</span><span class="o">.</span><span class="n">ModelAdmin</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">formfield_for_dbfield</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">db_field</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c"># For ForeignKey use a special Autocomplete widget.</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">db_field</span><span class="p">,</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s">"autocomplete_fields"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">db_field</span><span class="o">.</span><span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">autocomplete_fields</span><span class="p">:</span>
<span class="n">kwargs</span><span class="p">[</span><span class="s">'widget'</span><span class="p">]</span> <span class="o">=</span> <span class="n">ForeignKeyAutocompleteInput</span><span class="p">(</span><span class="n">db_field</span><span class="o">.</span><span class="n">rel</span><span class="p">)</span>
<span class="c"># extra HTML to the end of the rendered output.</span>
<span class="k">if</span> <span class="s">'request'</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s">'request'</span><span class="p">)</span>
<span class="n">formfield</span> <span class="o">=</span> <span class="n">db_field</span><span class="o">.</span><span class="n">formfield</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c"># Don't wrap raw_id fields. Their add function is in the popup window.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">db_field</span><span class="o">.</span><span class="n">name</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">raw_id_fields</span><span class="p">:</span>
<span class="c"># formfield can be None if it came from a OneToOneField with</span>
<span class="c"># parent_link=True</span>
<span class="k">if</span> <span class="n">formfield</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">formfield</span><span class="o">.</span><span class="n">widget</span> <span class="o">=</span> <span class="n">AutocompleteWidgetWrapper</span><span class="p">(</span><span class="n">formfield</span><span class="o">.</span><span class="n">widget</span><span class="p">,</span> <span class="n">db_field</span><span class="o">.</span><span class="n">rel</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">admin_site</span><span class="p">)</span>
<span class="k">return</span> <span class="n">formfield</span>
<span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">AutocompleteModelAdmin</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">formfield_for_dbfield</span><span class="p">(</span><span class="n">db_field</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</pre></div>
<p>One class overriding one function. If the field is a ForeignKey and is named in autocomplete_fields, then we use our widget from above, and do a bit of funky wrapping to make the "+" appear. More on the AutocompleteWidgetWrapper later.</p>
<p>I have separated out the next class. We use it on models we are targeting with an autocomplete field like this.</p>
<div class="highlight"><pre><span class="n">admin</span><span class="o">.</span><span class="n">site</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">Foreign</span><span class="p">,</span> <span class="n">AutocompleteTargetModelAdmin</span><span class="p">)</span>
</pre></div>
<p>The class looks like this:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">AutocompleteTargetModelAdmin</span><span class="p">(</span><span class="n">admin</span><span class="o">.</span><span class="n">ModelAdmin</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">response_add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">post_url_continue</span><span class="o">=</span><span class="s">'../</span><span class="si">%s</span><span class="s">/'</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Determines the HttpResponse for the add_view stage.</span>
<span class="sd"> """</span>
<span class="n">opts</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="n">_meta</span>
<span class="n">pk_value</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="n">_get_pk_val</span><span class="p">()</span>
<span class="n">msg</span> <span class="o">=</span> <span class="n">_</span><span class="p">(</span><span class="s">'The </span><span class="si">%(name)s</span><span class="s"> "</span><span class="si">%(obj)s</span><span class="s">" was added successfully.'</span><span class="p">)</span> <span class="o">%</span> <span class="p">{</span><span class="s">'name'</span><span class="p">:</span> <span class="n">force_unicode</span><span class="p">(</span><span class="n">opts</span><span class="o">.</span><span class="n">verbose_name</span><span class="p">),</span> <span class="s">'obj'</span><span class="p">:</span> <span class="n">force_unicode</span><span class="p">(</span><span class="n">obj</span><span class="p">)}</span>
<span class="c"># Here, we distinguish between different save types by checking for</span>
<span class="c"># the presence of keys in request.POST.</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"_continue"</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message_user</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">msg</span> <span class="o">+</span> <span class="s">' '</span> <span class="o">+</span> <span class="n">_</span><span class="p">(</span><span class="s">"You may edit it again below."</span><span class="p">))</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"_popup"</span><span class="p">):</span>
<span class="n">post_url_continue</span> <span class="o">+=</span> <span class="s">"?_popup=</span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'_popup'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">HttpResponseRedirect</span><span class="p">(</span><span class="n">post_url_continue</span> <span class="o">%</span> <span class="n">pk_value</span><span class="p">)</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"_popup"</span><span class="p">):</span>
<span class="c">#htturn response to Autocomplete PopUp</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"_popup"</span><span class="p">):</span>
<span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="s">'<script type="text/javascript">opener.dismissAutocompletePopup(window, "</span><span class="si">%s</span><span class="s">", "</span><span class="si">%s</span><span class="s">");</script>'</span> <span class="o">%</span> <span class="p">(</span><span class="n">escape</span><span class="p">(</span><span class="n">pk_value</span><span class="p">),</span> <span class="n">escape</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span>
<span class="k">elif</span> <span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s">"_addanother"</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message_user</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">msg</span> <span class="o">+</span> <span class="s">' '</span> <span class="o">+</span> <span class="p">(</span><span class="n">_</span><span class="p">(</span><span class="s">"You may add another </span><span class="si">%s</span><span class="s"> below."</span><span class="p">)</span> <span class="o">%</span> <span class="n">force_unicode</span><span class="p">(</span><span class="n">opts</span><span class="o">.</span><span class="n">verbose_name</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">HttpResponseRedirect</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">path</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message_user</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>
<span class="c"># Figure out where to redirect. If the user has change permission,</span>
<span class="c"># redirect to the change-list page for this object. Otherwise,</span>
<span class="c"># redirect to the admin index.</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">has_change_permission</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="bp">None</span><span class="p">):</span>
<span class="n">post_url</span> <span class="o">=</span> <span class="s">'../'</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">post_url</span> <span class="o">=</span> <span class="s">'../../../'</span>
<span class="k">return</span> <span class="n">HttpResponseRedirect</span><span class="p">(</span><span class="n">post_url</span><span class="p">)</span>
</pre></div>
<p>One class, one complete method copied except for one key line:</p>
<div class="highlight"><pre><span class="k">return</span> <span class="n">HttpResponse</span><span class="p">(</span><span class="s">'<script type="text/javascript">opener.dismissAutocompletePopup(window, "</span><span class="si">%s</span><span class="s">", "</span><span class="si">%s</span><span class="s">");</script>'</span> <span class="o">%</span> <span class="p">(</span><span class="n">escape</span><span class="p">(</span><span class="n">pk_value</span><span class="p">),</span> <span class="n">escape</span><span class="p">(</span><span class="n">obj</span><span class="p">)))</span>
</pre></div>
<p>Here "opener" is the window a user spawns by hitting the green "+". This change in the response lets us call our slightly different popup handling code which enables our autocomplete field to update with the new value after the hidden select drop-down is updated by the default Django Admin code.</p>
<p>There might be a better way, but I haven't been able to find it yet. Suggestions are welcome. The key problem is, though the select will change by default, it won't produce a change event, so there is no way to update the autocomplete field.</p>
<p>Finally the wrapper, which is all thanks to the author of django-autocomplete.</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">AutocompleteWidgetWrapper</span><span class="p">(</span><span class="n">admin</span><span class="o">.</span><span class="n">widgets</span><span class="o">.</span><span class="n">RelatedFieldWidgetWrapper</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">rel_to</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rel</span><span class="o">.</span><span class="n">to</span>
<span class="n">related_url</span> <span class="o">=</span> <span class="s">'../../../</span><span class="si">%s</span><span class="s">/</span><span class="si">%s</span><span class="s">/'</span> <span class="o">%</span> <span class="p">(</span><span class="n">rel_to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">app_label</span><span class="p">,</span> <span class="n">rel_to</span><span class="o">.</span><span class="n">_meta</span><span class="o">.</span><span class="n">object_name</span><span class="o">.</span><span class="n">lower</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">widget</span><span class="o">.</span><span class="n">choices</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">choices</span>
<span class="n">output</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">widget</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)]</span>
<span class="k">if</span> <span class="n">rel_to</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">admin_site</span><span class="o">.</span><span class="n">_registry</span><span class="p">:</span> <span class="c"># If the related object has an admin interface:</span>
<span class="c"># TODO: "id_" is hard-coded here. This should instead use the correct</span>
<span class="c"># API to determine the ID dynamically.</span>
<span class="n">output</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">u'<a href="</span><span class="si">%s</span><span class="s">add/" class="add-another" id="add_id_</span><span class="si">%s</span><span class="s">" onclick="return showAutocompletePopup(this);"> '</span> <span class="o">%</span> \
<span class="p">(</span><span class="n">related_url</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
<span class="n">output</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">u'<img src="</span><span class="si">%s</span><span class="s">img/admin/icon_addlink.gif" width="10" height="10" alt="</span><span class="si">%s</span><span class="s">" /></a>'</span> <span class="o">%</span> <span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">ADMIN_MEDIA_PREFIX</span><span class="p">,</span> <span class="n">_</span><span class="p">(</span><span class="s">'Add Another'</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">mark_safe</span><span class="p">(</span><span class="s">u''</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">output</span><span class="p">))</span>
</pre></div>
<p>That is basically it except for a bit of Javascript that's needed to stitch together the popup logic which I placed in a file called autocomplete.popup.js</p>
<div class="highlight"><pre><span class="kd">function</span> <span class="nx">showAutocompletePopup</span><span class="p">(</span><span class="nx">triggeringLink</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">name</span> <span class="o">=</span> <span class="nx">triggeringLink</span><span class="p">.</span><span class="nx">id</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/^add_/</span><span class="p">,</span> <span class="s1">''</span><span class="p">);</span>
<span class="nx">name</span> <span class="o">=</span> <span class="nx">id_to_windowname</span><span class="p">(</span><span class="nx">name</span><span class="p">);</span>
<span class="nx">href</span> <span class="o">=</span> <span class="nx">triggeringLink</span><span class="p">.</span><span class="nx">href</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">href</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="s1">'?'</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">href</span> <span class="o">+=</span> <span class="s1">'?_popup=2'</span><span class="p">;</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">href</span> <span class="o">+=</span> <span class="s1">'&_popup=2'</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">win</span> <span class="o">=</span> <span class="nb">window</span><span class="p">.</span><span class="nx">open</span><span class="p">(</span><span class="nx">href</span><span class="p">,</span> <span class="nx">name</span><span class="p">,</span> <span class="s1">'height=500,width=800,resizable=yes,scrollbars=yes'</span><span class="p">);</span>
<span class="nx">win</span><span class="p">.</span><span class="nx">focus</span><span class="p">();</span>
<span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">dismissAutocompletePopup</span><span class="p">(</span><span class="nx">win</span><span class="p">,</span> <span class="nx">newId</span><span class="p">,</span> <span class="nx">newRepr</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">newId</span> <span class="o">=</span> <span class="nx">html_unescape</span><span class="p">(</span><span class="nx">newId</span><span class="p">);</span>
<span class="nx">newRepr</span> <span class="o">=</span> <span class="nx">html_unescape</span><span class="p">(</span><span class="nx">newRepr</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">name</span> <span class="o">=</span> <span class="nx">windowname_to_id</span><span class="p">(</span><span class="nx">win</span><span class="p">.</span><span class="nx">name</span><span class="p">);</span>
<span class="nx">dismissAddAnotherPopup</span><span class="p">(</span><span class="nx">win</span><span class="p">,</span> <span class="nx">newId</span><span class="p">,</span> <span class="nx">newRepr</span><span class="p">);</span>
<span class="nx">$</span><span class="p">(</span><span class="s2">"#"</span> <span class="o">+</span> <span class="nx">name</span><span class="p">).</span><span class="nx">change</span><span class="p">();</span>
<span class="p">}</span>
</pre></div>
<p>Not super simple, but an effective way to get a friendlier admin user interface experience.</p>
<p>If you find this useful please let me know. If there is enough interest I will look into cleaning out the cruft and making an installable package out of this code.</p>Showing Current Values for FileFields in Django2010-08-09T16:47:00-04:00Albert O'Connortag:albertoconnor.ca,2010-08-09:showing-current-values-filefields.html<p>I am working on a dynamic form which contained multiple FileFields which could have initial values. The FileInput widget doesn't show initial values since that could expose too much about the servers underlying file system. Despite this I wanted to show the file name as user feedback.</p>
<p>If the form was static, I could pass current file names in through the context and just be very explicit with my template code. Since the form was dynamic the template code would have to iterate though the fields, and there is no easy way to look up unrelated data at the same time.</p>
<p>To learn some nifty tricks on how to create dynamic forms check out <a href="http://www.b-list.org/weblog/2008/nov/09/dynamic-forms/" target="_blank"><i>So you want a dynamic form</i></a> on the <a href="http://www.b-list.org" target="_blank">b-list</a> blog.</p>
<p>Meanwhile I need a solution to pass in the initial file name data with the form. The built in initial dictionary didn't really work, and the field objects initial data member was empty. I had to look outside the form. Beyond the form to the truth: there is not form!</p>
<p><strong>There Is No Form</strong></p>
<p>A pearl of wisdom in my grasp, the form object did not matter, only the field objects were required to render a form. (Assuming you don't need form-wide errors ...)</p>
<p>In the view I passed in 2-uples of the initial file names of the files and the field objects. I did not need to pass the form object into the template at all:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">view</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">posting</span> <span class="o">=</span> <span class="n">get_object_or_404</span><span class="p">(</span><span class="n">Posting</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="nb">id</span><span class="p">)</span> <span class="c"># Get the thing the dynamic form depends on</span>
<span class="n">file_store</span> <span class="o">=</span> <span class="n">get_object_or_404</span><span class="p">(</span><span class="n">FileStore</span><span class="p">,</span> <span class="n">pk</span><span class="o">=</span><span class="nb">id</span><span class="p">)</span> <span class="c"># Something like this</span>
<span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s">"POST"</span><span class="p">:</span>
<span class="n">file_form</span> <span class="o">=</span> <span class="n">make_file_application_form</span><span class="p">(</span><span class="n">posting</span><span class="p">)(</span><span class="n">request</span><span class="o">.</span><span class="n">POST</span><span class="p">,</span> <span class="n">request</span><span class="o">.</span><span class="n">FILES</span><span class="p">)</span>
<span class="n">file_fields_and_names</span> <span class="o">=</span> <span class="p">[(</span><span class="n">file_form</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">file_store</span><span class="p">,</span> <span class="n">key</span><span class="p">))</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">file_form</span><span class="o">.</span><span class="n">fields</span><span class="p">]</span>
<span class="n">file_fields_and_names</span> <span class="o">=</span> <span class="p">[(</span><span class="n">field</span><span class="p">,</span> <span class="n">file_name_from_file_obj</span><span class="p">(</span><span class="nb">file</span><span class="p">))</span> <span class="k">for</span> <span class="n">field</span><span class="p">,</span><span class="nb">file</span> <span class="ow">in</span> <span class="n">file_fields_and_names</span><span class="p">]</span>
<span class="k">if</span> <span class="n">file_form</span><span class="o">.</span><span class="n">is_valid</span><span class="p">():</span>
<span class="n">file_data</span> <span class="o">=</span> <span class="n">file_form</span><span class="o">.</span><span class="n">cleaned_data</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">file_data</span><span class="p">:</span>
<span class="k">if</span> <span class="n">file_data</span><span class="p">[</span><span class="n">key</span><span class="p">]:</span>
<span class="nb">setattr</span><span class="p">(</span><span class="n">file_store</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">file_data</span><span class="p">[</span><span class="n">key</span><span class="p">])</span>
<span class="n">file_store</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
<span class="k">return</span> <span class="n">redirect</span><span class="p">(</span><span class="s">"view"</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">file_form</span> <span class="o">=</span> <span class="n">make_file_application_form</span><span class="p">(</span><span class="n">posting</span><span class="p">)()</span>
<span class="n">file_fields_and_names</span> <span class="o">=</span> <span class="p">[(</span><span class="n">file_form</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">file_store</span><span class="p">,</span> <span class="n">key</span><span class="p">))</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">file_form</span><span class="o">.</span><span class="n">fields</span><span class="p">]</span>
<span class="n">file_fields_and_names</span> <span class="o">=</span> <span class="p">[(</span><span class="n">field</span><span class="p">,</span> <span class="n">file_name_from_file_obj</span><span class="p">(</span><span class="nb">file</span><span class="p">))</span> <span class="k">for</span> <span class="n">field</span><span class="p">,</span><span class="nb">file</span> <span class="ow">in</span> <span class="n">file_fields_and_names</span><span class="p">]</span>
<span class="k">return</span> <span class="n">render_to_response</span><span class="p">(</span><span class="n">template</span><span class="p">,</span>
<span class="nb">dict</span><span class="p">(</span><span class="n">file_fields_and_names</span><span class="o">=</span><span class="n">file_fields_and_names</span><span class="p">,</span> <span class="o">...</span><span class="p">),</span>
<span class="n">context_instance</span><span class="o">=</span><span class="n">RequestContext</span><span class="p">(</span><span class="n">request</span><span class="p">))</span>
</pre></div>
<p>So if you need to, you can get creative with how you pass your fields into your template.</p>