<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[really tenacious guy]]></title>
  <link href="http://rtg.in.ua/atom.xml" rel="self"/>
  <link href="http://rtg.in.ua/"/>
  <updated>2012-05-21T11:46:37+03:00</updated>
  <id>http://rtg.in.ua/</id>
  <author>
    <name><![CDATA[Roman Yepishev]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[World IPv6 Launch]]></title>
    <link href="http://rtg.in.ua/blog/2012/05/world-ipv6-launch/"/>
    <updated>2012-05-21T11:18:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/05/world-ipv6-launch</id>
    <content type="html"><![CDATA[<p><img class="left" src="http://www.worldipv6launch.org/wp-content/themes/ipv6/downloads/World_IPv6_launch_logo_128.png"></p>

<p><strong>What is World IPv6 Launch?</strong></p>

<p>Organized by the Internet Society, <a href="http://www.worldipv6launch.org">World IPv6 Launch</a> on 6 June 2012 is
intended to motivate organizations across the industry – including Internet
service providers (ISPs), hardware makers, and web companies – to prepare for
and permanently enable Internet Protocol version 6 (IPv6) on their products
and services as Internet Protocol version 4 (IPv4) address space runs out.</p>

<p>This web site has been accessible via IPv6 since last year thanks to <a href="http://burst.net">burst.net</a>
link to <a href="http://www.he.net">Hurricane Electric</a>. And now I have configured
<a href="https://www.cloudflare.com">CloudFlare</a> to serve the content of the site
via their CDN network which is also IPv6-enabled.</p>

<p>For home IPv6 connectivity I am using Ukrainian <a href="http://tb.netassist.ua">NetAssist tunnel broker</a>
which has delegated me a /48 block. I am actually using IPv6 in my LAN for
all communications with occasional fallback to IPv6 for some services that
does not support it yet.</p>

<p>So I am prepared for World IPv6 Launch.</p>

<p>Frankly speaking, I will celebrate the World IPv6 Launch day when I notice
the first SSH brute force attack on my servers using IPv6 link.</p>

<p>Yandex-Import-Id:
61907928.366051546.1337588139.6716b286801600fc9c318745a3c0d97c</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[OpenStack Swift and Keystone. Setting up Cloud Storage]]></title>
    <link href="http://rtg.in.ua/blog/2012/05/openstack-swift-and-keystone-setting-up-cloud-storage/"/>
    <updated>2012-05-17T09:24:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/05/openstack-swift-and-keystone-setting-up-cloud-storage</id>
    <content type="html"><![CDATA[<p><img class="left" src="http://rtg.in.ua/assets/ubuntuone/70CRFsGEPOx8umfH2kXG6V/openstack.png">
Having played a lot with <a href="http://hpcloud.com/">HP Cloud</a> and Canonical OpenStack installation,
I decided to install OpenStack on my own home server.
In order to gain better understanding about all the processes
involved I decided not to use <a href="http://devstack.org/">devstack</a> script and follow the installation
guides with the packages already available in Ubuntu Server 12.04 LTS (Precise
Pangolin).</p>

<p>Surprisingly, setting up Keystone, Nova Compute, and Glance went really well
with almost no effort from my side and I was even able to launch an instance
of Ubuntu Server 12.04 running in nested KVM, which in turn was running on
Compute virtual instance in KVM. So I had 2 levels of virtualization and it
was still working quite well.</p>

<p>Then I decided to make Swift (Object Store) authenticating
against <a href="http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-swift-with-s3-emulation-to-use-keystone.html">Keystone</a> (instead of built-in swauth).
And this took a bit longer than expected.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/5ORxFKz5XiKMGefvqVANBE/storage-proxy.png"></p>

<!--More-->


<p>I was using the following documents to prepare the installation:</p>

<ul>
<li><a href="http://www.hastexo.com/resources/docs/installing-openstack-essex-20121-ubuntu-1204-precise-pangolin">Installing OpenStack Essex (2012.1) on Ubuntu 12.04</a></li>
<li><a href="http://swift.openstack.org/howto_installmultinode.html">Multiple Server Swift Installation (Ubuntu)</a></li>
</ul>


<p>This post assumes you are setting up a local private installation of
OpenStack without HTTPS, which is fine when it is on a private LAN, but is a
very bad decision for a public network. The passwords are traveling in
clear-text. And by clear text I mean this:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>POST /v2.0/tokens HTTP/1.1
</span><span class='line'>Host: compute.example.com:35357
</span><span class='line'>Content-Length: 132
</span><span class='line'>content-type: application/json
</span><span class='line'>accept-encoding: gzip, deflate
</span><span class='line'>accept: application/json
</span><span class='line'>user-agent: python-novaclient
</span><span class='line'>
</span><span class='line'>{"auth": {"tenantName": "you@example.com-tenant",
</span><span class='line'>          "passwordCredentials": {"username": "you@example.com",
</span><span class='line'>                                  "password": "testing123"}}}</span></code></pre></td></tr></table></div></figure>


<h1>Keystone</h1>

<p>The instructions at <a href="http://www.hastexo.com/resources/docs/installing-openstack-essex-20121-ubuntu-1204-precise-pangolin">hastexo</a> state they are missing the keystone
integration with swift. So when it says you need to use a templated catalog,
that catalog by default does not contain the swift endpoint.
Here&#8217;s what you will need to append to the default:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>catalog.RegionOne.object-store.publicURL = http://proxy-server.example.com:8080/v1.0/AUTH_$(tenant_id)s
</span><span class='line'>catalog.RegionOne.object-store.name = Swift Service</span></code></pre></td></tr></table></div></figure>


<p>See paragraph about AUTH_ prefix <a href="#auth-prefix">below</a>.</p>

<blockquote><p>Q: Where is &#8220;object-store&#8221; service defined? <br>
A: Keystone does not actually care about the names and URLs of these endpoints,
so the services themselves use the information. For example, swift
application queries the endpoints and searches for <code>object-store</code> type, then
directs all storage requests to the corresponding publicURL.
All other service types are listed in default_catalog.templates.</p>

<p>Q: What is <code>tenant_id</code>? <br>
A: Keystone knows several variables it can dynamically substitute with
values based on the client - <code>tenant_id</code> and <code>user_id</code>.
These are defined in <a href="https://github.com/openstack/keystone/blob/master/keystone/catalog/backends/templated.py#L115">TemplatedCatalog</a>.</p></blockquote>

<p>If you set up <code>cert_key</code> and <code>key_file</code> as the <a href="http://swift.openstack.org/howto_installmultinode.html">multinode howto</a> says, your proxy-server
will expect HTTPS (i.e. HTTP over SSL) connection. So adjust the schema accordingly.</p>

<h2>AUTH_ prefix <a id="auth-prefix"></a></h2>

<p>By default Keystone&#8217;s swift_auth middleware is
using <code>reseller_prefix = AUTH_</code>.
So if you set up tenant with ID <code>abc</code> and set publicURL without <code>AUTH_</code>
prefix, you will get a very unhelpful message in syslog of proxy-server:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>tenant mismatch: abc != abc</span></code></pre></td></tr></table></div></figure>


<p>The real-world log line will look like this:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>proxy-server tenant mismatch: ⤶ 
</span><span class='line'>    9830818c942e40bbabaad02454524597 != 9830818c942e40bbabaad02454524597 ⤶
</span><span class='line'>    (txn: txfd89a4d692394b3589c7d31c9f78ebb2) (client_ip: 192.168.1.110)</span></code></pre></td></tr></table></div></figure>


<p>The message does not show the real compared value and, in fact, should print &#8220;abc != AUTH_abc&#8221;.
You have two options:</p>

<ul>
<li><p>If you are setting up a single authentication server and the backend nodes
are going to serve only one set of accounts, then you can unset AUTH_
prefix in middleware configuration on proxy-server and in endpoints catalog.</p></li>
<li><p>If you have a massive deployment and it is possible to have resellers for
the storage you provide, you set up a different reseller_prefix on
each proxy-server that is going to serve a different reseller and update
keystone catalog and middleware configuration on proxy-server.</p></li>
</ul>


<p>So, to simplify our deployment, we are dropping AUTH_ prefix completely.
Update the keystone service template:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>catalog.RegionOne.object-store.publicURL = http://proxy-server.example.net:8080/v1.0/$(tenant_id)s</span></code></pre></td></tr></table></div></figure>


<p><a id="operator-roles-def"></a>Update the <code>reseller_prefix</code> in <code>swift.conf</code> on
proxy-server itself (you don&#8217;t need to change anything on storage nodes):</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[filter:keystone]
</span><span class='line'>paste.filter_factory = keystone.middleware.swift_auth:filter_factory
</span><span class='line'>reseller_prefix = 
</span><span class='line'>operator_roles = admin, swiftoperator</span></code></pre></td></tr></table></div></figure>


<p>I will get to <code>operator_roles</code> setting <a href="#operator-roles">later</a>.</p>

<h1>Proxy Server</h1>

<p><strong>Important</strong>: By default proxy-server will try to connect to keystone using HTTPS.
Earlier versions would hang indefinitely but since the fix in <a href="https://bugs.launchpad.net/bugs/891687" title="Launchpad Bug #891687: Auth Token hangs if calling Keystone on wrong protocol" class="launchpad-bug">891687</a> the response from swift will be:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Account HEAD failed: http://proxy-server.example.com:8080/v1.0/a...3 503 Service Unavailable</span></code></pre></td></tr></table></div></figure>


<p>If you are following the tutorials, keystone will not listen for HTTPS so you will need to modify <code>proxy-server.conf</code>:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[filter:authtoken]
</span><span class='line'>...
</span><span class='line'>auth_protocol = http</span></code></pre></td></tr></table></div></figure>


<h2>Zones</h2>

<p>This is not actually about keystone but it&#8217;s nice to know.</p>

<p>First of all, if you feel lost about the concept of a Ring and
<a href="http://swift.openstack.org/overview_ring.html">OpenStack: The Rings</a> is not
helping, you should definitely read
<a href="http://julien.danjou.info/blog/2012/openstack-swift-consistency-analysis">this analysis by Julien Danjou</a> -
you will find out slightly more than you need for the simple setup, the article
tells you about the bottlenecks too, which you must know should you start
a large-scale deployment.</p>

<p>So, a zone can be basically a disk, a server, a rack, or anything that groups
the storage devices. If you have two servers and each holds a
storage zone then PSU failure of one server will not affect the other zone.</p>

<p>For initial testing purposes you can actually start with a single zone,
just set the number of replicas for the ring to be 1:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>swift-ring-builder account.builder create   18 1 1
</span><span class='line'>swift-ring-builder container.builder create 18 1 1
</span><span class='line'>swift-ring-builder object.builder create    18 1 1
</span><span class='line'>#                    partition size --------^  ^
</span><span class='line'>#                       number of replicas ----'</span></code></pre></td></tr></table></div></figure>


<p>Please note that if you want to change the number of replicas later, you will
have to re-create the rings from scratch - <a href="https://bugs.launchpad.net/bugs/958144" title="Launchpad Bug #958144: Can the number of replicas be changeable?" class="launchpad-bug">958144</a>.</p>

<h1><a id="operator-roles"></a>Tenants and Roles</h1>

<p>Remember <code>operator_roles</code> in <code>[filter:keystone]</code>
<a href="#operator-roles-def">earlier</a>? Only the users
granted those roles will be able to access the storage server. If the user does
not have the role listed there, then all the requests will fail with
401 - Unauthorized.</p>

<p><a href="http://docs.openstack.org/trunk/openstack-compute/admin/content/users-and-projects.html">Tenants</a> (earlier called projects) allow grouping the users. In order to use
swift you will need to create a role, a tenant, a user
and assign the role to the user within the tenant:</p>

<blockquote><p>You can also use admin user and role set up in the tutorials above, but
creating these users is quite straightforward.</p></blockquote>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ keystone role-create --name swiftoperator
</span><span class='line'>+----------+----------------------------------+
</span><span class='line'>| Property |              Value               |
</span><span class='line'>+----------+----------------------------------+
</span><span class='line'>| id       | e77542b0b6884ea4ada3ec9eb6b75c6f |
</span><span class='line'>| name     | swiftoperator                    |
</span><span class='line'>+----------+----------------------------------+
</span><span class='line'>
</span><span class='line'>$ keystone user-create --name you@example.com --pass testing123
</span><span class='line'>+----------+-----------------------------------------+
</span><span class='line'>| Property |                                   Value |
</span><span class='line'>+----------+-----------------------------------------+
</span><span class='line'>| email    | None                                    |
</span><span class='line'>| enabled  | True                                    |
</span><span class='line'>| id       | 866be627384044a49b3a172a1cbef74f        |
</span><span class='line'>| name     | you@example.com                         |
</span><span class='line'>| password | $6$rounds=40000$Bl/EL3YgSUqIpSr/$Lc...  |
</span><span class='line'>| tenantId | None                                    |
</span><span class='line'>+----------+-----------------------------------------+
</span><span class='line'>
</span><span class='line'>$ keystone tenant-create --name "you@example.com-tenant"
</span><span class='line'>+-------------+----------------------------------+
</span><span class='line'>|   Property  |              Value               |
</span><span class='line'>+-------------+----------------------------------+
</span><span class='line'>| description | None                             |
</span><span class='line'>| enabled     | True                             |
</span><span class='line'>| id          | aaee4b5b277c40aaad201742db9e20c3 |
</span><span class='line'>| name        | you@example.com-tenant           |
</span><span class='line'>+-------------+----------------------------------+
</span><span class='line'>
</span><span class='line'>$ keystone user-role-add --user 866be627384044a49b3a172a1cbef74f \
</span><span class='line'>        --role e77542b0b6884ea4ada3ec9eb6b75c6f\
</span><span class='line'>        --tenant_id aaee4b5b277c40aaad201742db9e20c3y
</span></code></pre></td></tr></table></div></figure>


<p>The names of the user and tenant can be arbitrary. For example, hpcloud uses
<code>your-email@example.com-default-tenant</code> for tenant name and your email as
username.</p>

<p>Great, now we check whether the user can get auth token:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ nova --os_username you@example.com --os_password testing123 \
</span><span class='line'>       --os_tenant_name=you@example.com-tenant \
</span><span class='line'>       --os_auth_url http://compute.example.com:35357/v2.0 credentials 
</span><span class='line'>
</span><span class='line'>+------------------+---------------------------------------------------------------------------+
</span><span class='line'>| User Credentials |                                   Value                                   |
</span><span class='line'>+------------------+---------------------------------------------------------------------------+
</span><span class='line'>| id               | 866be627384044a49b3a172a1cbef74f                                          |
</span><span class='line'>| name             | you@example.com                                                           |
</span><span class='line'>| roles            | [{u'id': u'e77542b0b6884ea4ada3ec9eb6b75c6f', u'name': u'swiftoperator'}] |
</span><span class='line'>| roles_links      | []                                                                        |
</span><span class='line'>| username         | you@example.com                                                           |
</span><span class='line'>+------------------+---------------------------------------------------------------------------+
</span><span class='line'>...</span></code></pre></td></tr></table></div></figure>


<p>And finally, we use swift cli:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ swift -U you@example.com-tenant:you@example.com -K testing123 -A \
</span><span class='line'>        http://compute.openstack.lappyfamily.net:35357/v2.0 -V 2.0 \
</span><span class='line'>        stat
</span><span class='line'>
</span><span class='line'>   Account: aaee4b5b277c40aaad201742db9e20c3
</span><span class='line'>Containers: 0
</span><span class='line'>   Objects: 0
</span><span class='line'>     Bytes: 0
</span><span class='line'>Accept-Ranges: bytes</span></code></pre></td></tr></table></div></figure>


<p>Congratulations, the Swift/Keystone pairing is set up properly.</p>

<p>The next step is to play with the storage and I&#8217;ve already described
what operations are available in my <a href="http://rtg.in.ua/blog/2012/03/hpcloud-object-storage-speed/">HP Cloud-related post</a>.</p>

<p>Finally, there is a bug in <code>python-webob</code> that causes Swift HEAD request to return the following response:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Content-Type: text/plain
</span><span class='line'>Content-Length: 13</span></code></pre></td></tr></table></div></figure>


<p>This is <a href="https://bugs.launchpad.net/bugs/920197" title="Launchpad Bug #920197: webob last stable version 1.1.1 response header bug" class="launchpad-bug">920197</a> and can be fixed by the patch attached to the bug
report until I or somebody else forwards the patch to Debian maintainer (the
fix is already in the trunk of webob thanks to quick Martin Vidner response).</p>

<h1>S3 Emulation and Keystone</h1>

<p>If you follow the instructions at
<a href="http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-swift-with-s3-emulation-to-use-keystone.html">Configuring Swift with S3 emulation to use Keystone</a>, proxy-server will
not actually be able to talk to Keystone because the /s3tokens URL is not
being handled in the default configuration.</p>

<p>I found the solution in a message by Thanathip Limna at <a href="https://bugs.launchpad.net/bugs/956562" title="Launchpad Bug #956562: Issues running keystone swift with no S3 " class="launchpad-bug">956562</a> -
<code>[filter:s3_extension]</code> config needs to be added along
with adding <code>s3_extension</code> to pipelines of
<code>public_api</code> and <code>admin_api</code> right after <code>ec2_extension</code>:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[filter:s3_extension]
</span><span class='line'>paste.filter_factory = keystone.contrib.s3:S3Extension.factory
</span><span class='line'>
</span><span class='line'>[pipeline:public_api]
</span><span class='line'>pipeline = token_auth admin_token_auth xml_body json_body debug ec2_extension s3_extension public_service
</span><span class='line'>
</span><span class='line'>[pipeline:admin_api]
</span><span class='line'>pipeline = token_auth admin_token_auth xml_body json_body debug ec2_extension s3_extension crud_extension admin_service</span></code></pre></td></tr></table></div></figure>


<p>If you don&#8217;t add this, any requests to Swift with S3 credentials are failing
with Internal Server Error:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>HTTP/1.1 500 Internal Server Error
</span><span class='line'>Content-Type: text/plain
</span><span class='line'>Content-Length: 1119
</span><span class='line'>Date: Sun, 20 May 2012 08:31:12 GMT
</span><span class='line'>Connection: close
</span><span class='line'>
</span><span class='line'>Traceback (most recent call last):
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 336, in
</span><span class='line'>handle_one_response
</span><span class='line'>    result = self.application(self.environ, start_response)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/healthcheck.py",
</span><span class='line'>line 38, in __call__
</span><span class='line'>    return self.app(env, start_response)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/memcache.py",
</span><span class='line'>line 47, in __call__
</span><span class='line'>    return self.app(env, start_response)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/swift3.py",
</span><span class='line'>line 479, in __call__
</span><span class='line'>    res = getattr(controller, req.method)(env, start_response)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/swift/common/middleware/swift3.py",
</span><span class='line'>line 181, in GET
</span><span class='line'>    body_iter = self.app(env, self.do_start_response)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/keystone/middleware/s3_token.py",
</span><span class='line'>line 157, in __call__
</span><span class='line'>    (resp, output) = self._json_request(creds_json)
</span><span class='line'>  File "/usr/lib/python2.7/dist-packages/keystone/middleware/s3_token.py",
</span><span class='line'>line 91, in _json_request
</span><span class='line'>    response.reason))
</span><span class='line'>ServiceError: Keystone reply error: status=404 reason=Not Found</span></code></pre></td></tr></table></div></figure>


<p>If you have modified the <code>reseller_prefix</code> in <code>swift_auth</code> middleware of
proxy-server, you will need to change it in <code>[filter:s3token]</code> too, otherwise
you will get this response from S3 emulation layer</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
</span><span class='line'>&lt;Error&gt;
</span><span class='line'>  &lt;Code&gt;InvalidURI&lt;/Code&gt;
</span><span class='line'>  &lt;Message&gt;Could not parse the specified URI&lt;/Message&gt;
</span><span class='line'>&lt;/Error&gt;</span></code></pre></td></tr></table></div></figure>


<p>And that&#8217;s what will be in proxy-server syslog:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>tenant mismatch: AUTH_abc != abc</span></code></pre></td></tr></table></div></figure>


<p>So now we have Swift S3 emulation working with keystone too.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Чем я пользуюсь]]></title>
    <link href="http://rtg.in.ua/blog/2012/05/stuff-i-am-using/"/>
    <updated>2012-05-13T13:09:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/05/stuff-i-am-using</id>
    <content type="html"><![CDATA[<p>Для работы мне требуется несколько вещей, в этом посте я опишу интернет
провайдера, банк и оборудование, которыми я пользуюсь.</p>

<!-- More -->


<h2>Интернет</h2>

<p>Одним из условий подписания контракта с Canonical было наличие стабильного
подключения к Интернет. После года попыток наладить нормальное подключение с
домашним интернетом Билайна (теперь Киевстар, но VPN костыль у них никуда не
делся) я решил перейти на <a href="http://akson45.kiev.ua/">akson45</a>. За 2 года работы с этим провайдером
суммарно связь пропадала на 8 часов (99.96% up) а в техническую поддержку
потребовалось звонить два раза, первый раз - при подключении внешнего IP
адреса, второй - уже во время падения канала связи.</p>

<p>С билайном я помнил телефон службы поддержки наизусть.</p>

<h2>Банк</h2>

<p>Уже долгое время я являюсь клиентом <a href="http://www.piraeusbank.ua/">Piraeus Bank МКБ</a> и пользуюсь практически
всем спектром услуг - платежными картами, расчетными счетами, интернет-банкингом для физический
лиц и предпринимателей. Специалист банка, Ольга Виценко, помогла разобраться
в тонкостях работы с зарубежным клиентом и радостно отвечала на все мои
вопросы (иногда довольно глупые) касательно банковских процессов, за что я ей
очень благодарен.</p>

<p>Использование интернет-банкинга позволило оплачивать услуги различных компаний без
комиссии, через стандартный банковский перевод. А SMS оповещение о транзакциях
позволило создать достаточно простую и удобную систему учета расходов, о которой,
возможно, я напишу в следующий раз.</p>

<p>Для того, чтобы поиграться с <a href="http://clubs.ya.ru/4611686018427461168/3">НСМЕП</a>, я завел карту в Экспресс-Банке.
Отношение к нему нейтральное. <a href="http://www.expres-bank.ua/">Сайт</a> не содержит тарифов по обслуживанию
карты, а для того, чтобы узнать о новых тарифах, требуется приехать в
отделение. Почему-то для получения листка с тарифами, требуется предоставить
свой паспорт. Тем не менее, их НСМЕП карта работает и я время от времени
пересобираю wine с поддержкой смарт-карт в своем <a href="https://launchpad.net/~rye/+archive/nsmep">PPA</a>.</p>

<h2>Оборудование</h2>

<p>Рабочий ноутбук у меня - <a href="http://rtg.in.ua/blog/2012/02/my-new-lenovo-edge-e420/">Lenovo E420</a> (после того, как Acer Aspire 5520 сжег
себе видеокарту). На батарее 25++ 94Wh (которую пришлось покупать в США, т.к. в
Украине рынок батарей какой-то странный) система обещает проработать 6 часов.
Этот ноутбук не может похвастаться низким энергопотреблением
(сравнение здесь - <a href="https://wiki.ubuntu.com/Kernel/PowerManagementRC6">тестирование RC6</a>),
однако у него очень удобная клавиатура, отличный экран и технические
характеристики).</p>

<p>Между моей локальной сетью и сетью провайдера находится TP-Link TL-WR1043ND
под управлением OpenWRT. Он служит как маршрутизатором, так и точкой доступа в
режиме 802.11g (хотя позволяет и .11n). Дополнительно он выступает как
шлюз в IPv6 сеть <a href="http://tb.netassist.ua/" title="Netassist IPv6 Tunnel Broker">NetAssist</a> и OpenVPN сервером на случай
необходимости подключения к рабочим ресурсам, которые не защищены SSL.</p>

<p>В моей локальной сети есть также сервер, который используется как хост для
хранения данных, вируальных машин различного назначения. Используется
четырехъядерный AMD Athlon&trade; II 630 @ 2.8Ггц. На материнскую плату
Asus M4A785T-M установлено 8Гб памяти, в качестве жестких дисков выступают 2x1.5Тб
Seagate ST31500341AS после того, как при скачке напряжения <a href="http://identi.ca/notice/60629003">вышли из строя</a> 2
винчестера Samsung SpinPoint F3 (как оказалось, довольно обычное явление
для этих дисков). В сеть подключены 2 гигабитные карты (RTL 8168B на
материнской плате и D-Link DGE-528T). Требуется обе, так как при
использования моста для виртуальных машин включение или выключение одной из
них приводит к временному зависанию сети на данной карте (видимо, из-за
перестройки топологии).</p>

<p>На сегодня все, надеюсь из данного поста удалось получить немного полезной
информации. Удачи!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Testing Libvirt Over TLS]]></title>
    <link href="http://rtg.in.ua/blog/2012/05/testing-libvirt-over-tls/"/>
    <updated>2012-05-05T12:38:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/05/testing-libvirt-over-tls</id>
    <content type="html"><![CDATA[<p>libvirt is a technology by RedHat that implements a single interface for
different types of virtualization methods. Personally I am using it for
kvm/qemu and LXC but the list on <a href="http://libvirt.org">their website</a> is much
larger and includes <a href="http://rtg.in.ua/blog/2009/12/ubuntu-910-running-in-uml/">UML</a>, Xen,
OpenVZ and even VMWare ESX and GSX hypervisors.</p>

<!-- more -->


<p>If you are using KVM directly, you will construct the command line manually
and will eventually end up with something like this:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>/usr/bin/kvm -S -M pc-1.0 -cpu qemu32 -enable-kvm -m 1024 \
</span><span class='line'>  -smp 4,sockets=4,cores=1,threads=1 \
</span><span class='line'>  -name plum -uuid 1da8c95b-0586-d41a-8c64-760034aa2453 \
</span><span class='line'>  -nodefconfig -nodefaults \
</span><span class='line'>  -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/plum.monitor,server,nowait \
</span><span class='line'>  -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown \
</span><span class='line'>  -drive file=/dev/vg0/vm-nfs-test,if=none,id=drive-virtio-disk0,format=raw\
</span><span class='line'>  ...</span></code></pre></td></tr></table></div></figure>


<p>I&#8217;ve been there and wrote scripts to help maintaining running kvm instances.
For a small number of options that&#8217;s OK. But interacting with the instances
through qemu monitor quickly became very tedious.</p>

<p>When I found out about libvirt I was really excited. Well, I am still excited
:)</p>

<p>The most user-visible part of libvirt is virt-manager which provides the GUI
for libvirt API. It looks like this:</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/5QME4mXXbmwLFnljsKiAdU/virt-manager.png"></p>

<p>It allows creating, reconfiguring, cloning and deleting virtual machines and storage,
configure virtual networks and control instance power states – almost
everything you need.</p>

<p>Another client is a CLI application called virsh:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>virsh # list
</span><span class='line'> Id Name                 State
</span><span class='line'>----------------------------------
</span><span class='line'>  4 plum                 running
</span><span class='line'>  5 WIN7PNX64EN          running
</span><span class='line'>  6 fedora-12            running
</span></code></pre></td></tr></table></div></figure>


<p>It allows modifying the VM description directly (that&#8217;s an XML file),
managing the storage pool&#8230; Well, pretty much everything that
virt-manager does and a little bit more, such as snapshotting the
VM (<strong>warning</strong>: I am currently investigating the
corruption happening with qcow2 disks but have not yet obtained any
results).</p>

<p>libvirt is a client library. The server side is implemented by libvirtd. libvirtd can
be connected to using Unix domain sockets or remotely via TCP.</p>

<p>To make contacting virtual hosts easier, the clients can connect to remote server
through SSH. Basically client creates a SSH session, starts <code>netcat</code> on the
remote end and makes it relay the commands from the local end to the Unix
domain socket, in this case you will see the following in the process list on
the server:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>sh -c if nc -q 2&gt;&1 | grep "requires an argument" &gt;/dev/null 2&gt;&1; then \
</span><span class='line'>  ARG=-q0;else ARG=;fi;nc $ARG -U /var/run/libvirt/libvirt-sock</span></code></pre></td></tr></table></div></figure>


<p>The grep is needed because there are 2 versions of netcat in existence,
netcat-openbsd and netcat-traditional, each with a bit different options.</p>

<p>The user account on the server should have access to
<code>/var/run/libvirt/libvirt-sock</code> file. The easiest thing to make this on
Ubuntu is to add the user account to <code>libvirtd</code> group. In Fedora it is
customary to use root account and allow root to log in via SSH (which I will
<a href="http://rtg.in.ua/blog/2009/06/permitrootlogin-yes-is-default-value/">never</a> do again).</p>

<p>For remote virtualization servers the SSH access has another benefit, you can
use VNC console without exposing the ports to an outside world. In this case
upon starting the virt-viewer or using graphical console from virt-manager a
new SSH connection is being created with&#8230;</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>sh -c nc -q 2&gt;&1 | grep "requires an argument" &gt;/dev/null; \
</span><span class='line'>    if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5906"; \
</span><span class='line'>    else   CMD="nc 127.0.0.1 5906";fi;eval "$CMD";</span></code></pre></td></tr></table></div></figure>


<p>That allows kvm to listen on loopback interface only, while still being
accessible to somebody with  a shell account on the VM server.</p>

<p>Now this is all nice but using SSH has one drawback, it
requires a shell account on the server which may not always be good.
Additionally every request to virsh requires the SSH session to be set up and
then torn down which adds to the total time of virsh command running. And when
I got 10 virtual machines, volume listing in virt-manager became really slow, which
I attributed completely to the SSH session. So I decided to set up everything
for TLS connection and compare the speed of qemu+tls vs qemu+ssh.</p>

<p>I am not using SASL authentication and rely on TLS certificates only.</p>

<p>So, connecting to TLS libvirtd faster? Yes. Is virt-manager working faster?
No.</p>

<p>First of all I followed the detailed instruction
at <a href="http://wiki.libvirt.org/page/TLSDaemonConfiguration">libvirt wiki</a>,
creating all the necessary keys and certificates. Since I wanted only one
account to become administrative one and not the whole host (even though I am
the only user on the laptop), I copied
the <code>clientcert.pem</code>, <code>clientkey.pem</code> and <code>cacert.pem</code> to <code>~/.pki/libvirt/</code>.</p>

<p>Then I added <code>-l</code> to the list of libvirtd options on the server in
<code>/etc/default/libvirt-bin</code> and restarted the service:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>restart libvirt-bin</span></code></pre></td></tr></table></div></figure>




<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ time virsh -c qemu+ssh://fridge.lappyfamily.net/system list
</span><span class='line'> Id Name                 State
</span><span class='line'>----------------------------------
</span><span class='line'>  4 plum                 running
</span><span class='line'>  6 fedora-12            running
</span><span class='line'>
</span><span class='line'>real    0m0.798s
</span><span class='line'>user    0m0.040s
</span><span class='line'>sys 0m0.012s
</span><span class='line'>
</span><span class='line'>$ time virsh -c qemu://fridge.lappyfamily.net/system list
</span><span class='line'> Id Name                 State
</span><span class='line'>----------------------------------
</span><span class='line'>  4 plum                 running
</span><span class='line'>  6 fedora-12            running
</span><span class='line'>
</span><span class='line'>real    0m0.134s
</span><span class='line'>user    0m0.052s
</span><span class='line'>sys 0m0.008s</span></code></pre></td></tr></table></div></figure>


<p>Yes! It is indeed faster! (qemu+tls:// is the same as qemu:// alone)</p>

<p>Checking virt-manager proved that while connecting is indeed faster, all other
operations are carried over an existing ssh connection and take exactly the
same amount of time, and that needs to be investigated in the code.</p>

<p>Remember that VNC connection is also proxied via SSH? VNC is gone for
virt-manager as it relies on proxying. No SSH - no VNC.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/2v94znglittEqojXjpmi8d/file.png"></p>

<p>You can get VNC console back if you reconfigure the VNC server to listen
on all interfaces. While virt-manager still can&#8217;t connect to the graphical
console, virt-viewer works perfectly.</p>

<p>So, having performed all these steps, I can say it was not really worth it but I will
definitely continue using the TLS connection because of initial speed-up. You
will want to use TLS connections under very particular circumstances
where using SSH may add an additional point of failure (such as automated
provisioning of virtual machines) or when you don&#8217;t want to expose SSH to an
external world and still be able to connect to libvirtd (with TLS
your traffic is both encrypted and authenticated using SSL).
Using SASL will allow to have libvirt-specific accounts separate from your server
accounts and that may be useful sometimes too.</p>

<h2>Serial console</h2>

<p>Now my remote libvirt server is accessible via SSH and TLS socket and, since
VNC is not available, I started to look into making the linux virtual machines
log to serial console as it <em>is</em> accessible over TLS socket. That&#8217;s
actually quite easy for kernel and getty on Ubuntu, haven&#8217;t found a way to
make grub cooperate yet:</p>

<p>Edit <code>/etc/default/grub</code> and alter the GRUB_CMDLINE_LINUX_DEFAULT to contain
the consoles you want the boot messages to appear on:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0"</span></code></pre></td></tr></table></div></figure>


<p>Run <code>update-grub</code> for the changes to take effect. This command will combine
all the files needed to bring grub up and write <code>/boot/grub/grub.cfg</code> file.
Never modify that file directly because the changes will be overwritten.</p>

<p>Now we run a login session on ttyS0 with this
<code>/etc/init/ttyS0.conf</code> <a href="http://rtg.in.ua/blog/2012/04/a-few-notes-on-upstart/">upstart</a> file:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># ttyS0 - getty
</span><span class='line'>#
</span><span class='line'># This service maintains a getty on tty1 from the point the system is
</span><span class='line'># started until it is shut down again.
</span><span class='line'>
</span><span class='line'>start on stopped rc RUNLEVEL=[2345] and (
</span><span class='line'>            not-container or
</span><span class='line'>            container CONTAINER=lxc or
</span><span class='line'>            container CONTAINER=lxc-libvirt)
</span><span class='line'>
</span><span class='line'>stop on runlevel [!2345]
</span><span class='line'>
</span><span class='line'>respawn
</span><span class='line'>exec /sbin/getty -8 38400 ttyS0</span></code></pre></td></tr></table></div></figure>


<p><code>start ttyS0</code> to make console appear and run <code>virsh console</code>:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ virsh console plum
</span><span class='line'>Connected to domain plum
</span><span class='line'>Escape character is ^]
</span><span class='line'>*press &lt;Enter&gt;*
</span><span class='line'>Ubuntu 12.04 LTS plum ttyS0
</span><span class='line'>
</span><span class='line'>plum login: </span></code></pre></td></tr></table></div></figure>


<p>Upon reboot the console will have all the info the kernel prints out.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ virsh console plum
</span><span class='line'>Connected to domain plum
</span><span class='line'>Escape character is ^]
</span><span class='line'>
</span><span class='line'>Ubuntu 12.04 LTS plum ttyS0
</span><span class='line'>
</span><span class='line'>plum login: rtg
</span><span class='line'>Password: 
</span><span class='line'>[...]
</span><span class='line'>rtg@plum:~$ sudo reboot
</span><span class='line'>[...]
</span><span class='line'> * Will now restart
</span><span class='line'>[  507.815197] Restarting system.
</span><span class='line'>[    0.000000] Initializing cgroup subsys cpuset
</span><span class='line'>[    0.000000] Initializing cgroup subsys cpu
</span><span class='line'>[    0.000000] Linux version 3.2.0-24-generic-pae (buildd@vernadsky) (gcc
</span><span class='line'>version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #37-Ubuntu SMP Wed Apr 25
</span><span class='line'>10:47:59 UTC 2012 (Ubuntu 3.2.0-24.37-generic-pae 3.2.14)
</span><span class='line'>[...]</span></code></pre></td></tr></table></div></figure>


<p>Nice, huh?</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[iSCSI in Ubuntu]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/iscsi-in-ubuntu/"/>
    <updated>2012-04-29T22:49:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/iscsi-in-ubuntu</id>
    <content type="html"><![CDATA[<p>Browsing for various OpenStack docks had eventually brought me to an extremely
nice <a href="https://fedoraproject.org/wiki/Scsi-target-utils_Quickstart_Guide">howto on scsi-target-utils</a> on fedoraproject.org.</p>

<p>A while ago I was searching for the docs on how to actually become an iSCSI
target but the one I found - <a href="http://www.howtoforge.com/using-iscsi-on-ubuntu-10.04-initiator-and-target">using iscsi on ubuntu 10.04</a> required to use a
kernel module and I had some weird problems (too weird to start diagnosing
just for fun).</p>

<p>So if you combine the target (server) setup from the fedoraproect.org with
initiator (client) from the howtoforge, you will get a pretty decent iSCSI
playground.</p>

<p>In ubuntu the package required for target is <code>tgt</code>.</p>

<p>In order for the server-side settings to be preserved across
reboots, the configuration must be stored via <code>tgt-admin</code> utility (notice that
this is a frontend to tgtadm):</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>tgt-admin -dump &gt; /etc/tgt/conf.d/my-target.conf</span></code></pre></td></tr></table></div></figure>


<!-- more -->


<p>iSCSI:</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/5l0xkQpJAoce0pLr2SKfHJ/iscsi.png"></p>

<p>Compare this to my local Lenovo E420 hard disk:</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/0dGvoY9Zsq5N6hEMclNrRS/local-e420.png"></p>

<p>The local hard disk is 5400RPM while the remote one was connected over GigE
connection (with 500MBit/s max throughput, therefore 62MB max read speed).</p>

<p>To whoever is wondering the images were created by <code>palimpsest</code>, a disk
utility by Redhat.</p>

<p>Now I just need to find a use case for this&#8230;</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Міжрегіональний центр видачі паспортів - отримання документів]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/passport-ua-centre-last-visit/"/>
    <updated>2012-04-28T12:08:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/passport-ua-centre-last-visit</id>
    <content type="html"><![CDATA[<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/51alMf5uJS4LNbdzCVOjJP/building.jpg"></p>

<p>Місяць тому я <a href="http://rtg.in.ua/blog/2012/03/passport-ua-centre-first-visit/">подав документи</a> у Міжрегіональний центр видачі паспортів, а
вчора отримав новий паспорт.</p>

<p>Знову їздив двічі, цього разу через те, що пропустив наступний пункт у
правилах:</p>

<blockquote><p>Після закінчення відповідного терміну для отримання паспорта громадянина
України для виїзду за кордон необхідно з внутрішнім паспортом <strong>та (за
наявності) діючого закордонного</strong> підійти до робочих місць 8,9.</p></blockquote>

<p>Дата видачі у самому паспорті - 10 квітня, тобто документ був готовий через 16 робочих днів.</p>

<p>Видача паспортів працює протягом усього робочого дня центру і займає 5 хвилин,
але зранку люди створюють чергу на отримання паспортів ще до відкриття центру.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Fixing IPCError in Ubuntu One]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/fixing-ipcerror-in-ubuntu-one/"/>
    <updated>2012-04-17T13:12:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/fixing-ipcerror-in-ubuntu-one</id>
    <content type="html"><![CDATA[<p>If you have not extensively used Ubuntu One earlier and migrated to Ubuntu
Precise (12.04), you may see the following IPCError when open Ubuntu One Control Panel:</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/1WmSMyBkfkHEhHI1QUWw7H/IPCError.png"></p>

<p>The reason for this is <a href="https://bugs.launchpad.net/bugs/927572" title="Launchpad Bug #927572: Cannot enable file sync once disabled via control panel" class="launchpad-bug">927572</a> in the old Control Panel code, in
Ubuntu Precise this switch was removed. However some users might have already
disabled their synchronization service.</p>

<p>To re-enable this, open <code>~/.config/ubuntuone/syncdaemon.conf</code> in your favorite
editor (don&#8217;t use <code>sudo</code>) find the following text:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>files_sync_enabled = False</span></code></pre></td></tr></table></div></figure>


<p>Please change the value to <code>True</code> so it reads</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>files_sync_enabled = True</span></code></pre></td></tr></table></div></figure>


<p>Save the file and re-open the control panel. Syncdaemon should now start and
control panel will work.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Ubuntuone Indicator 1.0.3 released]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/ubuntuone-indicator-1-dot-0-3-released/"/>
    <updated>2012-04-17T11:07:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/ubuntuone-indicator-1-dot-0-3-released</id>
    <content type="html"><![CDATA[<p>1.0.3 is mostly a bugfix release.</p>

<ul>
<li>Properly format estimation time (<a href="https://bugs.launchpad.net/bugs/982221" title="Launchpad Bug #982221: ETA time is shown in a funny way" class="launchpad-bug">982221</a>)</li>
<li>Order the Recently Published Files list by mtime (<a href="https://bugs.launchpad.net/bugs/981726" title="Launchpad Bug #981726: Preloaded Recently published list is ordered by UUID, i.e. randomly" class="launchpad-bug">981726</a>)</li>
<li>Use ubuntuone-client icons if icon-theme is not ubuntu-mono (<a href="https://bugs.launchpad.net/bugs/981350" title="Launchpad Bug #981350: indicator-ubuntuone is running but not displayed on panel if not using the default icon pack" class="launchpad-bug">981350</a>)</li>
<li>Remove old ubuntuone-indicator.desktop file (<a href="https://bugs.launchpad.net/bugs/982118" title="Launchpad Bug #982118: 1.0 should remove local autostart .desktop file from earlier versions" class="launchpad-bug">982118</a>)</li>
<li>Refresh quota after uploads, downloads and unlinks (<a href="https://bugs.launchpad.net/bugs/982164" title="Launchpad Bug #982164: Schedule quota update" class="launchpad-bug">982164</a>)</li>
<li>Automatically restart after upgrade (<a href="https://bugs.launchpad.net/bugs/983361" title="Launchpad Bug #983361: Restart indicator-ubuntuone automatically when updated" class="launchpad-bug">983361</a>)</li>
</ul>


<p>Subsequent upgrades won&#8217;t require the users to relogin as indicator will
restart itself automatically after every upgrade.</p>

<p>Further announcements will be posted at <a href="https://launchpad.net/indicator-ubuntuone/+announcements">Announcements page</a> only.</p>

<p>If you add <strong>ppa:rye/ubuntuone-extras</strong>, you&#8217;ll get all the updates
automatically:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>sudo apt-add-repository ppa:rye/ubuntuone-extras
</span><span class='line'>sudo apt-get update
</span><span class='line'>sudo apt-get install indicator-ubuntuone</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A Brand New Ubuntu One Indicator for Precise]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/a-brand-new-ubuntu-one-indicator-for-precise/"/>
    <updated>2012-04-13T19:01:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/a-brand-new-ubuntu-one-indicator-for-precise</id>
    <content type="html"><![CDATA[<p><img class="left" src="http://rtg.in.ua/assets/ubuntuone/4f1athl6AgeXrDcqtd4Yqe/indicator-ubuntuone.png"></p>

<p>About two years ago I published the <a href="http://rtg.in.ua/blog/2010/10/have-you-seen-my-weather-applet/">post on Ubuntu One Indicator</a>, which at
that time simply displayed the current synchronization state right from
syncdaemon by listening to DBus signals. Then it started to grow, attempting
to estimate the time needed for Ubuntu One to complete the operations.</p>

<p>Basically it looked like this:</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/5okeTEWc3ByMIP7w9oIrv2/ubuntuone-indicator-synchronizing.png"></p>

<p>Earlier this week I decided that I need to get some real-world experience with
<a href="https://live.gnome.org/Vala/">Vala</a> and since I have found the python-based indicator to consume about 40Mb of
residential memory I decided to rewrite the indicator completely.</p>

<p>A couple of days and bug reports later&#8230;</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/56mPm5bgPftbtmBx1hvmkD/precise-indicator-1.png"></p>

<p>Well, not much different, added some menu entries, but let&#8217;s add the file.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/1VcZbQ46LWyi0DxGzPM8JP/precise-indicator-2.png"></p>

<p>Yes, finally, it calculates the current transfer rate and provides the ETA for
the whole queue to be completed. It takes into account only Upload and
Download processes. Since Ubuntu One has greately improved the metadata
processing speed, metadata operations can be almost neglected.</p>

<p>However at the moment Downloads are not estimated properly since SyncDaemon
does not provide size info on Downloads, bugreport pending.</p>

<p>The 1.0.0 version is available in <strong>ppa:rye/ubuntuone-extras</strong>, for precise
the package was renamed to <strong>indicator-ubuntuone</strong> to be in line with other
packages. Installing transitional <code>ubuntuone-indicator</code> package will
install <code>indicator-ubuntuone</code>, so don&#8217;t be alarmed.</p>

<p>The <a href="https://launchpad.net/indicator-ubuntuone">project page</a> was renamed to <code>indicator-ubuntuone</code> too, &#8220;one-indicator&#8221; was
a weird name, really.</p>

<p>The new indicator installs an autostart file to <code>/etc/xdg/autostart</code> so it
should not be launched automatically. Old per-user autostart file can be
safely removed, by default it is at <code>~/.config/autostart/ubuntuone-indicator.desktop</code>). For
those who are interested, the compiled binary is located at
<code>/usr/lib/indicator-ubuntuone/indicator-ubuntuone</code>.</p>

<p>This is a call for testing. The upcoming 1.0.2 version contains the
estimation process described above and if you have some spare time please help
me testing it.</p>

<p>Grab a package below according to your architecture and install it using the
Software Center.</p>

<ul>
<li><a href="https://launchpad.net/~rye/+archive/daily/+files/indicator-ubuntuone_1.0.1%2B38%7Eprecise1_i386.deb">i386 indicator-ubuntuone</a></li>
<li><a href="https://launchpad.net/~rye/+archive/daily/+files/indicator-ubuntuone_1.0.1%2B38%7Eprecise1_amd64.deb">amd64 indicator-ubuntuone</a></li>
</ul>


<p>These all come from my <a href="https://launchpad.net/~rye/+archive/daily">daily PPA</a> which gets automatically built by Launchpad
using the recipe.</p>

<p>Please report the bugs you may find via <a href="https://bugs.launchpad.net/indicator-ubuntuone/+filebug">Launchpad bug tracker</a> and if you&#8217;d
like to help translating it to your native language, head over to
<a href="https://translations.launchpad.net/indicator-ubuntuone">Launchpad translations</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A few notes on Upstart]]></title>
    <link href="http://rtg.in.ua/blog/2012/04/a-few-notes-on-upstart/"/>
    <updated>2012-04-04T23:32:00+03:00</updated>
    <id>http://rtg.in.ua/blog/2012/04/a-few-notes-on-upstart</id>
    <content type="html"><![CDATA[<blockquote><p>Upstart is an event-based replacement for the /sbin/init daemon which handles<br/>starting of tasks and services during boot, stopping them during shutdown and<br/>supervising them while the system is running.</p><footer><strong>Upstart Cookbook</strong> <cite><a href='http://upstart.ubuntu.com/cookbook/'>upstart.ubuntu.com/cookbook/&hellip;</a></cite></footer></blockquote>


<p>Upstart scripts have always seemed to be weird to me, somebody who learned
about SysV init subsystem in Mandrake and Slackware long time ago. It
seemed like Upstart was lacking lot of features but it turns out I
just needed to understand what I was looking for.</p>

<p>So, when I found a need of a startup script, I decided to stop treating
upstart as magic and here are my Upstart examples.</p>

<ul>
<li>Respawning a dead daemon.</li>
<li>Monitoring the respawn limit.</li>
<li>Disable an Upstart task.</li>
<li>Listing, starting, stopping and restarting Upstart jobs.</li>
</ul>


<!-- more -->


<h1>Respawning a dead daemon</h1>

<p>To stream files to my XBox (yes, I admit I have an XBox 360 console) I am
using <a href="http://ushare.geexbox.org">uShare</a>, an outstanding piece of software that is actually compatible
with Microsoft&#8217;s <a href="http://gxben.wordpress.com/2008/08/24/why-do-i-hate-dlna-protocol-so-much/">horrible DLNA specification</a>. (That&#8217;s also the reason
why I am so passionate about <a href="https://bugs.launchpad.net/bugs/973295" title="Launchpad Bug #973295: [precise] Enable grilo plugin" class="launchpad-bug">973295</a> and <a href="https://bugs.launchpad.net/bugs/880076" title="Launchpad Bug #880076: Coherence DLNA/UPnP client not showing on plugins list" class="launchpad-bug">880076</a>)</p>

<p>The default startup script simply launches the daemon as root and does nothing to keep
it alive. As I found out one can make uShare exit by sending too much requests
to reload the contents via its web server (this resulted from a bug in my inotify
monitoring script, though). How do we ask upstart to restart the daemon?</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># uShare UPnP A/V & DLNA Media Server
</span><span class='line'>
</span><span class='line'>description     "uShare UPnP A/V & DLNA Media Server"
</span><span class='line'>author          "Roman Yepishev &lt;rtg@rtg.in.ua&gt;"
</span><span class='line'>
</span><span class='line'>start on filesystem
</span><span class='line'>stop on runlevel [016]
</span><span class='line'>
</span><span class='line'>respawn
</span><span class='line'>
</span><span class='line'>setuid nobody
</span><span class='line'>setgid nogroup
</span><span class='line'>
</span><span class='line'>exec /usr/bin/ushare</span></code></pre></td></tr></table></div></figure>


<p>Original script reads $USHARE_OPTIONS from /etc/default/ushare which does not
actually exist so I left this out.</p>

<p>What we have above is a simple script that:</p>

<ul>
<li>Starts uShare when filesystem is available.</li>
<li>Stops uShare when system is rebooted or shut down.</li>
<li>Respawns the process if it exits.</li>
<li>Makes the process run as nobody:nogroup because I don&#8217;t want it to run as
root.</li>
</ul>


<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ sudo start ushare
</span><span class='line'>ushare start/running, process 5892
</span><span class='line'>$ sudo kill 5892
</span><span class='line'>$ sudo status ushare
</span><span class='line'>ushare start/running, process 7800</span></code></pre></td></tr></table></div></figure>


<p>Yep, it works as expected. <code>restart</code> stanza (that&#8217;s how these directives are
called in Upstart) by default have a <code>respawn limit 10 5</code> - 10 respawns per 5 seconds.
If the application respawns faster than that (basically meaning that it cannot start
successfully), the init will say</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>init: ushare respawning too fast, stopped</span></code></pre></td></tr></table></div></figure>


<p>While <code>respawn limit</code> can be made <code>unlimited</code> such rate of failure signals
about a real problem.</p>

<h1>Monitoring the respawn limit</h1>

<p>uShare crashing is not such a big deal but it was still upsetting my wife
since it meant I had to log into the server and start the process again
(before I fixed a bug in my script). But then I thought that notifying an
administrator about such kind of an issue is a good thing.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># upstart-respawn-monitor.conf
</span><span class='line'>
</span><span class='line'>description "Notify root about ushare respawn limit reached"
</span><span class='line'>author "Roman Yepishev &lt;rtg@rtg.in.ua&gt;"
</span><span class='line'>
</span><span class='line'># This is a short-lived job
</span><span class='line'>task
</span><span class='line'>
</span><span class='line'>start on stopped ushare PROCESS=respawn
</span><span class='line'>
</span><span class='line'>script
</span><span class='line'>    echo "uShare reached respawn limit" | mail -s "uShared crashed" root
</span><span class='line'>end script</span></code></pre></td></tr></table></div></figure>


<p>This is another type of Upstream job, a <a href="http://upstart.ubuntu.com/cookbook/#task-job">task</a>. It does not keep running
forever and once it has finished it will not be respawned. Same thing as
setting up the hostname. But what we just did is we created a simple
monitoring task that will be executed once <code>ushare</code> reaches <code>stopped</code> state and
a special PROCESS variable will have <code>respawn</code> as the reason why the service
stopped. And this is all was done using Upstart only.</p>

<h1>Disable Upstart task</h1>

<p>By default when you install a service in Ubuntu it will be immediately
started. Sometimes, however, you might want to start service manually. Upstart
has a concept of <a href="http://upstart.ubuntu.com/cookbook/#override-files">overrides</a> that basically let you override any stanza in the
original job file by creating a <code>job.override</code> file in <code>/etc/init</code> and placing
the stanzas here.</p>

<p>For example, I am using LXC for some machines now and these lack tty[2..6].
Actually I don&#8217;t need these but <code>getty</code> keeps trying to start on these ttys. It
starts and stops pretty slowly so it is not caught by the <code>respawn limit</code>.
Disabling a task in Upstart is trivial, you just need to add <code>manual</code> to the
stanzas:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># /etc/init/tty2.override
</span><span class='line'>manual</span></code></pre></td></tr></table></div></figure>


<p>Yes, it is that easy. Another example when you&#8217;d want to use an override file
is when you want to use different options on the <code>exec</code> line (you will copy the
original <code>exec</code> line and change the options then) or change the <code>start on/stop
on</code> conditions.</p>

<h1>Starting, stopping and restarting</h1>

<p>If the job was converted to upstart, then interacting with it is easy:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class=''><span class='line'># Getting the info about particular job:
</span><span class='line'>$ sudo status cups
</span><span class='line'>cups start/running, process 1503
</span><span class='line'>
</span><span class='line'># Getting the info about all the jobs
</span><span class='line'>$ sudo initctl list
</span><span class='line'>avahi-daemon start/running, process 1526
</span><span class='line'>cgroup-lite start/running
</span><span class='line'>mountall-net stop/waiting
</span><span class='line'>nmbd start/running, process 1649
</span><span class='line'>qemu-kvm start/running
</span><span class='line'>...
</span><span class='line'>
</span><span class='line'># Stopping a job
</span><span class='line'>$ sudo stop cups
</span><span class='line'>cups stop/waiting
</span><span class='line'>
</span><span class='line'># Starting a job
</span><span class='line'>$ sudo start cups
</span><span class='line'>cups start/running, process 7870
</span><span class='line'>
</span><span class='line'># Ask the job to reload the configuration (SIGHUP)
</span><span class='line'>$ sudo reload ushare</span></code></pre></td></tr></table></div></figure>


<p>For SysV compatibility, the packages that were converted to upstart
ship the symlinks pointing from /etc/init.d/something to
/lib/init/upstart-job.</p>

<p>When you run e.g. <code>/etc/init.d/cups restart</code> you will receive an explanation
how can you run the upstart job without the compatibility layer:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Rather than invoking init scripts through /etc/init.d, use the service(8)
</span><span class='line'>utility, e.g. service cups restart
</span><span class='line'>
</span><span class='line'>Since the script you are attempting to invoke has been converted to an
</span><span class='line'>Upstart job, you may also use the stop(8) and then start(8) utilities,
</span><span class='line'>e.g. stop cups ; start cups. The restart(8) utility is also available.
</span><span class='line'>cups stop/waiting
</span><span class='line'>cups start/running, process 4028</span></code></pre></td></tr></table></div></figure>


<h1>Cookbook</h1>

<p>The Upstart cookbook may look like it is an unnecessary long document but once
you start reading it you realise the potential of upstart. I did not want to
learn about upstart until I found myself in a need of a startup script that
could launch and control my daemon, the whole script ended up having 7 stanzas
and I did not even need to handle forking, respawning and privilege dropping in the
daemon itself.</p>

<p>Seriously, if you are in doubt, read the <a href="http://upstart.ubuntu.com/cookbook/">Upstart Cookbook</a>. Have fun!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Міжрегіональний центр видачі паспортів - подача документів]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/passport-ua-centre-first-visit/"/>
    <updated>2012-03-20T12:16:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/passport-ua-centre-first-visit</id>
    <content type="html"><![CDATA[<p><strong>Оновлено</strong>: Паспорт я <a href="http://rtg.in.ua/blog/2012/04/passport-ua-centre-last-visit/">отримав</a>.</p>

<p>Через 5 років після того, як я отримав свій перший закордонний паспорт, в ньому
закінчилися сторінки для віз. Варіантів отримання нового - два, за місцем
прописки (м. Бровари, де &#8220;немає апаратури&#8221;) або міжрегіональний центр видачі
паспортів, що в інтернеті відомий як <a href="http://passport-ua.org">passport-ua.org</a>.</p>

<p>На сайті присутня можливість запису у електронну чергу, але через те, що
кількість місць у електронній черзі приблизно 10 на день, то протягом перших
секунд восьмої години ранку (коли &#8220;відкривається&#8221; запис) розбираються всі
наявні місця.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/0DRWIRw83sJNK1N30efWhO/road.jpg"></p>

<p>Але можна приїхати туди і без запису. Різні джерела свідчать про різні часові
проміжки, комусь пощастило приїхати і пройти всі процедури після обіду, хтось
приїжджав о сьомій годині ранку і записувався у чергу на вхід (відкривається
центр о дев’ятій). Я приїхав о восьмій і був тридцятим у списку на листочку на
підвіконні.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/4GmlSvT0hggc2K0GajDT7i/index.jpg"></p>

<p>Більш логічно було б зробити тільки електронну чергу (наприклад, як в візовому
центрі великобританії), майже всі люди, що були в черзі, перед цим намагалися
записатися онлайн.</p>

<p>Близько дев’ятої години ми пройшли в середину, отримали номери вже у
внутрішній черзі. Через те, що я забув оригінал ідентифікаційного коду,
довелося повертатися додому і вдруге їхати у центр, за півтори години
прийому документи встигли подати 30 чоловік.</p>

<p>Для отримання паспорту <a href="http://passport-ua.org/ua/documents.html">потрібні</a>:</p>

<ul>
<li>Оригінал внутрішнього паспорту та 2 копії перших двох сторінок та всіх
сторінок з штампами про реєстрацію. Якщо були вклеєні фотокартки у 25 та
45 років, то також потрібні копії сторінок з фотографією та даними про
вклеювання.</li>
<li>Оригінал та копія ідентифікаційного коду на листку A4.</li>
<li>У разі, якщо попередній паспорт залишається у себе, то також потрібна
копія першої сторінки закордонного паспорту.</li>
<li>Гроші</li>
</ul>


<p>Після подання документів потрібно оплатити рахунки у місцевому відділенні
<a href="http://cib.com.ua/">Комерційного індустріального банку</a>, який за кожну
транзакцію стягує 1% але не менше 10₴. Таких транзакцій буде 5, тож до суми
треба додати ще 50₴.</p>

<p>Після сплати рахунків треба чекати на свою чергу на фотографування та
перевірку даних, ця процедура чомусь займає найбільше часу.</p>

<p>В моєму випадку на це пішло 8 хвилин, після цього через місяць треба прийти та
отримати паспорт.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Upload to Ubuntu One using curl]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/upload-to-ubuntu-one-using-curl/"/>
    <updated>2012-03-18T21:35:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/upload-to-ubuntu-one-using-curl</id>
    <content type="html"><![CDATA[<p><strong>2012-04-15</strong>: I found it easier to work with signed URLs instead of a
separate header so the blog post was updated to use these.</p>

<p>I like to find new ways to use Ubuntu One as easily as possible. I have to
admit that the way described in <a href="http://rtg.in.ua/blog/2011/08/ubuntu-one-headless/">headless Ubuntu One article</a> is not quite
reliable since the file is read completely into the memory prior to being
transferred. For large files this can be a problem. There are ways to make
this work for streaming, but while investigating this I found out that we can
use curl for uploads and downloads, without any python wrapper.</p>

<p>Basically,</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'> curl -H "Content-Type: text/plain" \
</span><span class='line'>      -T filename `oauth-sign -m PUT -f url http://.../~/Folder/filename`</span></code></pre></td></tr></table></div></figure>


<p>But in order to make it work this way, you need just a tiny bit of
preparation.</p>

<!-- More -->


<p>First of all, you must know that all Ubuntu One clients <em>do not use your
password</em> for authentication, a mechanism called OAuth is used. This means
that for every connected device a separate set of credentials is being
assigned allowing you to invalidate the access whenever you need and keep the
really secret password to yourself.</p>

<p>OAuth is not natively supported by curl but since this is <a href="http://oauth.net/">a standard</a>, there
are multiple solutions you can use to generate the signed URL as in the
example, there are no proprietary extensions or cookie storage.</p>

<p>If you have ruby, installing <code>oauth</code> gem will
give you an <code>oauth</code> script. In case you are after the python one, you can grab
<a href="http://people.canonical.com/~roman.yepishev/us/oauth-sign">my implementation</a> from <a href="https://code.launchpad.net/~rye/+junk/ubuntuone-scripts">ubuntuone-scripts</a> branch.</p>

<p>First you need to get the tokens, we are using pretty much the same
<a href="http://people.canonical.com/~roman.yepishev/us/ubuntuone-sso-login.py">ubuntuone-sso-login.py</a> script I mentioned in the article about the REST
client, however in order to make it simpler to pass the arguments to other
application, i added the <code>--format</code> (<code>-f</code>) option, which does the following:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ python ubuntuone-sso-login.py --format shell
</span><span class='line'>Ubuntu SSO Login: sso-test-2@rtg.in.ua
</span><span class='line'>Password: 
</span><span class='line'>export OAUTH_CONSUMER_KEY=RwLrAWeWF
</span><span class='line'>export OAUTH_CONSUMER_SECRET=fGztsAJHqfWFirGBwXNhiEinrfqCvDVtEHJ
</span><span class='line'>export OAUTH_TOKEN=HFvMpWiwuOcGOsSSRbkaHZHMARjHZrfYYRa
</span><span class='line'>export OAUTH_TOKEN_SECRET=BjnVKEhQqCaTcHWBEywVGMhlqvtTBzFFeykYC</span></code></pre></td></tr></table></div></figure>


<p>Copy the lines with export to some shell script file, e.g.
~/.ubuntuone-credentials, we will
re-use the names further.</p>

<p>Now create the signed URL with ruby implementation:
We are requesting account information, so the URL is
https://one.ubuntu.com/api/account/.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ SIGNED_URL=`oauth --consumer-key $OAUTH_CONSUMER_KEY \
</span><span class='line'>        --consumer-secret $OAUTH_CONSUMER_SECRET \
</span><span class='line'>        --method GET --uri https://one.ubuntu.com/api/account/ \
</span><span class='line'>        --secret $OAUTH_TOKEN_SECRET \
</span><span class='line'>        --token $OAUTH_TOKEN -v -Q sign | grep 'OAuth Request URI:' \
</span><span class='line'>                                    | sed 's/OAuth Request URI: //'`</span></code></pre></td></tr></table></div></figure>


<p>The weird grep/sed dance is needed to get the URL to the
format we will use later, you will not need to do this in python oauth-sign
version.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ curl $SIGNED_URL
</span><span class='line'>{"username": "https://login.ubuntu.com/+id/RwLWeWF", "openid":
</span><span class='line'>"https://login.ubuntu.com/+id/RwLWeWF", "last_name": "Roman"
</span><span class='line'>...</span></code></pre></td></tr></table></div></figure>


<p>Great, it works. Now let&#8217;s work with the actual files.</p>

<p>According to the <a href="https://one.ubuntu.com/developer/files/store_files/cloud">docs</a> the file root is
<code>https://files.one.ubuntu.com/</code> and PUT is being used to upload
the files.</p>

<p>Please note the following paragraph in the docs -</p>

<blockquote><p>Note also that content_paths may not be rooted under the API root, and may
change without warning, so they must be read from the API and not hardcoded.</p></blockquote>

<p>The current way assumes that the content path is
/content/~/path/to/volume/path/to/file.txt, this works for now, but may change
in the future.</p>

<p>First let&#8217;s construct the URL we will be using</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ URL=https://files.one.ubuntu.com/content/~/Ubuntu%20One/hello.txt</span></code></pre></td></tr></table></div></figure>


<p>Now we need to get the signed URL. Let&#8217;s use the python version. Since
it reads the environment variables, there is no need to specify the token:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ SIGNED_URL=`oauth-sign -f url -m PUT $URL`</span></code></pre></td></tr></table></div></figure>


<p>Yes, that&#8217;s it.</p>

<p>Now create a hello.txt file locally and then upload it with curl, specifying <code>-T</code> makes
curl use &#8220;PUT&#8221; request and transfer the file without buffering it in the memory:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ echo "Hello, Ubuntu One" &gt; hello.txt
</span><span class='line'>$ curl -T hello.txt -H "Content-Type: text/plain" $SIGNED_URL
</span><span class='line'>
</span><span class='line'>{"kind": "file", "public_url": null, "hash":
</span><span class='line'>"sha1:b50a416c31dbf3620b72cd7841980541805db44e", ...</span></code></pre></td></tr></table></div></figure>


<p>Great! Now let&#8217;s get the file:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ curl `oauth-sign -f url $URL`
</span><span class='line'>Hello, Ubuntu One</span></code></pre></td></tr></table></div></figure>


<p>Please note that I am specifying Content-Type header too. Usually when you ask
curl to upload with <code>--data-binary</code>, it sets the Content-Type to
application/octet-stream. This is a mandatory header for Ubuntu One servers.
In case you omit the header, you will receive an error.</p>

<p>I have some SQL databases of text messages, expenses and a test openfire
installation. I upload the backup regularly to Ubuntu One with the following
simple script. I put the OAUTH exports to ~/.ubuntuone-credentials file which
gets sourced.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="c">#!/bin/bash</span>
</span><span class='line'>
</span><span class='line'><span class="nv">URL</span><span class="o">=</span>https://files.one.ubuntu.com/content/~/Ubuntu%20One
</span><span class='line'>
</span><span class='line'><span class="nb">source</span> ~/.ubuntuone/credentials
</span><span class='line'>
</span><span class='line'><span class="k">for </span>DB in webapp openfire; <span class="k">do</span>
</span><span class='line'><span class="k">    </span><span class="nv">RESULT</span><span class="o">=</span>fridge-<span class="nv">$DB</span>.sql.gz
</span><span class='line'>
</span><span class='line'>    <span class="nv">LOCALFILE</span><span class="o">=</span><span class="sb">`</span>mktemp /tmp/backup-XXXXXX<span class="sb">`</span>
</span><span class='line'>
</span><span class='line'>    pg_dump -h localhost -U backup <span class="nv">$DB</span> | gzip &gt; <span class="nv">$LOCALFILE</span>
</span><span class='line'>
</span><span class='line'>    <span class="nv">REMOTE</span><span class="o">=</span><span class="nv">$URL</span>/<span class="nv">$RESULT</span>
</span><span class='line'>
</span><span class='line'>    <span class="nv">SIGNED_URL</span><span class="o">=</span><span class="sb">`</span>~/bin/oauth-sign -f url -m PUT <span class="nv">$REMOTE</span><span class="sb">`</span>
</span><span class='line'>
</span><span class='line'>    <span class="nv">HEADERS</span><span class="o">=</span><span class="sb">`</span>mktemp /tmp/headers-XXXXXX<span class="sb">`</span>
</span><span class='line'>    <span class="nv">OUTPUT</span><span class="o">=</span><span class="sb">`</span>mktemp /tmp/output-XXXXXX<span class="sb">`</span>
</span><span class='line'>
</span><span class='line'>    <span class="nv">STATUS</span><span class="o">=</span><span class="sb">`</span>curl --silent --fail --dump-header <span class="nv">$HEADERS</span> <span class="se">\</span>
</span><span class='line'>         -T <span class="s2">&quot;$LOCALFILE&quot;</span> <span class="se">\</span>
</span><span class='line'>         -H <span class="s2">&quot;Content-Type: application/octet-stream&quot;</span> <span class="se">\</span>
</span><span class='line'>         -w <span class="s1">&#39;%{http_code}&#39;</span> <span class="nv">$SIGNED_URL</span> -o <span class="nv">$OUTPUT</span><span class="sb">`</span>
</span><span class='line'>    <span class="nv">rv</span><span class="o">=</span><span class="nv">$?</span>
</span><span class='line'>    <span class="k">if</span> <span class="o">[</span> <span class="s2">&quot;$rv&quot;</span> !<span class="o">=</span> <span class="s2">&quot;0&quot;</span> -o <span class="se">\(</span> <span class="s2">&quot;$STATUS&quot;</span> !<span class="o">=</span> <span class="s2">&quot;201&quot;</span> -a <span class="s2">&quot;$STATUS&quot;</span> !<span class="o">=</span> <span class="s2">&quot;200&quot;</span> <span class="se">\)</span> <span class="o">]</span>; <span class="k">then</span>
</span><span class='line'><span class="k">        </span><span class="nb">echo</span> <span class="s2">&quot;Backup of $DB failed!&quot;</span>
</span><span class='line'>        <span class="nb">echo</span> <span class="s2">&quot;The server replied with:&quot;</span>
</span><span class='line'>        <span class="nb">echo</span>
</span><span class='line'><span class="nb">        </span>cat <span class="nv">$HEADERS</span>
</span><span class='line'>        <span class="nb">echo</span>
</span><span class='line'><span class="nb">        </span>cat <span class="nv">$OUTPUT</span>
</span><span class='line'>    <span class="k">fi</span>
</span><span class='line'>
</span><span class='line'><span class="k">    </span>rm <span class="nv">$OUTPUT</span>
</span><span class='line'>    rm <span class="nv">$HEADERS</span>
</span><span class='line'>    rm <span class="nv">$LOCALFILE</span>
</span><span class='line'>
</span><span class='line'><span class="k">done</span>
</span></code></pre></td></tr></table></div></figure>



]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[nxtractor: My Simple Data Recovery Application for Linux]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/nxtractor-my-simple-data-recovery-application-for-linux/"/>
    <updated>2012-03-18T13:06:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/nxtractor-my-simple-data-recovery-application-for-linux</id>
    <content type="html"><![CDATA[<p>Once upon a time I had no stable internet connection, <a href="http://mult.ru">mult.ru</a> distributed
their &#8220;Масяня&#8221; movies as .exe files and I was not happy with them on the linux
system.</p>

<p>In order to make a standalone movie, flashplayer.exe was simply concatenated
with .SWF file so in order to get .SWF file I simply needed the offsets and
end markers (if any).</p>

<p>I wrote a simple application in C that was doing exactly this.</p>

<p>At some point in time my friend Alex asked me to see whether it is possible to
recover any data from his hard drive which had the file system table
completely overwritten by something and no data recovery software was able to
extract anything.</p>

<p>I dumped the image of the whole 20Gb drive and began thinking about the way
the data can be recovered if we don&#8217;t know where to look.</p>

<p>And so the <code>nxtractor</code> project was born. During next couple of days I was
examining the signatures and markers of the most common file types, wrote the
scanner that was sufficiently good and extensible.</p>

<!-- More -->


<p>I did not recover much information from that drive, the data was too
fragmented.</p>

<p>This was back in 2003.</p>

<p>Fast forward to today, when SD cards hold the same amount of data as the hard
drives of the past, data storage devices are used by music players, phones and
cameras.</p>

<p>Now imagine you have deleted all the photos from the camera. What would you
do?</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>sudo dd if=/dev/mmcblk0p1 of=~/mmcblk0p1.bin
</span><span class='line'>bzr branch lp:~rye/+junk/nxtractor
</span><span class='line'>cd nxtractor
</span><span class='line'>make
</span><span class='line'>./nxtractor ~/mmcblk0p1.bin</span></code></pre></td></tr></table></div></figure>


<p>Yes, that&#8217;s the ideal variant, in some cases you will need to do adjustments
to the types.dat file, for example to prevent false positives on TIFF files
you comment out the whole type=tiff block. To make sure you are not extracting
embedded data in JPEG files, set minsize in type=jpeg to 1000000 which means
that it will not consider files that are less than 1Mb to be real jpeg files.</p>

<p>For example, my recent removal of the photos resulted in the following:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>nxtractor ~/mmcblk0p1.bin
</span><span class='line'>Scanning started at Sun Mar 18 13:34:01 2012
</span><span class='line'>@365056: Found magic numbers for jpeg. Size: 1043563
</span><span class='line'>@1413632: Found magic numbers for jpeg. Size: 1249863
</span><span class='line'>@2691584: Found magic numbers for jpeg. Size: 1225140
</span><span class='line'>@3936768: Found magic numbers for jpeg. Size: 726782
</span><span class='line'>@4690432: Found magic numbers for jpeg. Size: 685591
</span><span class='line'>@5378560: Found magic numbers for jpeg. Size: 860730
</span><span class='line'>@6263296: Found magic numbers for jpeg. Size: 712114
</span><span class='line'>@6984192: Found magic numbers for jpeg. Size: 757930
</span><span class='line'>@7770624: Found magic numbers for jpeg. Size: 872590
</span><span class='line'>...
</span><span class='line'>@59707904: Found magic numbers for jpeg. Size: 1775034
</span><span class='line'>Scanning finished at Sun Mar 18 13:34:05 2012
</span><span class='line'>Extraction started at Sun Mar 18 13:34:05 2012
</span><span class='line'>Writing mmcblk0p1.bin.nx/jpeg/365056.jpg (1043563 bytes)
</span><span class='line'>...
</span><span class='line'>Writing mmcblk0p1.bin.nx/jpeg/59707904.jpg (1775034 bytes)
</span><span class='line'>Extraction finished at Sun Mar 18 13:34:05 2012</span></code></pre></td></tr></table></div></figure>


<p>And I got my files back! Several years after I wrote the software. It&#8217;s
amazing :)</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/54gNKv3YEWxUJuLTJJAaHO/nxtractor-result.png"></p>

<p>This software is provided AS IS and I have no plans developing it any more,
use at your own risk. For another Open Source data recovery software, see
<a href="http://www.cgsecurity.org/">CGSecurity&#8217;s TestDisk and PhotoRec</a>.</p>

<p>And please, backup your sensitive data. NOW.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[HPCloud: Object Storage Speed]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/hpcloud-object-storage-speed/"/>
    <updated>2012-03-18T01:59:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/hpcloud-object-storage-speed</id>
    <content type="html"><![CDATA[<p>So several weeks into beta-testing I decided to check how useful Object
Storage (Amazon S3-like service) is. My use case was a 4Gb file and I wanted to run data
recovery application on it.</p>

<!-- more -->


<h1>Setting up</h1>

<p>Object Storage used to be called &#8220;Swift&#8221; in RackSpace, so the command I tried
to upload the data was called <code>swift</code>. The versions prior to the ones included
in Ubuntu Precise (or <a href="http://wiki.openstack.org/PPAs">OpenStack PPA</a>) will not work, they will cause the
server to emit Internal Server Error response.</p>

<p>In order not to repeat the credentials on the command line, set the following
environment variables</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>ST_AUTH=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/
</span><span class='line'>ST_USER=$OS_TENANT_NAME:$OS_USERNAME
</span><span class='line'>ST_KEY=$OS_PASSWORD</span></code></pre></td></tr></table></div></figure>


<p>Where OS_TENANT_NAME is the name of the tenant as seen on the management web
ui and OS_USERNAME in HPCloud is your EMail address. OS_PASSWORD is your
HPCloud password. These OS_ variables are actually useful to be set for <code>nova</code>
command, in this case OS_AUTH_URL will be equal to the identity service one.</p>

<p>Once you get the vars set up, run <code>swift stat</code></p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>   Account: 12122100844760
</span><span class='line'>Containers: 3
</span><span class='line'>   Objects: 1
</span><span class='line'>     Bytes: 352116
</span><span class='line'>Accept-Ranges: bytes
</span><span class='line'>X-Trans-Id: txefd92a9c9312434d9892a9c4b272f29e</span></code></pre></td></tr></table></div></figure>


<h1>Uploading</h1>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>$ swift upload -S segment_size_in_bytes bucketname filename.ext</span></code></pre></td></tr></table></div></figure>


<p>Why do I use the segment size? In ideal world, network connections never drop
so all uploads and downloads terminate successfully. In our world there may be
some weird hickup right at the last byte and the whole transfer is considered
corrupted.</p>

<p>So S3 tools learned to split the original file in blocks of arbitrary sizes
and writing manifest file that links to the original ones. In case one of the
parts fails to transfer properly, it gets retried, not the whole file. To
reconstruct the original file client reads the mainfest and fetches all the
parts.</p>

<p>OpenStack has a similar approach, yet <a href="http://docs.openstack.org/trunk/openstack-object-storage/admin/content/direct-api-management-of-large-objects.html">a bit different</a> implementation.</p>

<p>I found that <code>swift</code> started eating my CPU like mad, spiking at 130% CPU.
It looks like the client (which is based on
eventlet.green.httplib) is doing more than it should between the writes. This
will need to be looked into - <a href="https://bugs.launchpad.net/bugs/959221" title="Launchpad Bug #959221: swift consumes over 100% of cpu during upload" class="launchpad-bug">959221</a>. It took quite
some time to start printing anything to the terminal, being impatient I shut
it down several times and decided to use <code>nice</code> during the last attempt.</p>

<p>During the last test, the upload speed to Object Storage from Ukraine was
around 1.4MBit/s (174kB/s). That&#8217;s slow.</p>

<h1>Downloading</h1>

<p>The instance I was using (Ubuntu 11.10) came with outdated packages so I added the
milestone-proposed <a href="http://wiki.openstack.org/PPAs">OpenStack PPA</a> and upgraded swift.</p>

<p>There I issued</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>swift download bucketname filename.ext</span></code></pre></td></tr></table></div></figure>


<p>and the file started to reassemble locally.</p>

<p>The fastest connection to Object Storage is from the HPCloud itself, but I was
not able to get more than <em>32MBit/s</em> (that&#8217;s around <em>4MiB/s</em> which is slower than
my home uplink at the moment). This does not appear to be directly connected
with Object Storage, all other connections to outside world are also throttled
this way.</p>

<p>I hope this is just because of the beta stage and the attempt to provide reasonable speed
for everyone.</p>

<h1>Publishing</h1>

<p>Yes, OpenStack Object Storage support ACLs on the buckets, so while we are at
it, here&#8217;s how the bucket can be published:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>swift put --read-acl .r:* bucketname</span></code></pre></td></tr></table></div></figure>


<p>Basically, it <del>assigns a read permission for everyone</del> instructs object storage
to allow bucket access for any referrer (see <a href="http://swift.openstack.org/misc.html#acls">ACLs</a> for more info). I haven&#8217;t looked into
sharing with specific accounts yet.</p>

<p>In order to test the download speed from Object Storage, I put an awesome
<a href="http://www.bigbuckbunny.org">Big Buck Bunny</a> made with open source tools to such public bucket where it
will be available until 2012-04-18 - <a href="https://region-a.geo-1.objects.hpcloudsvc.com/v1.0/12122100844760/rtg.in.ua-test/big_buck_bunny_480p_stereo.ogg">big_buck_bunny_480p_stereo.ogg</a>, you can
test the download speed from your location. For me it is 1.5Mbit/s.</p>

<p>Since HPCloud is located in the US, the data had to travel around the globe,
so it impacted the speed. In this case having multiple availability zones
(e.g. USA, Europe and Asia) is a must.</p>

<p>Actually I ended up recovering the files on my local machine, however I
performed the same steps on the HPCloud instance afterwards. Should I suddenly stop
having my local machine, I&#8217;d need about 2 instance-hours to get everything
I need excluding the initial upload of 4GiB file and subsequent download of the recovered data.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Setting up my Xerox WC3119]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/setting-up-my-wc3119-again/"/>
    <updated>2012-03-12T00:07:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/setting-up-my-wc3119-again</id>
    <content type="html"><![CDATA[<p>I am doing this over and over again. Along with writing puppet config I am
posting the recipe for my WC3119 configuration here in case somebody finds it
useful too. I will describe the setup of Xerox WC3119 for remote printing and
scanning in Ubuntu Precise Pangolin (12.04).</p>

<!-- more -->


<h1>Remote CUPS access</h1>

<p>By default CUPS does not listen to the network and does not accept remote
configuration. Edit <code>/etc/cups/cupsd.conf</code> to enable:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class='apache'><span class='line'><span class="c"># For IPv6</span>
</span><span class='line'><span class="nb">Listen</span> [::]:631
</span><span class='line'><span class="c"># For IPv4</span>
</span><span class='line'><span class="nb">Listen</span> <span class="m">0.0.0.0</span>:631
</span><span class='line'>
</span><span class='line'><span class="c"># Restrict access to the server...</span>
</span><span class='line'><span class="nt">&lt;Location</span> <span class="s">/</span><span class="nt">&gt;</span>
</span><span class='line'>    <span class="nb">Order</span> allow,deny
</span><span class='line'>    <span class="nb">Allow</span> @LOCAL
</span><span class='line'><span class="nt">&lt;/Location&gt;</span>
</span><span class='line'>
</span><span class='line'><span class="nt">&lt;Location</span> <span class="s">/admin</span><span class="nt">&gt;</span>
</span><span class='line'>    <span class="nb">Order</span> allow,deny
</span><span class='line'>    <span class="nb">Allow</span> @LOCAL
</span><span class='line'><span class="nt">&lt;/Location&gt;</span>
</span><span class='line'>
</span><span class='line'><span class="c"># Restrict access to configuration files...</span>
</span><span class='line'><span class="nt">&lt;Location</span> <span class="s">/admin/conf</span><span class="nt">&gt;</span>
</span><span class='line'>    <span class="nb">AuthType</span> Default
</span><span class='line'>    <span class="nb">Require</span> <span class="k">user</span> @SYSTEM
</span><span class='line'>    <span class="nb">Order</span> allow,deny
</span><span class='line'>    <span class="nb">Allow</span> @LOCAL
</span><span class='line'><span class="nt">&lt;/Location&gt;</span>
</span></code></pre></td></tr></table></div></figure>


<p><code>@LOCAL</code> <a href="http://www.cups.org/documentation.php/ref-cupsd-conf.html">expands</a> to the names of all the local interfaces.</p>

<p>The printer is supported by <a href="http://splix.ap2c.org/">SpliX</a> driver:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sudo apt-get install openprinting-ppds printer-driver-splix
</span></code></pre></td></tr></table></div></figure>


<h1>Remote scanner access with saned</h1>

<p>Install <code>sane-utils</code> and edit <code>/etc/sane.d/saned.conf</code>:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="c">## Access list</span>
</span><span class='line'><span class="c"># My local IPv4 network</span>
</span><span class='line'>192.168.1.1/24
</span><span class='line'><span class="c"># My local IPv6 network</span>
</span><span class='line'>2a01:d0:801a::2/64
</span></code></pre></td></tr></table></div></figure>


<p>Enable daemon startup in <code>/etc/default/saned</code>:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">RUN</span><span class="o">=</span>yes
</span></code></pre></td></tr></table></div></figure>


<p>On the client end, the hostname/IP of the server should be added to
<code>/etc/sane.d/net.conf</code></p>

<p>Due to a <a href="https://bugs.launchpad.net/bugs/773617" title="Launchpad Bug #773617: Saned group is missing permissions to access scanners via udev-acl" class="launchpad-bug">773617</a> the <code>saned</code> user is not added to <code>scanner</code> group, fix
this with</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>sudo gpasswd -a saned scanner
</span></code></pre></td></tr></table></div></figure>


<p>And then restart the <code>saned</code> service.</p>

<p>To test whether client can see the saned server, run</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'><span class="nv">$ </span>scanimage -L
</span><span class='line'>device <span class="sb">`</span>net:example.net:xerox_mfp:libusb:001:005<span class="err">&#39;</span> is a SAMSUNG ORION multi-function peripheral
</span></code></pre></td></tr></table></div></figure>


<p>If the client crashes and you haven&#8217;t rebooted after you installed sane-utils,
unplug/plug in the device for udev rules to be reapplied.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[How much is the blog?]]></title>
    <link href="http://rtg.in.ua/blog/2012/03/how-much-is-the-blog/"/>
    <updated>2012-03-07T02:30:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/03/how-much-is-the-blog</id>
    <content type="html"><![CDATA[<p>Today I received my invite to <a href="http://www.hpcloud.com">HPCloud.com</a>, the <a href="http://www.openstack.org">OpenStack</a> installation that
provides <a href="http://aws.amazon.com/ec2/">EC2</a>-like functionality. At the moment the system is in beta testing
mode so I decided to see how much it costs to have a blog running in their
system.</p>

<p>The A record for rtg.in.ua has been temporarily switched over to hpcloud floating
IP, will see how it goes. IPv6 connectivity is not available in HPCloud, however
based on my website stats, there are 3 unique IPv6 addresses. All of them belong to
my subnet, so does not appear to be a great loss.</p>

<p>So far it is only $0.10.</p>

<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/6qsO78zT5znljW8oO4WwXu/hpcloud-blog.png"></p>

<h2>Update</h2>

<p>I am pretty new to the EC2-like pricing for the computing nodes. It turns out
that the <strong>instance-hour</strong> EC2 and RackSpace are using is nothing else than a
regular hour.</p>

<p>It does not matter whether you are using all your CPU allocation or it idles
for days serving static content to your visitors, you will still pay for the
time instance is running.</p>

<p>For such simple site as this blog it is clearly an overkill. Now I understand
why <a href="http://www.jorgecastro.org/">Jorge Castro</a> <a href="http://www.jorgecastro.org/2012/02/12/new-blog-powered-by-octopress-and-aws/">switched</a> to S3 to serve his blog.</p>

<p>For everything else requiring some computation power, such services are really
great. I suppose I will switch the host back to burst.net VPS even though
HPCloud Beta is completely free.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Проездной для киевского метро теперь с картой]]></title>
    <link href="http://rtg.in.ua/blog/2012/02/kiev-metro-travel-card-with-map/"/>
    <updated>2012-02-29T10:46:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/02/kiev-metro-travel-card-with-map</id>
    <content type="html"><![CDATA[<p><img class="center" src="http://rtg.in.ua/assets/ubuntuone/0qkF8uOh92q25nvYpV1j0M/kiev_travel_card.jpg">
Удобно.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Весы Tefal и ошибка 114]]></title>
    <link href="http://rtg.in.ua/blog/2012/02/tefal-bathroom-scales-and-114-error/"/>
    <updated>2012-02-28T22:26:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/02/tefal-bathroom-scales-and-114-error</id>
    <content type="html"><![CDATA[<p><img class="left" src="http://rtg.in.ua/assets/ubuntuone/0HIJre52SCKTyxOG2F3MJO/scales.jpg">
В 2010-м, через год после покупки весов Tefal PP1051, последние начали показывать 114 вместо
веса. Сегодня Оксана окончательно на них обиделась, и я решил узнать, есть ли у
кого-то еще такая проблема. Когда это только началось, я не мог найти никакой
информации о том, как их разобрать и что на чем держится, а многочисленные
попытки обнаружить крепления не увенчались успехом.</p>

<p>Но сегодня быстрый поиск в гугле привел к
<a href="http://fotos.ua/tefal/pp-1051/reviews/">посту Славика</a> на сайте магазина
Fotos (где, кстати, мы их и купили).</p>

<blockquote><p>Сегодня отдали с угрозой не возьмешь - выкинем напольные весы Тефаль (модели
нет, т.к. без стикера, но на картинке точно они:</p>

<p>Проблема на индикаторе веса значок _| при надавливании пальцами на дисплее
что-то по-сегментно отображалось, видимо на него наступили.</p>

<p>Вскрыл, поддев двумя отвертками, разобрал блок индикатора, протер пластиковый
токопроводящий &#8220;шлейф&#8221; между платой и ЖК спиртом, собрал - заработало. (был
перекос платы от механического воздействия).</p>

<p>разберите и протрите все контакты. мне помогло</p></blockquote>

<p>Значит, их все-таки можно разобрать.</p>

<!-- more -->


<p>Надо снять ножки, вставить какой-то плоский предмет в щель и
отогнуть защелку в каждой из ножек.</p>

<p><img src="http://rtg.in.ua/assets/ubuntuone/35UfFnXI2TJUeRNJo922JR/image1.jpg"></p>

<p>Вот знакомые цифры:</p>

<p><img src="http://rtg.in.ua/assets/ubuntuone/3HwiEOC20pDAaRRw3sSU8j/image2.jpg"></p>

<p>Эти контакты я почистил, и все снова заработало!</p>

<p><img src="http://rtg.in.ua/assets/ubuntuone/3IPezvkxuyAQnqynX8yWCu/image3.jpg">
<img src="http://rtg.in.ua/assets/ubuntuone/5jNwY9erPqqkvHvyXlt7Hv/image4.jpg"></p>

<p>При разборке желательно отщелкнуть все крепления, перевернуть весы и снять
верхнюю панель, иначе выпадут пластины с датчиками. В таком случае их нужно
вставить назад схемой вверх.</p>

<p><img src="http://rtg.in.ua/assets/ubuntuone/6qUD8EzqkuXVqOeu6b0Mvr/image5.jpg"></p>

<p>Судя по всему, 114 появляется, если не удается прочитать информацию с
датчиков. К сожалению, об этом нет никакой публичной информации.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Bugs affecting me in Precise]]></title>
    <link href="http://rtg.in.ua/blog/2012/02/bugs-affecting-me/"/>
    <updated>2012-02-27T11:28:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/02/bugs-affecting-me</id>
    <content type="html"><![CDATA[<p>This is the list of the bugs I am watching closely:</p>

<ul>
<li><em>Fixed</em> <a href="https://bugs.launchpad.net/bugs/940803" title="Launchpad Bug #940803: Super+Shift+Right doesn't work" class="launchpad-bug">940803</a> - switching to left/right workspace is not possible.</li>
<li><em>Fixed</em> <a href="https://bugs.launchpad.net/bugs/939521" title="Launchpad Bug #939521: Shortcut overlay appears even if you Super + another key" class="launchpad-bug">939521</a> - annoying help hint when switching the remaining
workspaces.</li>
<li><a href="https://bugs.launchpad.net/bugs/937822" title="Launchpad Bug #937822: [precise] F10 always opens the menu, cannot be overriden (after xkeyboard-config update)" class="launchpad-bug">937822</a> - press F10 in any app and get a menu. I did not know about
this &#8220;feature&#8221; until it broke (should be Shift-F10 only).</li>
<li><em>Fixed</em> <a href="https://bugs.launchpad.net/bugs/934072" title="Launchpad Bug #934072: drop-down menus only partially drawn and sometimes displaced" class="launchpad-bug">934072</a> - looks like intel-specific, menu shadows appear but not
the menu itself.</li>
<li><em>Fixed</em> <a href="https://bugs.launchpad.net/bugs/932906" title="Launchpad Bug #932906: HUD loses keypresses for the first second after opening" class="launchpad-bug">932906</a> - HUD is not receiving keyboard input immediately.</li>
<li><em>Fixed</em> <a href="https://bugs.launchpad.net/bugs/925895" title="Launchpad Bug #925895: Ambiance sub-menus light like Radiance after latest light-themes update." class="launchpad-bug">925895</a> - Menus are light instead of being dark, also looks like
empathy tooltips are also affected.</li>
<li><a href="https://bugs.launchpad.net/bugs/744812" title="Launchpad Bug #744812: FontConfig/Qt stack choke on Ubuntu Medium font meta-data (No medium in Inkscape and too bold in Qt apps)" class="launchpad-bug">744812</a> - Qt applications display labels in bold Ubuntu fonts.</li>
</ul>


<p>And my wife&#8217;s computer has X segfaulting while trying to play the video, <a href="https://bugs.launchpad.net/bugs/941881" title="Launchpad Bug #941881: [precise] Xorg segfault when attempting to play video via Xv" class="launchpad-bug">941881</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[БС на Драгобрате]]></title>
    <link href="http://rtg.in.ua/blog/2012/02/bs-at-draghobrat/"/>
    <updated>2012-02-25T14:34:00+02:00</updated>
    <id>http://rtg.in.ua/blog/2012/02/bs-at-draghobrat</id>
    <content type="html"><![CDATA[<p><a href="http://rtg.in.ua/assets/ubuntuone/0uDYm87WDwKdhjj9wCephY/base_station_dragobrat-1200.jpg"><img class="center" src="http://rtg.in.ua/assets/ubuntuone/5lnim3ugu5nyaxLSpjtNUI/base_station_dragobrat-664.jpg"></a>
Фото сделала <a href="https://plus.google.com/110168703039111277149/">Оксана</a>.</p>
]]></content>
  </entry>
  
</feed>

