Friday, March 31, 2017
Thursday, March 30, 2017
Wednesday, March 29, 2017
Tuesday, March 28, 2017
Docker From the Ground Up: Building Images
Docker containers are on the rise as a best practice for deploying and managing cloud-native distributed systems. Containers are instances of Docker images. It turns out that there is a lot to know and understand about images.
In this two-part tutorial, I'm covering Docker images in depth. In part one I discussed the basic principles, design considerations, and inspecting image internals. In this part, I cover building your own images, troubleshooting, and working with image repositories.
When you come out on the other side, you'll have a solid understanding of what Docker images are exactly and how to utilize them effectively in your own applications and systems.
Building Images
There are two ways to build images. You can modify and existing container and then commit it as a new image, or you can write a Dockerfile and build it to an image. We'll go over both and explain the pros and cons.
Manual Builds
With manual builds, you treat your container like a regular computer. You install packages, you write files, and when it's all said and done, you commit it and end up with a new image that you use as a template to create many more identical containers or even base other images on.
Let's start with the alpine image, which is a very small and spartan image based on Alpine Linux. We can run it in interactive mode to get into a shell. Our goal is to add a file called "yeah" that contains the text "it works!" to the root directory and then create a new image from it called "yeah-alpine".
Here we go. Nice, we're already in the root dir. Let's see what's there.
> docker run -it alpine /bin/sh / # ls bin dev etc home lib linuxrc media mnt proc root run sbin srv sys tmp usr var
What editor is available? No vim, no nano?
/ # vim /bin/sh: vim: not found / # nano /bin/sh: nano: not found
Oh, well. We just want to create a file:
/ # echo "it works!" > yeah / # cat yeah it works!
I exited from the interactive shell, and I can see the container named "vibrant_spenc" with docker ps --all. The --all flag is important because the container is not running anymore.
> docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS NAMES c8faeb05de5f alpine "/bin/sh" 6 minutes ago Exited vibrant_spence
Here, I create a new image from the "vibrate_spence" container. I added the commit message "mine, mine, mine" for good measure.
> docker commit -m "mine, mine, mine" vibrant_spence yeah-alpine sha256:e3c98cd21f4d85a1428...e220da99995fd8bf6b49aa
Let's check it out. Yep, there is a new image, and in its history you can see a new layer with the "mine, mine, mine" comment.
> docker images REPOSITORY TAG IMAGE ID SIZE yeah-alpine latest e3c98cd21f4d 4.8 MB python latest 775dae9b960e 687 MB d4w/nsenter latest 9e4f13a0901e 83.8 kB ubuntu-with-ssh latest 87391dca396d 221 MB ubuntu latest bd3d4369aebc 127 MB hello-world latest c54a2cc56cbb 1.85 kB alpine latest 4e38e38c8ce0 4.8 MB nsqio/nsq latest 2a82c70fe5e3 70.7 MB > docker history yeah-alpine IMAGE CREATED SIZE COMMENT e3c98cd21f4d 40 seconds ago 66 B mine, mine, mine 4e38e38c8ce0 7 months ago 4.8 MB
Now for the real test. Let's delete the container and create a new container from the image. The expected result is that the "yeah" file will be present in the new container.
> docker rm vibrant_spence vibrant_spence > docker run -it yeah-alpine /bin/sh / # cat yeah it works! / #
What can I say? Yeah, it works!
Using a Dockerfile
Creating images out of modified containers is cool, but there is no accountability. It's hard to keep track of the changes and know what the specific modifications were. The disciplined way to create images is to build them using a Dockerfile.
The Dockerfile is a text file that is similar to a shell script, but it supports several commands. Every command that modifies the file system creates a new layer. In part one we discussed the importance of dividing your image into layers properly. The Dockerfile is a big topic in and of itself.
Here, I'll just demonstrate a couple of commands to create another image, "oh-yeah-alpine", based on a Dockerfile. In addition to creating the infamous "yeah" file, let's also install vim. The alpine Linux distribution uses a package management system called "apk". Here is the Dockerfile:
FROM alpine # Copy the "yeah" file from the host COPY yeah /yeah # Update and install vim using apk RUN apk update && apk add vim CMD cat /yeah
The base image is alpine. It copies the "yeah" file from the same host directory where the Dockerfile is (the build context path). Then, it runs apk update and installs vim. Finally, it sets the command that is executed when the container runs. In this case it will print to the screen the content of the "yeah" file.
OK. Now that we know what we're getting into, let's build this thing. The "-t" option sets the repository. I didn't specify a tag, so it will be the default "latest".
> docker build -t oh-yeah-alpine . Sending build context to Docker daemon 3.072 kB Step 1/4 : FROM alpine ---> 4e38e38c8ce0 Step 2/4 : COPY yeah /yeah ---> 1b2a228cc2a5 Removing intermediate container a6221f725845 Step 3/4 : RUN apk update && apk add vim ---> Running in e2c0524bd792 fetch http://ift.tt/2o6tJMc fetch http://dl-cdn.alpinelinux.org.../x86_64/APKINDEX.tar.gz v3.4.6-60-gc61f5bf [http://ift.tt/1tcYqhs] v3.4.6-33-g38ef2d2 [http://ift.tt/2ncJ1u6] OK: 5977 distinct packages available (1/5) Installing lua5.2-libs (5.2.4-r2) (2/5) Installing ncurses-terminfo-base (6.0-r7) (3/5) Installing ncurses-terminfo (6.0-r7) (4/5) Installing ncurses-libs (6.0-r7) (5/5) Installing vim (7.4.1831-r2) Executing busybox-1.24.2-r9.trigger OK: 37 MiB in 16 packages ---> 7fa4cba6d14f Removing intermediate container e2c0524bd792 Step 4/4 : CMD cat /yeah ---> Running in 351b4f1c1eb1 ---> e124405f28f4 Removing intermediate container 351b4f1c1eb1 Successfully built e124405f28f4
Looks good. Let's verify the image was created:
> docker images | grep oh-yeah oh-yeah-alpine latest e124405f28f4 About a minute ago 30.5 MB
Note how installing vim and its dependencies bloated the size of the container from the 4.8MB of the base alpine image to a massive 30.5MB!
It's all very nice. But does it work?
> docker run oh-yeah-alpine it works!
Oh yeah, it works!
In case you're still suspicious, let's go into the container and examine the "yeah" file with our freshly installed vim.
> docker run -it oh-yeah-alpine /bin/sh / # vim yeah it works! ~ ~ . . . ~ "yeah" 1L, 10C
The Build Context and the .dockerignore file
I didn't tell you, but originally when I tried to build the oh-yeah-alpine image, it just hung for several minutes. The issue was that I just put the Dockerfile in my home directory. When Docker builds an image, it first packs the whole directory where the Dockerfile is (including sub-directories) and makes it available for COPY commands in the Dockerfile.
Docker is not trying to be smart and analyze your COPY commands. It just packs the whole thing. Note that the build content will not end in your image, but it will slow down your build command if your build context is unnecessarily large.
In this case, I simply copied the Dockerfile and the "yeah" into a sub-directory and ran the docker build command in that sub-directory. But sometimes you have a complicated directory tree from which you want to copy specific sub-directories and files and ignore others. Enter the .dockerignore file.
This file lets you control exactly what goes into the build context. My favorite trick is to first exclude everything and then start including the bits and pieces I need. For example, in this case I could create the following .dockerignore file and keep the Docker file and the "yeah" in my home directory:
# Exclude EVERYTHING first * # Now selectively include stuff !yeah
There is no need to include the "Dockerfile" itself or the ".dockerignore" file in the build context.
Copying vs. Mounting
Copying files into the image is sometimes what you need, but in other cases you may want your containers to be more dynamic and work with files on the host. This is where volumes and mounts come into play.
Mounting host directories is a different ball game. The data is owned by the host and not by the container. The data can be modified when the container is stopped. The same container can be started with different host directories mounted.
Tagging Images
Tagging images is very important if you develop a microservices-based system and you generate a lot of images that must be sometimes associated with each other. You can add as many tags as you want to an image.
You've already seen the default "latest" tag. Sometimes, it makes sense to add other tags, like "tested", "release-1.4", or the git commit that corresponds to the image.
You can tag an image during a build or later. Here's how to add a tag to an existing image. Note that while it's called a tag, you can also assign a new repository.
> docker tag oh-yeah-alpine oh-yeah-alpine:cool-tag > docker tag oh-yeah-alpine oh-yeah-alpine-2 > docker images | grep oh-yeah oh-yeah-alpine-2 latest e124405f28f4 30.5 MB oh-yeah-alpine cool-tag e124405f28f4 30.5 MB oh-yeah-alpine latest e124405f28f4 30.5 MB
You can also untag by removing an image by its tag name. This is a little scary because if you remove the last tag by accident, you lose the image. But if you build images from a Dockerfile, you can just rebuild the image.
> docker rmi oh-yeah-alpine-2 Untagged: oh-yeah-alpine-2:latest > docker rmi oh-yeah-alpine:cool-tag Untagged: oh-yeah-alpine:cool-tag
If I try to remove the last remaining tagged image, I get an error because it is used by a container.
> docker rmi oh-yeah-alpine Error response from daemon: conflict: unable to remove repository reference "oh-yeah-alpine" (must force) - container a1443a7ca9d2 is using its referenced image e124405f28f4
But if I remove the container...
> docker rmi oh-yeah-alpine Untagged: oh-yeah-alpine:latest Deleted: sha256:e124405f28f48e...441d774d9413139e22386c4820df Deleted: sha256:7fa4cba6d14fdf...d8940e6c50d30a157483de06fc59 Deleted: sha256:283d461dadfa6c...dbff864c6557af23bc5aff9d66de Deleted: sha256:1b2a228cc2a5b4...23c80a41a41da4ff92fcac95101e Deleted: sha256:fe5fe2290c63a0...8af394bb4bf15841661f71c71e9a > docker images | grep oh-yeah
Yep. It's gone. But don't worry. We can rebuild it:
> docker build -t oh-yeah-alpine . > docker images | grep oh-yeah oh-yeah-alpine latest 1e831ce8afe1 1 minutes ago 30.5 MB
Yay, it's back. Dockerfile for the win!
Working With Image Registries
Images are very similar in some respects to git repositories. They are also built from an ordered set of commits. You can think of two images that use the same base images as branches (although there is no merging or rebasing in Docker). An image registry is the equivalent of a central git hosting service like GitHub. Guess what's the name of the official Docker image registry? That's right, Docker Hub.
Pulling Images
When you run an image, if it doesn't exist, Docker will try to pull it from one of your configured image registries. By default it goes to Docker Hub, but you can control it in your "~/.docker/config.json" file. If you use a different registry, you can follow their instructions, which typically involve logging in using their credentials.
Let's delete the "hello-world" image and pull it again using the docker pull command.
> dockere images | grep hello-world hello-world latest c54a2cc56cbb 7 months ago 1.85 kB > docker rmi hello-world hello-world
It's gone. Let's pull now.
> docker pull hello-world Using default tag: latest latest: Pulling from library/hello-world 78445dd45222: Pull complete Digest: sha256:c5515758d4c5e1e...07e6f927b07d05f6d12a1ac8d7 Status: Downloaded newer image for hello-world:latest > dockere images | grep hello-world hello-world latest 48b5124b2768 2 weeks ago 1.84 kB
The latest hello-world was replaced with a newer version.
Pushing Images
Pushing images is a little more involved. First you need to create an account on Docker Hub (or other registry). Next, you log in. Then you need to tag the image you want to push according to your account name ("g1g1" in my case).
> docker login -u g1g1 -p <password> Login Succeeded > docker tag hello-world g1g1/hello-world > docker images | grep hello g1g1/hello-world latest 48b5124b2768 2 weeks ago 1.84 kB hello-world latest 48b5124b2768 2 weeks ago 1.84 kB
Now, I can push the g1g1/hello-world tagged image.
> docker push g1g1/hello-world The push refers to a repository [http://ift.tt/2o6AGwG] 98c944e98de8: Mounted from library/hello-world latest: digest: sha256:c5515758d4c5e...f6d12a1ac8d7 size: 524
Conclusion
Docker images are the templates to your containers. They are designed to be efficient and offer maximum reuse by using a layering file system storage driver.
Docker provides a lot of tools for listing, inspecting, building and tagging images. You can pull and push images to image registries like Docker Hub to easily manage and share your images.
Monday, March 27, 2017
PyQuery: Python's jQuery
In this tutorial, you'll have a look at PyQuery, a Python library which allows you to make jQuery queries on XML documents. Syntactically it's quite similar to jQuery, and if you are familiar with jQuery it should be easier to follow.
Getting Started With PyQuery
To get started with PyQuery, install the Python package using PIP.
pip install pyquery
Once you have installed PyQuery, import it into the Python program.
from pyquery import PyQuery as pq
Let's start with a basic example of how it works. Consider the following HTML:
<div id="divMain">
<span>
Hello World
</span>
</div>
Pass the input XML to the PyQuery object and you should be able to apply jQuery style queries to it.
divMain = pq("<div id='divMain'><span>Hello World</span></div>")
Assume divMain as a jQuery object and print the HTML content of the div.
print divMain('#divMain').html()
Save the changes and try running the program. You should have the following output:
<span>Hello World</span>
To access the Hello World text from the inside the span, the code would be:
print divMain('#divMain').text()
Save the changes and try running the code. You should have the following output:
Hello World
Attributes Manipulation Using PyQuery
Now let's have a look at how to read the attributes using the PyQuery library. Assume that you have an HTML element as shown:
<ul id="ulMain">
<li>Roy</li>
<li>Hari</li>
</ul>
Use the PyQuery library to read the above HTML content.
ulMain = pq("<ul id='ulMain'><li>Roy</li><li>Hari</li></ul>")
Let's try to access the ID attribute of the ulMain ul.
print ulMain.attr('id')
Save the above changes and try running the Python program. You should have ID ulMain printed on the terminal.
You saw how to access the ID attribute of the ul ulMain. Let's try to set a class name for the same ulMain element. To specify a class name attribute, you need to specify the attribute name and its corresponding value as shown:
ulMain.attr('class','testclass')
Print the ulMain object and you should have the element with the class attribute.
<ul id="ulMain" class="testclass">
<li>Roy</li>
<li>Hari</li>
</ul>
You can also add and remove classes directly without using the attr method. To add a class to the element, you can make use of the method addClass.
ulMain.addClass('test')
To remove an existing class from the element, you can make use of the removeClass method.
ulMain.removeClass('test')
Handling CSS Using PyQuery
Apart from attributes, the element would have some CSS properties. To add or modify the CSS properties, you can use the css method and specify the properties. Let's say that you want to specify the height in the style of the ul ulMain. The following code would add the required style properties to the element.
ulMain.css('height','100px')
Save the above changes and try executing the Python program. You should have the ul ulMain printed along with the new style.
<ul id="ulMain" style="height: 100px">
<li>Roy</li>
<li>Hari</li>
</ul>
To add multiple styles to the element, you can specify them as shown:
ulMain.css({'height':'100px','width':'100px'})
Run the program and you should have the styles added to the ulMain.
<ul id="ulMain" style="width: 100px; height: 100px">
<li>Roy</li>
<li>Hari</li>
</ul>
Creating & Appending Elements
During dynamic element creation, you are required to create new elements and append them to the existing parent element where they'll be rendered. Let's have a look at how to create and append elements using PyQuery.
Assume you have a main container div called divMain.
divMain = pq("<div id='divMain'></div>")
Let's create a new span element using PyQuery.
span = pq('<span>Hello World</span>')
Add some CSS properties to the span.
span.css({'color':'blue','font-weight':'bold'})
PyQuery provides a method to add elements to existing elements. You can use the append method to append the span element to the div divMain. Here is the code:
divMain.append(span) print divMain
Save the changes and run the program. You should be able to see the divMain printed with the newly created span appended to it.
<div id="divMain">
<span style="color: blue; font-weight: bold">
Hello World
</span>
</div>
You used the append method to append the span to the div. You have another alternative method called appendTo which would append the nodes to value. In the above case you can use the method like so:
span.appendTo(divMain)
Finding Elements Using PyQuery
PyQuery provides methods to find children, the next elements, the closest elements, etc. It also provides methods to filter and find elements inside a particular node.
Assume that you have a particular piece of HTML as shown:
<div id="divMain">
<div id="content">
<ul>
<li>Jade</li>
<li>Jim</li>
</ul>
</div>
<div id="list">
<span>Hello World</span>
</div>
</div>
Add the following HTML to the PyQuery object:
divMain = pq("<div id='divMain'>"+
"<div id='content'>"+
"<ul>"+
"<li>Jade</li>"+
"<li>Jim</li>"+
"</ul>"
"</div>"+
"<div id='list'>"+
"<span>Hello World</span>"
"</div>"+
"</div>")
Let's use PyQuery to find the children of the div divMain. Add the following line of code to print the children of divMain.
print divMain.children()
On running the program, you should have the following output:
<div id="content">
<ul>
<li>Jade</li>
<li>Jim</li>
</ul>
</div>
<div id="list"><span>Hello World</span></div>
To find the closest element to an element, you can use the method closest. To find the closest div element to the span, the code would be:
print divMain('span').closest('div')
The above command would return the following output:
<div id="list"><span>Hello World</span></div>
A find method is also available to find the elements. For example, to find a span inside the divMain, you need to call the method as shown:
print divMain.find('span')
The above command would return the span.
<span>Hello World</span>
Insert an Element Inside HTML
While append does the work of adding elements to the existing elements, sometimes you need to insert elements after or before certain elements. PyQuery provides a method to insert elements before or after other elements.
Let's define a new paragraph element to be inserted after the span inside the divMain div.
p = pq('<p>Welcome</p>')
Call the insertAfter method on the paragraph p element to insert it after the span.
p.insertAfter(divMain('span'))
print divMain
Save the above changes and run the Python program. You should have the following output:
<div id="divMain">
<div id="content">
<ul>
<li>Jade</li>
<li>Jim</li>
</ul>
</div>
<div id="list">
<span>Hello World</span>
<p>Welcome</p>
</div>
</div>
Similarly, you have the insertBefore method, which inserts before the element. Modify the code as shown below to use the insertBefore method:
p.insertBefore(divMain('span'))
print divMain
Save the changes and run the program. You should be able to see the following output on the terminal:
<div id="divMain">
<div id="content">
<ul>
<li>Jade</li>
<li>Jim</li>
</ul>
</div>
<div id="list">
<p>Welcome</p>
<span>Hello World</span>
</div>
</div>
Wrapping It Up
In this tutorial, you saw how to get started with PyQuery, a Python library which allows you to make jQuery queries on XML documents. You saw how to manipulate the attributes and CSS styles of the HTML elements.
You learnt how to create and append elements to existing elements and insert new elements before and after elements. What you saw in this tutorial is just the tip of the iceberg, and there is a lot more that this library has to offer.
For more detailed information on using this library, I would recommend reading the official documentation. Do let us know your suggestions in the comments below.
Sunday, March 26, 2017
Saturday, March 25, 2017
Effective Chord Changes on Guitar—Part 2
A good magician never lets the mechanics of a trick obscure an illusion. As for guitar playing, an audience should never hear how you’re producing sounds. In this context that means working towards seamless chord changes.
This is the second in a two-part tutorial and here are some more concepts and ideas for you to consider.
- GuitarEffective Chord Changes on Guitar—Part 1
Finger Trickery
Most people favour the index or middle finger, and with good reason, as they are stronger than the others. This is due to the arrangement and distribution of tendons in the hand.
Consequently, we tend to think our fingers operate best from index (one) to little (four), but this isn't the case.
To illustrate my point, hold your hand up in front of you, then slowly close your fingers into your palm. Rather than 1-2-3-4, as you might expect, they close 4-3-2-1. This is the natural order of your fingers.
If you're unsure, try closing them 1-2-3-4; you'll experience resistance, and it’ll feel unnatural.
This leads onto my next point.
Direction of Play
Consider the number of basic open chord shapes that use your index finger for the lowest fretted note. I can think of two—A and Em.
If you're still starting a change with the index finger, therefore, the chances are the chord constructions are occurring against the order of the strings. This will inevitably slow you down, as you wait for the chord to be built before striking the strings.
You should lead with the finger that you need first, which isn't always necessarily your first (index) finger.
In an Instant
Whilst some shapes may become formed almost instantly upon arrival, others by definition take longer to construct. Even if you've overcome the issue discussed in the previous point you may still experience gaps in your own playing.
This is because beginners don't commit to playing until the entire shape has been built. More experienced players recognise a fundamental truth:
A string only needs to be fretted just before it’s struck
Knowing this means that you can play a chord on a rolling basis; as long as each note is fretted just ahead of the string being played, the desired sound will be achieved.
Furthermore, as you're no longer waiting for the whole shape before playing gaps become almost non-existent.
Don't Let Go
Another source of gaps can be taking too many steps when executing a change. The greatest of these is completely letting go of the strings.
This can sometimes be inevitable, dependent on the requirements of the change, but otherwise should be avoided.
Taking the fingers away from the strings is easy, but having to rearrange all of them back down again takes time. Every time this occurs you risk creating extraneous string noise.
Correcting it not only speeds up the changes but cleans up the sound.
Shared Notes
Not all shapes need dismantling when a change is required. This is especially true when two chords share one or more fretted notes.
Here's an example: the chords of C major and A minor. On the face of it they’re different types of chord. Both, however, feature the same two notes: C and E.
When played in open positions, E is on the second fret of the D string, and C is on the first fret of the B string. These notes are fretted in the same places using the same fingers in both shapes.
Consider why you would you lift the fingers from the strings when moving from one to the other. It makes no sense. Beginners, however, often do so considering it to be a new shape.
When looking at a chord progression, try to identify any shared fretted notes between chords. As well as improving the changes, it’ll increase your understanding of which notes your chords contain; this in turn enhances your knowledge of the fretboard, which is useful for all aspects of playing guitar.
Overcoming Differences
Shared fretted notes provide a pivot point. By this I mean you can lean on them to pivot the fingers, or indeed, entire hand, in order to move from one shape to the next.
Of course, not every chord change features such notes. You still don’t have to let go of the strings. If there aren't any pivot points you'll need to create one.
For example, the change from G major to C major in the open positions. Whilst both feature the note of G, it's only fretted during the G chord (being an open string for C major) and thus is of no use to us in this instance.
When changing, lift the third finger away from its position in the G chord, and place it on the third fret of the A string; you're now on the root note of C major. If you've done this correctly, you should feel the third finger butt up against the back of the second finger.
Press down on the third finger, bringing your weight to bear. Keeping it planted, you can now swivel your hand into position for the C chord. If you're unsure as to what I mean, look at the thumb when you fret the G chord; it’s on the bass side of the neck.
When you fret the C chord it moves towards the treble side which involves swivelling the hand into a new orientation.
It takes a little getting used to, but it's cleaner and quicker than letting go, plus it ensures you fret the chord in the direction of play.
Conclusion
In this tutorial I've shown you the following:
- The fingers work back-to-front
- Make changes in the direction of play
- A string needs to be fretted only just before it’s struck
- Try not to let go
- Notes shared between shapes speed up changes
- If they aren’t shared notes, create a pivot point note
Implementing these ideas will take time but by practicing they’ll make you a better player.
Friday, March 24, 2017
Docker From the Ground Up: Understanding Images
Docker containers are on the rise as a best practice for deploying and managing cloud-native distributed systems. Containers are instances of Docker images. It turns out that there is a lot to know and understand about images.
In this two-part tutorial I'll cover Docker images in depth. In this part, I'll start with the basic principles, and then I'll move on to design consideration and inspecting image internals. In part two, I'll cover building your own images, troubleshooting, and working with image repositories.
When you come out on the other side, you'll have a solid understanding of what Docker images are exactly and how to utilize them effectively in your own applications and systems.
Understanding Layers
Docker manages images using a back-end storage driver. There are several supported drivers such as AUFS, BTRFS, and overlays. Images are made of ordered layers. You can think of a layer as a set of file system changes. When you take all the layers and stack them together, you get a new image that contains all the accumulated changes.
The ordered part is important. If you add a file in one layer and remove it in another layer, you'd better do it in the right order. Docker keeps track of each layer. An image can be composed of dozens of layers (the current limit is 127). Each layer is very lightweight. The benefit of layers is that images can share layers.
If you have lots of images based on similar layers, like base OS or common packages, then all these common layers will be stored just once, and the overhead per image will be just the unique layers of that image.
Copy on Write
When a new container is created from an image, all image layers are read-only and a thin read-write layer is added on top. All the changes made to the specific container are stored in that layer.
Now, it doesn't mean that the container can't modify files from its image layer. It definitely can. But it will create a copy in its top layer, and from that point on, anyone who tries to access the file will get the top layer copy. When files or directories are removed from lower layers, they become hidden. The original image layers are identified by a cryptographic content-based hash. The container's read-write layer is identified by a UUID.
This allows a copy-on-write strategy for both images and containers. Docker reuses the same items as much as possible. Only when an item is modified will Docker create a new copy.
Design Considerations for Docker Images
The unique organization in layers and the copy-on-write strategy promotes some best practices for creating and compositing Docker images.
Minimal Images: Less Is More
Docker images get enormous benefits from the point of view of stability, security and loading time the smaller they are. You can create really tiny images for production purposes. If you need to troubleshoot, you can always install tools in a container.
If you write your data, logs and everything else only to mounted volumes then you can use your entire arsenal of debugging and troubleshooting tools on the host. We'll see soon how to control very carefully what files go into your Docker image.
Combine Layers
Layers are great, but there is a limit, and there is overhead associated with layers. Too many layers could hurt file system access inside the container (because every layer may have added or removed a file or directory), and they just clutter your own file system.
For example, if you install a bunch of packages, you can have a layer for each package, by installing each package in a separate RUN command in your Dockerfile:
RUN apt-get update RUN apt-get -y install package_1 RUN apt-get -y install package_2 RUN apt-get -y install package_3
Or you can combine them into one layer with a single RUN command.
RUN apt-get update && \
apt-get -y install package_1 && \
apt-get -y install package_2 && \
apt-get -y install package_3
Choosing a Base Image
Your base image (practically nobody builds images from scratch) is often a major decision. It may contain many layers and add a lot of capabilities, but also a lot of weight. The quality of the image and the author are also critical. You don't want to base your images on some flaky image where you're not sure exactly what's in there and if you can trust the author.
There are official images for many distributions, programming languages, databases, and runtime environments. Sometimes the options are overwhelming. Take your time and make a wise choice.
Inspecting Images
Let's look at some images. Here is a listing of the images currently available on my machine:
REPOSITORY TAG IMAGE ID CREATED SIZE python latest 775dae9b960e 12 days ago 687 MB d4w/nsenter latest 9e4f13a0901e 4 months ago 83.8 kB ubuntu-with-ssh latest 87391dca396d 4 months ago 221 MB ubuntu latest bd3d4369aebc 5 months ago 127 MB hello-world latest c54a2cc56cbb 7 months ago 1.85 kB alpine latest 4e38e38c8ce0 7 months ago 4.8 MB nsqio/nsq latest 2a82c70fe5e3 8 months ago 70.7 MB
The repository and the tag identify the image for humans. If you just try to run or pull using a repository name without specifying the tag, then the "latest" tag is used by default. The image ID is a unique identifier.
Let's dive in and inspect the hello-world image:
> docker inspect hello-world
[
{
"Id": "sha256:c54a2cc56cbb2f...e7e2720f70976c4b75237dc",
"RepoTags": [
"hello-world:latest"
],
"RepoDigests": [
"hello-world@sha256:0256e8a3...411de4cdcf9431a1feb60fd9"
],
"Parent": "",
"Comment": "",
"Created": "2016-07-01T19:39:27.532838486Z",
"Container": "562cadb4d17bbf30b58a...bf637f1d2d7f8afbef666",
"ContainerConfig": {
"Hostname": "c65bc554a4b7",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) CMD [\"/hello\"]"
],
"Image": "sha256:0f9bb7da10de694...5ab0fe537ce1cd831e",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"DockerVersion": "1.10.3",
"Author": "",
"Config": {
"Hostname": "c65bc554a4b7",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/hello"
],
"Image": "sha256:0f9bb7da10de694b...b0fe537ce1cd831e",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"Architecture": "amd64",
"Os": "linux",
"Size": 1848,
"VirtualSize": 1848,
"GraphDriver": {
"Name": "aufs",
"Data": null
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:a02596fdd012f22b03a...079c3e8cebceb4262d7"
]
}
}
]
It's interesting to see how much information is associated with each image. I will not go over each item. I'll just mention an interesting tidbit that the "container" and "containerConfig" entries are for a temporary container that Docker creates when it builds the image. Here, I want to focus on the last section of "RootFS". You can get just this part using the Go templating support of the inspect command:
> docker inspect -f '' hello-world
{layers [sha256:a02596fdd012f22b03af6a...8357b079c3e8cebceb4262d7] }
It works, but we lost the nice formatting. I prefer to use jq:
> docker inspect hello-world | jq .[0].RootFS
{
"Type": "layers",
"Layers": [
"sha256:a02596fdd012f22b03af6a...7507558357b079c3e8cebceb4262d7"
]
}
You can see that the type is "Layers", and there is just one layer.
Let's inspect the layers of the Python image:
> docker inspect python | jq .[0].RootFS
{
"Type": "layers",
"Layers": [
"sha256:a2ae92ffcd29f7ede...e681696874932db7aee2c",
"sha256:0eb22bfb707db44a8...8f04be511aba313bdc090",
"sha256:30339f20ced009fc3...6b2a44b0d39098e2e2c40",
"sha256:f55f65539fab084d4...52932c7d4924c9bfa6c9e",
"sha256:311f330fa783aef35...e8283e06fc1975a47002d",
"sha256:f250d46b2c81bf76c...365f67bdb014e98698823",
"sha256:1d3d54954c0941a8f...8992c3363197536aa291a"
]
}
Wow. Seven layers. But what are those layers? We can use the history command to figure that out:
IMAGE CREATED CREATED BY SIZE
775dae9b960e 12 days ago /bin/sh -c #(nop) CMD ["python3"] 0 B
<missing> 12 days ago /bin/sh -c cd /usr/local/bin && { ... 48 B
<missing> 12 days ago /bin/sh -c set -ex && buildDeps=' ... 66.9 MB
<missing> 12 days ago /bin/sh -c #(nop) ENV PYTHON_PIP_V... 0 B
<missing> 12 days ago /bin/sh -c #(nop) ENV PYTHON_VERSI... 0 B
<missing> 12 days ago /bin/sh -c #(nop) ENV GPG_KEY=0D96... 0 B
<missing> 12 days ago /bin/sh -c apt-get update && apt-ge... 7.75 MB
<missing> 12 days ago /bin/sh -c #(nop) ENV LANG=C.UTF-8 0 B
<missing> 12 days ago /bin/sh -c #(nop) ENV PATH=/usr/lo... 0 B
<missing> 13 days ago /bin/sh -c apt-get update && apt-ge... 323 MB
<missing> 13 days ago /bin/sh -c apt-get update && apt-ge... 123 MB
<missing> 13 days ago /bin/sh -c apt-get update && apt-ge... 44.3 MB
<missing> 13 days ago /bin/sh -c #(nop) CMD ["/bin/bash"... 0 B
<missing> 13 days ago /bin/sh -c #(nop) ADD file:89ecb642... 123 MB
OK. Don't be alarmed. Nothing is missing. This is just a terrible user interface. The layers used to have an image ID before Docker 1.10, but not anymore. The ID of the top layer is not really the ID of that layer. It is the ID of the Python image. The "CREATED BY" is truncated, but you can see the full command if you pass --no-trunc. I'll save you from the output here because of page-width limitations that will require extreme line wrapping.
How do you get images? There are three ways:
- Pull/Run
- Load
- Build
When you run a container, you specify its image. If the image doesn't exist on your system, it is being pulled from a Docker registry (by default DockerHub). Alternatively, you can pull directly without running the container.
You can also load an image that someone sent you as a tar file. Docker supports it natively.
Finally, and most interestingly, you can build your own images, which is the topic of part two.
Conclusion
Docker images are based on a layered file system that offers many advantages and benefits for the use cases that containers are designed for, such as being lightweight and sharing common parts so many containers can be deployed and run on the same machine economically.
But there are some gotchas, and you need to understand the principles and mechanisms to utilize Docker images effectively. Docker provides several commands to get a sense of what images are available and how they are structured.
Thursday, March 23, 2017
Uploading Files With Rails and Dragonfly
Some time ago I wrote an article Uploading Files With Rails and Shrine that explained how to introduce a file uploading feature into your Rails application with the help of the Shrine gem. There are, however, a bunch of similar solutions available, and one of my favorites is Dragonfly—an easy-to-use uploading solution for Rails and Rack created by Mark Evans.
We covered this library early last year but, as with most software, it helps to take a look at libraries from time to time to see what's changed and how we can employ it in our application.
In this article I will guide you through the setup of Dragonfly and explain how to utilize its main features. You will learn how to:
- Integrate Dragonfly into your application
- Configure models to work with Dragonfly
- Introduce a basic uploading mechanism
- Introduce validations
- Generate image thumbnails
- Perform file processing
- Store metadata for uploaded files
- Prepare an application for deployment
To make things more interesting, we are going to create a small musical application. It will present albums and associated songs that can be managed and played back on the website.
The source code for this article is available at GitHub. You can also check out the working demo of the application.
Listing and Managing Albums
To start off, create a new Rails application without the default testing suite:
rails new UploadingWithDragonfly -T
For this article I will be using Rails 5, but most of the described concepts apply to older versions as well.
Creating the Model, Controller, and Routes
Our small musical site is going to contain two models: Album and Song. For now, let's create the first one with the following fields:
title(string)—contains the album's titlesinger(string)—album's performerimage_uid(string)—a special field to store the album's preview image. This field can be named anything you like, but it must contain the_uidsuffix as instructed by the Dragonfly documentation.
Create and apply the corresponding migration:
rails g model Album title:string singer:string image_uid:string rails db:migrate
Now let's create a very generic controller to manage albums with all the default actions:
albums_controller.rb
class AlbumsController < ApplicationController
def index
@albums = Album.all
end
def show
@album = Album.find(params[:id])
end
def new
@album = Album.new
end
def create
@album = Album.new(album_params)
if @album.save
flash[:success] = 'Album added!'
redirect_to albums_path
else
render :new
end
end
def edit
@album = Album.find(params[:id])
end
def update
@album = Album.find(params[:id])
if @album.update_attributes(album_params)
flash[:success] = 'Album updated!'
redirect_to albums_path
else
render :edit
end
end
def destroy
@album = Album.find(params[:id])
@album.destroy
flash[:success] = 'Album removed!'
redirect_to albums_path
end
private
def album_params
params.require(:album).permit(:title, :singer)
end
end
Lastly, add the routes:
config/routes.rb
resources :albums
Integrating Dragonfly
It's time for Dragonfly to step into the limelight. First, add the gem into the Gemfile:
Gemfile
gem 'dragonfly'
Run:
bundle install rails generate dragonfly
The latter command will create an initializer named dragonfly.rb with the default configuration. We will put it aside for now, but you may read about various options at Dragonfly's official website.
The next important thing to do is equip our model with Dragonfly's methods. This is done by using the dragonfly_accessor:
models/album.rb
dragonfly_accessor :image
Note that here I am saying :image—it directly relates to the image_uid column that we created in the previous section. If you, for example, named your column photo_uid, then the dragonfly_accessor method would need to receive :photo as an argument.
If you are using Rails 4 or 5, another important step is to mark the :image field (not :image_uid!) as permitted in the controller:
albums_controller.rb
params.require(:album).permit(:title, :singer, :image)
This is pretty much it—we are ready to create views and start uploading our files!
Creating Views
Start off with the index view:
views/albums/index.html.erb
<h1>Albums</h1> <%= link_to 'Add', new_album_path %> <ul> <%= render @albums %> </ul>
Now the partial:
views/albums/_album.html.erb
<li>
<%= image_tag(album.image.url, alt: album.title) if album.image_stored? %>
<%= link_to album.title, album_path(album) %> by
<%= album.singer %>
| <%= link_to 'Edit', edit_album_path(album) %>
| <%= link_to 'Remove', album_path(album), method: :delete, data: {confirm: 'Are you sure?'} %>
</li>
There are two Dragonfly methods to note here:
album.image.urlreturns the path to the image.album.image_stored?says whether the record has an uploaded file in place.
Now add the new and edit pages:
views/albums/new.html.erb
<h1>Add album</h1> <%= render 'form' %>
views/albums/edit.html.erb
<h1>Edit <%= @album.title %></h1> <%= render 'form' %>
views/albums/_form.html.erb
<%= form_for @album do |f| %>
<div>
<%= f.label :title %>
<%= f.text_field :title %>
</div>
<div>
<%= f.label :singer %>
<%= f.text_field :singer %>
</div>
<div>
<%= f.label :image %>
<%= f.file_field :image %>
</div>
<%= f.submit %>
<% end %>
The form is nothing fancy, but once again note that we are saying :image, not :image_uid, when rendering the file input.
Now you may boot the server and test the uploading feature!
Removing Images
So the users are able to create and edit albums, but there is a problem: they have no way to remove an image, only to replace it with another one. Luckily, this is very easy to fix by introducing a "remove image" checkbox:
views/albums/_form.html.erb
<% if @album.image_thumb_stored? %>
<%= image_tag(@album.image.url, alt: @album.title) %>
<%= f.label :remove_image %>
<%= f.check_box :remove_image %>
<% end %>
If the album has an associated image, we display it and render a checkbox. If this checkbox is set, the image will be removed. Note that if your field is named photo_uid, then the corresponding method to remove attachment will be remove_photo. Simple, isn't it?
The only other thing to do is permit the remove_image attribute in your controller:
albums_controller.rb
params.require(:album).permit(:title, :singer, :image, :remove_image)
Adding Validations
At this stage, everything is working fine, but we're not checking the user's input at all, which is not particularly great. Therefore, let's add validations for the Album model:
models/album.rb
validates :title, presence: true validates :singer, presence: true validates :image, presence: true validates_property :width, of: :image, in: (0..900)
validates_property is the Dragonfly method that may check various aspects of your attachment: you may validate a file's extension, MIME type, size, etc.
Now let's create a generic partial to render the errors that were found:
views/shared/_errors.html.erb
<% if object.errors.any? %>
<div>
<h4>The following errors were found:</h4>
<ul>
<% object.errors.full_messages.each do |msg| %>
<li><%= msg %></li>
<% end %>
</ul>
</div>
<% end %>
Employ this partial inside the form:
views/albums/_form.html.erb
<%= form_for @album do |f| %>
<%= render 'shared/errors', object: @album %>
<%# ... %>
<% end %>
Style the fields with errors a bit to visually depict them:
stylesheets/application.scss
.field_with_errors {
display: inline;
label {
color: red;
}
input {
background-color: lightpink;
}
}
Retaining an Image Between Requests
Having introduced validations, we run into yet another problem (quite a typical scenario, eh?): if the user has made mistakes while filling in the form, he or she will need to choose the file again after clicking the Submit button.
Dragonfly can help you solve this problem as well by using a retained_* hidden field:
views/albums/_form.html.erb
<%= f.hidden_field :retained_image %>
Don't forget to permit this field as well:
albums_controller.rb
params.require(:album).permit(:title, :singer, :image, :remove_image, :retained_image)
Now the image will be persisted between requests! The only small problem, however, is that the file upload input will still display the "choose a file" message, but this can be fixed with some styling and a dash of JavaScript.
Processing Images
Generating Thumbnails
The images uploaded by our users can have very different dimensions, which can (and probably will) cause a negative impact on the website's design. You probably would like to scale images down to some fixed dimensions, and of course this is possible by utilizing the width and height styles. This is, however, not an optimal approach: the browser will still need to download full-size images and then shrink them.
Another option (which is usually much better) is to generate image thumbnails with some predefined dimensions on the server. This is really simple to achieve with Dragonfly:
views/albums/_album.html.erb
<li>
<%= image_tag(album.image.thumb('250x250#').url, alt: album.title) if album.image_stored? %>
<%# ... %>
</li>
250x250 is, of course, the dimensions, whereas # is the geometry that means "resize and crop if necessary to maintain the aspect ratio with center gravity". You may find information about other geometries on Dragonfly's website.
The thumb method is powered by ImageMagick—a great solution for creating and manipulating images. Therefore, in order to see the working demo locally, you'll need to install ImageMagick (all major platforms are supported).
Support for ImageMagick is enabled by default inside Dragonfly's initializer:
config/initializers/dragonfly.rb
plugin :imagemagick
Now thumbnails are being generated, but they are not stored anywhere. This means each time a user visits the albums page, thumbnails will be regenerated. There are two ways to overcome this problem: by generating them after the record is saved or by performing generation on the fly.
The first option involves introducing a new column to store the thumbnail and tweaking the dragonfly_accessor method. Create and apply a new migration:
rails g migration add_image_thumb_uid_to_albums image_thumb_uid:string rails db:migrate
Now modify the model:
models/album.rb
dragonfly_accessor :image do
copy_to(:image_thumb){|a| a.thumb('250x250#') }
end
dragonfly_accessor :image_thumb
Note that now the first call to dragonfly_accessor sends a block that actually generates the thumbnail for us and copies it into the image_thumb. Now just use the image_thumb method in your views:
views/albums/_album.html.erb
<%= image_tag(album.image_thumb.url, alt: album.title) if album.image_thumb_stored? %>
This solution is the simplest, but it's not recommended by the official docs and, what's worse, at the time of writing it does not work with the retained_* fields.
Therefore, let me show you another option: generating thumbnails on the fly. It involves creating a new model and tweaking Dragonfly's config file. First, the model:
rails g model Thumb uid:string job:string rake db:migrate
The thumbs table will host your thumbnails, but they will be generated on demand. To make this happen, we need to redefine the url method inside the Dragonfly initializer:
config/initializers/dragonfly.rb
Dragonfly.app.configure do
define_url do |app, job, opts|
thumb = Thumb.find_by_job(job.signature)
if thumb
app.datastore.url_for(thumb.uid, :scheme => 'https')
else
app.server.url_for(job)
end
end
before_serve do |job, env|
uid = job.store
Thumb.create!(
:uid => uid,
:job => job.signature
)
end
# ...
end
Now add a new album and visit the root page. The first time you do it, the following output will be printed into the logs:
DRAGONFLY: shell command: "convert" "some_path/public/system/dragonfly/development/2017/02/08/3z5p5nvbmx_Folder.jpg" "-resize" "250x250^^" "-gravity" "Center" "-crop" "250x250+0+0" "+repage" "some_path/20170208-1692-1xrqzc9.jpg"
This effectively means that the thumbnail is being generated for us by ImageMagick. If you reload the page, however, this line won't appear anymore, meaning that the thumbnail was cached! You may read a bit more about this feature on Dragonfly's website.
More Processing
You may perform virtually any manipulation to your images after they were uploaded. This can be done inside the after_assign callback. Let's, for example, convert all our images to JPEG format with 90% quality:
dragonfly_accessor :image do
after_assign {|a| a.encode!('jpg', '-quality 90') }
end
There are many more actions you can perform: rotate and crop the images, encode with a different format, write text on them, mix with other images (for example, to place a watermark), etc. To see some other examples, refer to the ImageMagick section on the Dragonfly website.
Uploading and Managing Songs
Of course, the main part of our musical site is songs, so let's add them now. Each song has a title and a musical file, and it belongs to an album:
rails g model Song album:belongs_to title:string track_uid:string rails db:migrate
Hook up the Dragonfly methods, as we did for the Album model:
models/song.rb
dragonfly_accessor :track
Don't forget to establish a has_many relation:
models/album.rb
has_many :songs, dependent: :destroy
Add new routes. A song always exists in the scope of an album, so I'll make these routes nested:
config/routes.rb
resources :albums do
resources :songs, only: [:new, :create]
end
Create a very simple controller (once again, don't forget to permit the track field):
songs_controller.rb
class SongsController < ApplicationController
def new
@album = Album.find(params[:album_id])
@song = @album.songs.build
end
def create
@album = Album.find(params[:album_id])
@song = @album.songs.build(song_params)
if @song.save
flash[:success] = "Song added!"
redirect_to album_path(@album)
else
render :new
end
end
private
def song_params
params.require(:song).permit(:title, :track)
end
end
Display the songs and a link to add a new one:
views/albums/show.html.erb
<h1><%= @album.title %></h1> <h2>by <%= @album.singer %></h2> <%= link_to 'Add song', new_album_song_path(@album) %> <ol> <%= render @album.songs %> </ol>
Code the form:
views/songs/new.html.erb
<h1>Add song to <%= @album.title %></h1>
<%= form_for [@album, @song] do |f| %>
<div>
<%= f.label :title %>
<%= f.text_field :title %>
</div>
<div>
<%= f.label :track %>
<%= f.file_field :track %>
</div>
<%= f.submit %>
<% end %>
Lastly, add the _song partial:
views/songs/_song.html.erb
<li> <%= audio_tag song.track.url, controls: true %> <%= song.title %> </li>
Here I am using the HTML5 audio tag, which will not work for older browsers. So, if you're aiming to support such browsers, use a polyfill.
As you see, the whole process is very straightforward. Dragonfly does not really care what type of file you wish to upload; all in you need to do is provide a dragonfly_accessor method, add a proper field, permit it, and render a file input tag.
Storing Metadata
When I open a playlist, I expect to see some additional information about each song, like its duration or bitrate. Of course, by default this info is not stored anywhere, but we can fix that quite easily. Dragonfly allows us to provide additional data about each uploaded file and fetch it later by using the meta method.
Things, however, are a bit more complex when we are working with audio or video, because to fetch their metadata, a special gem streamio-ffmpeg is needed. This gem, in turn, relies on FFmpeg, so in order to proceed you will need to install it on your PC.
Add streamio-ffmpeg into the Gemfile:
Gemfile
gem 'streamio-ffmpeg'
Install it:
bundle install
Now we can employ the same after_assign callback already seen in the previous sections:
models/song.rb
dragonfly_accessor :track do
after_assign do |a|
song = FFMPEG::Movie.new(a.path)
mm, ss = song.duration.divmod(60).map {|n| n.to_i.to_s.rjust(2, '0')}
a.meta['duration'] = "#{mm}:#{ss}"
a.meta['bitrate'] = song.bitrate ? song.bitrate / 1000 : 0
end
end
Note that here I am using a path method, not url, because at this point we are working with a tempfile. Next we just extract the song's duration (converting it to minutes and seconds with leading zeros) and its bitrate (converting it to kilobytes per second).
Lastly, display metadata in the view:
views/songs/_song.html.erb
<li> <%= audio_tag song.track.url, controls: true %> <%= song.title %> (<%= song.track.meta['duration'] %>, <%= song.track.meta['bitrate'] %>Kb/s) </li>
If you check the contents on the public/system/dragonfly folder (the default location to host the uploads), you'll note some .yml files—they are storing all meta information in YAML format.
Deploying to Heroku
The last topic we'll cover today is how to prepare your application before deploying to the Heroku cloud platform. The main problem is that Heroku does not allow you to store custom files (like uploads), so we must rely on a cloud storage service like Amazon S3. Luckily, Dragonfly can be integrated with it easily.
All you need to do is register a new account at AWS (if you don't have it already), create a user with permission to access S3 buckets, and write down the user's key pair in a safe location. You might use a root key pair, but this is really not recommended. Lastly, create an S3 bucket.
Going back to our Rails application, drop in a new gem:
Gemfile
group :production do gem 'dragonfly-s3_data_store' end
Install it:
bundle install
Then tweak Dragonfly's configuration to use S3 in a production environment:
config/initializers/dragonfly.rb
if Rails.env.production?
datastore :s3,
bucket_name: ENV['S3_BUCKET'],
access_key_id: ENV['S3_KEY'],
secret_access_key: ENV['S3_SECRET'],
region: ENV['S3_REGION'],
url_scheme: 'https'
else
datastore :file,
root_path: Rails.root.join('public/system/dragonfly', Rails.env),
server_root: Rails.root.join('public')
end
To provide ENV variables on Heroku, use this command:
heroku config:add SOME_KEY=SOME_VALUE
If you wish to test integration with S3 locally, you may use a gem like dotenv-rails to manage environment variables. Remember, however, that your AWS key pair must not be publicly exposed!
Another small issue I've run into while deploying to Heroku was the absence of FFmpeg. The thing is that when a new Heroku application is being created, it has a set of services that are commonly being used (for example, ImageMagick is available by default). Other services can be installed as Heroku addons or in the form of buildpacks. To add an FFmpeg buildpack, run the following command:
heroku buildpacks:add http://ift.tt/2naIaf0
Now everything is ready, and you can share your musical application with the world!
Conclusion
This was a long journey, wasn't it? Today we have discussed Dragonfly—a solution for file uploading in Rails. We have seen its basic setup, some configuration options, thumbnail generation, processing, and metadata storing. Also, we've integrated Dragonfly with the Amazon S3 service and prepared our application for deployment on production.
Of course, we have not discussed all aspects of Dragonfly in this article, so make sure to browse its official website to find extensive documentation and useful examples. If you have any other questions or are stuck with some code examples, don't hesitate to contact me.
Thank you for staying with me, and see you soon!
Wednesday, March 22, 2017
Managing Cron Jobs Using Python
In this tutorial, you'll learn the importance of cron jobs and why you need them. You'll have a look at python-crontab, a Python module to interact with the crontab. You'll learn how to manipulate cron jobs from a Python program using the python-crontab module.
What Is Cron?
During system administration, it's necessary to run background jobs on a server to execute routine tasks. Cron is a system process which is used to execute background tasks on a routine basis. Cron requires a file called crontab which contains the list of tasks to be executed at a particular time. All these jobs are executed in the background at the specified time.
To view cron jobs running on your system, navigate to your terminal and type in:
crontab -l
The above command displays the list of jobs in the crontab file. To add a new cron job to the crontab, type in:
crontab -e
The above command will display the crontab file where you can schedule a job. Let's say you have a file called hello.py which looks like:
print "Hello World"
Now, to schedule a cron job to execute the above script to output to another file, you need to add the following line of code:
50 19 * * * python hello.py >> a.txt
The above line of code schedules the execution of the file with output to a file called a.txt. The numbers before the command to execute define the time of execution of the job. The timing syntax has five parts:
- minute
- hour
- day of month
- month
- day of week
Asterisks(*) in the timing syntax indicate it will run every time.
Introducing Python-Crontab
python-crontab is a Python module which provides access to cron jobs and enables us to manipulate the crontab file from the Python program. It automates the process of modifying the crontab file manually. To get started with python-crontab, you need to install the module using pip:
pip install python-crontab
Once you have python-crontab installed, import it into the python program.
from crontab import CronTab
Writing Your First Cron Job
Let's use the python-crontab module to write our first cron job. Create a Python program called writeDate.py. Inside writeDate.py, add the code to print the current date and time to a file. Here is how writeDate.py would look:
import datetime
with open('dateInfo.txt','a') as outFile:
outFile.write('\n' + str(datetime.datetime.now()))
Save the above changes.
Let's create another Python program which will schedule the writeDate.py Python program to run at every minute. Create a file called scheduleCron.py.
Import the CronTab module into the scheduleCron.py program.
from crontab import CronTab
Using the CronTab module, let's access the system crontab.
my_cron = CronTab(user='your username')
The above command creates an access to the system crontab of the user. Let's iterate through the cron jobs and you should be able to see any manually created cron jobs for the particular username.
for job in my_cron:
print job
Save the changes and try executing the scheduleCron.py and you should have the list of cron jobs, if any, for the particular user. You should be able to see something similar on execution of the above program:
50 19 * * * python hello.py >> a.txt # at 5 a.m every week with:
Let's move on with creating a new cron job using the CronTab module. You can create a new cron by using the new method and specifying the command to be executed.
job = my_cron.new(command='python /home/jay/writeDate.py')
As you can see in the above line of code, I have specified the command to be executed when the cron job is executed. Once you have the new cron job, you need to schedule the cron job.
Let's schedule the cron job to run every minute. So, in an interval of one minute, the current date and time would be appended to the dateInfo.txt file. To schedule the job for every minute, add the following line of code:
job.minute.every(1)
Once you have scheduled the job, you need to write the job to the cron tab.
my_cron.write()
Here is the scheduleCron.py file:
from crontab import CronTab my_cron = CronTab(user='roy') job = my_cron.new(command='python /home/roy/writeDate.py') job.minute.every(1) my_cron.write()
Save the above changes and execute the Python program.
python scheduleCron.py
Once it gets executed, check the crontab file using the following command:
crontab -l
The above command should display the newly added cron job.
* * * * * python /home/roy/writeDate.py
Wait for a minute and check your home directory and you should be able to see the dateInfo.txt file with the current date and time. This file will get updated each minute, and the current date and time will get appended to the existing content.
Updating an Existing Cron Job
To update an existing cron job, you need to find the cron job using the command or by using an Id. You can have an Id set to a cron job in the form of a comment when creating a cron job using python-crontab. Here is how you can create a cron job with a comment:
job = my_cron.new(command='python /home/roy/writeDate.py', comment='dateinfo')
As seen in the above line of code, a new cron job has been created using the comment as dateinfo. The above comment can be used to find the cron job.
What you need to do is iterate through all the jobs in crontab and check for the job with the comment dateinfo. Here is the code:
my_cron = CronTab(user='roy')
for job in my_cron:
print job
Check for each job's comment using the job.comment attribute.
my_cron = CronTab(user='jay')
for job in my_cron:
if job.comment == 'dateinfo':
print job
Once you have the job, reschedule the cron job and write to the cron. Here is the complete code:
from crontab import CronTab
my_cron = CronTab(user='roy')
for job in my_cron:
if job.comment == 'dateinfo':
job.hour.every(10)
my_cron.write()
print 'Cron job modified successfully'
Save the above changes and execute the scheduleCron.py file. List the items in the crontab file using the following command:
crontab -l
You should be able to see the cron job with updated schedule time.
* */10 * * * python /home/jay/writeDate.py # dateinfo
Clearing Jobs From Crontab
python-crontab provides methods to clear or remove jobs from crontab. You can remove a cron job from the crontab based on the schedule, comment, or command.
Let's say you want to clear the job with comment dateinfo from the crontab. The code would be:
from crontab import CronTab
my_cron = CronTab(user='roy')
for job in my_cron
if job.comment == 'dateinfo':
my_cron.remove(job)
my_cron.write()
Similarly, to remove a job based on a comment, you can directly call the remove method on the my_cron without any iteration. Here is the code:
my_cron.remove(comment='dateinfo')
To remove all the jobs from the crontab, you can call the remove_all method.
my_cron.remove_all()
Once done with the changes, write it back to the cron using the following command:
my_cron.write()
Calculating Job Frequency
To check how many times your job gets executed using python-crontab, you can use the frequency method. Once you have the job, you can call the method called frequency, which will return the number of times the job gets executed in a year.
from crontab import CronTab
my_cron = CronTab(user='roy')
for job in my_cron:
print job.frequency()
To check the number of times the job gets executed in an hour, you can use the method frequency_per_hour.
my_cron = CronTab(user='roy')
for job in my_cron:
print job.frequency_per_hour()
To check the job frequency in a day, you can use the method frequency_per_day.
Checking the Job Schedule
python-crontab provides the functionality to check the schedule of a particular job. For this to work, you'll need the croniter module to be installed on your system. Install croniter using pip:
pip install croniter
Once you have croniter installed, call the schedule method on the job to get the job schedule.
sch = job.schedule(date_from=datetime.datetime.now())
Now you can get the next job schedule by using the get_next method.
print sch.get_next()
Here is the complete code:
import datetime
from crontab import CronTab
my_crons = CronTab(user='jay')
for job in my_crons:
sch = job.schedule(date_from=datetime.datetime.now())
print sch.get_next()
You can even get the previous schedule by using the get_prev method.
Wrapping It Up
In this tutorial, you saw how to get started with using python-crontab for accessing system crontab from a Python program. Using python-crontab, you can automate the manual process of creating, updating, and scheduling cron jobs.
Have you used python-crontab or any other libraries for accessing system crontab? I would love to hear your thoughts. Do let us know your suggestions in the comments below.
Tuesday, March 21, 2017
Monday, March 20, 2017
Sunday, March 19, 2017
Friday, March 17, 2017
Crafty Beyond the Basics: Collisions
It is very important in a game that you detect collisions properly. Nobody is going to play a game where things start exploding even when they are many pixels apart. Besides the graphics and sounds, this is one more thing that you should try to keep as accurate as possible.
In this tutorial, you will learn about detecting and debugging collisions in detail.
Detecting and Ignoring Hits
Before you can detect any collisions, you need to add the Collision component to an entity. This component will successfully detect a collision between any two convex polygons. This component has two events: HitOn and HitOff. The HitOn event is triggered when a collision occurs. It will not be triggered again unless collisions of that specific type cease. The HitOff event is triggered when a collision ceases.
If you are checking for collision with multiple components and all these collisions occur simultaneously, each collision will fire its own HitOn event. The data received from a collision event is only valid as long as the collision is still occurring.
You can use the .checkHits() method to perform collision checks against all entities with a specified component. Calling this method multiple times will not result in multiple redundant checks.
Keep in mind that hit checks are performed upon entering each new frame. Let's say there are two objects which have not yet collided when the hit check is performed. Now, if one of the objects moves to a new location and overlaps with the second object later in the same frame, the hit events will not be fired until a collision check is performed again in the next frame.
If you have to detect only the first collision between different entities, you can use the .ignoreHits(String componentList) method. The componentList variable is a comma-separated list of components with which you no longer want to detect collisions. When no arguments are provided, it will stop collision detection with all entities. Here is an example:
littleBox.bind("HitOn", function(hitData) {
Crafty("Obstacle").color('red');
this.ignoreHits('Obstacle');
});
You can see that Crafty not only starts detecting the HitOn event but also the HitOff event. That's why the color of the big box does not change back to black.
Another similar method called .resetHitChecks(String componentList) can be used to re-check for collision between specific components. This method will keep firing the HitOn event again and again as long as the collision is still happening.
Debugging Collisions
When entities are moving continuously, it is very hard to see if the collisions are being fired at the right time. In the above demo, it looks as if the HitOn event is firing slightly before the actual event. Crafty gives you the option of drawing hit boxes around entities so that you can actually see what's going on.
There are two components that you can use for debugging purposes. These are WiredHitBox and SoldHitBox.
The first component will allow you to use the .debugStroke([String strokeColor]) method, which will draw an outline around the entity with a given color. When no color is provided, the color red is used to draw the outline.
Similarly, the second component is used to fill the entities with a given color using the .debugFill([String fillStyle]) method. When no color is provided, the color red is used. Here is a demo with the .debugStroke() method.
Creating a Custom Hit Box
You can also create a custom hit box for collision detection. This is helpful when you are using image sprites in your game that are not rectangular. The .collision() method that you can use to create a custom hit area accepts a single parameter that is used to set the coordinates of the new hit box.
These coordinates can be supplied in the form of a polygon object, an array of x,y coordinate pairs, or a list of x,y coordinate pairs. The points of the polygon are marked in a clockwise manner, and they are positioned relative to the unrotated state of our entity. The custom hit area will automatically rotate itself when the entity rotates.
There are a few things that you should keep in mind when using custom hit areas. The hit area that you define should form a convex polygon for the collision detection to work properly. For those who are unfamiliar with the term, a convex polygon is a polygon with all of the interior angles less than 180°. You might also see slight performance degradation when the hit area that you defined lies outside the entity itself.
The custom hit area that you defined won't have any effect unless you add the Collision component to every entity with which your hit area needs to detect a collision.
littleBox.collision(80, 0, 100, 100, 50, 100)
.debugStroke('green')
.checkHits('Obstacle');
In the above demo, we have defined a custom hit box that lies outside the orange box. As you can see, the big block turns blue only when it collides with the triangle. The position of the orange box doesn't matter.
Let's take a look at another example which uses a spaceship by Gumichan01. The default hit box for the spaceship is the boundary of the sprite itself. In the current scenario, the top right corner of the spaceship touches the block first, but that space is actually empty. For users who are playing your game, this is a case of bad collision detection.
What you can do here is define your own hit area using a triangular shape like the following code. The custom hit box polygon that you define can have as many sides as you want. Just make sure that it is still a convex polygon.
spaceShip.collision(8, 0, 0, 48, 70, 48);
Conclusion
After completing all these tutorials, you should now be able to create your own little games with great graphics, nice sound effects, scenes, and collision detection. I should remind you that I have used Crafty version 0.7.1 in this tutorial, and the demos might not work with other versions of the library.
JavaScript has become one of the de-facto languages of working on the web. It’s not without it’s learning curves, and there are plenty of frameworks and libraries to keep you busy, as well. If you’re looking for additional resources to study or to use in your work, check out what we have available in the Envato marketplace.
If you have any questions, let me know in the comments.
Thursday, March 16, 2017
Crafty Beyond the Basics: Sounds and Scenes
Just like images or sprites, sound also plays a vital role in games. The right background music can set the mood for the game. Similarly, including sound effects for things like a crash or gunfire will make the game much more interesting.
You can also add scenes to your game to make it more organised. For example, instead of directly showing the game screen to users, you can first show them the home screen where they can choose a difficulty level for the game or increase/decrease the volume of background music.
In this tutorial, you will learn how to add sounds and scenes to your games using Crafty.
Adding Sounds
The process for adding sounds to a game is similar to adding sprite sheets. You need to create an asset object and then supply an array of audio files for different sound effects. Crafty will then load the first file that is supported by the browser. Here is an example:
var game_assets = {
"audio": {
"back_music": ["back_music.wav", "back_music.ogg", "back_music.mp3"],
"gun_shot": ["gun_shot.wav", "gun_shot.ogg", "gun_shot.mp3"]
}
};
Crafty.load(game_assets);
Once you have added the audio files, you can play them using Crafty.audio.play(String id, Number repeatCount, Number volume). The first parameter is the Id of the file we want to play. To play the background music, you can pass "back_music" as Id.
You can control how many times an audio file is played using the second parameter. When this parameter is not specified, the file is played only once. You would probably want to continuously keep playing some sounds. One such example is the background music of a game. This can be achieved by setting the second parameter to -1.
The third parameter controls the volume of the given audio file. It can have any value between 0.0 and 1.0. This is the code to play background music:
Crafty.audio.play("back_music", -1, 0.5);
You can also play audio files based on some events like collision between entities or a key press.
walking_hero.bind('KeyDown', function(evt) {
if (evt.key == Crafty.keys.UP_ARROW) {
walking_hero.animate("jumping", 1);
Crafty.audio.play("gun_shot", 1);
}
});
Keep in mind that you need to add the Keyboard component to your hero before it can detect the KeyDown event. The above code binds the KeyDown event to the hero and checks if the key was pressed using evt.key. If the Up Arrow key is pressed, a jumping animation is played for the hero once. A gunshot sound is played as well.
Try pressing the Up Arrow key in the following demo and you will see it all in action. I have commented out the line that plays the background music, but you can just uncomment it while playing with the demo.
The background music in the demo has been created by Rosalila, and the gunshot sound is by Luke.RUSTLTD.
There are a lot of other methods that you can use to manipulate the sounds played by Crafty. You can mute and unmute all the audio files that are being played in the game by using .mute() and .unmute() respectively. You can also pause and resume audio files based on their Id by using the .pause(Id) and .unpause(Id) method.
There is a limit on the maximum number of sounds that can be played simultaneously in Crafty. The default limit for this value is 7. Each of the different sounds playing simultaneously is a channel. However, you can set your own value by using Crafty.audio.setChannels(Number n). You can also check if a given audio file is currently playing on at least one channel using the .isPlaying(string ID) method. It will return a Boolean indicating if the audio is playing or not.
Scenes in Crafty
The game screen is generally not the first thing that you see in a game. Usually, the first screen that you see is the loading screen or the main menu screen. Then, once you have set different options like audio or difficulty level, you can click the play button to move on to the actual game screen. Finally, when the game is over, you can show users a game over screen.
These different game screens or scenes make your game more organized. A scene in Crafty can be created by calling Crafty.defineScene(String sceneName, Function init[, Function uninit]).
The first parameter is the name of the scene. The second parameter is the initialization function, which sets things up when the scene is played. The third parameter is an optional function which is executed before the next scene is played and after all the entities with 2D component in the current scene have been destroyed.
Here is the code for defining the loading screen:
Crafty.defineScene("loading_screen", function() {
Crafty.background("orange");
var loadingText = Crafty.e("2D, Canvas, Text, Keyboard")
.attr({ x: 140, y: 120 })
.text("Scenes Demo")
.textFont({ size: '50px', weight: 'bold' })
.textColor("white");
loadingText.bind('KeyDown', function(evt) {
Crafty.enterScene("game_screen");
});
});
In the above code, I have defined a "loading_screen" scene. The initialization function set the background color to orange and shows some text to give the user some information about what's coming next. You can include a logo and some menu options in an actual game here. Pressing any key while the canvas is in focus will take you to the actual game screen. The .enterScene(String sceneName) method has been used here to load the "game_screen".
In the following demo, you can press the UP key 10 times to go to the final screen.
Conclusion
After completing this tutorial, you should be able to add a variety of sound effects to your game and be able to control the audio output. You can now also show different screens to a user in different situations. I should remind you that I have used Crafty version 0.7.1 in this tutorial, and the demos might not work with other versions of the library.
In the next and final tutorial of this series, you will learn how to improve collision detection in Crafty.
