Saturday, March 31, 2018
Friday, March 30, 2018
Creating an Image Editor Using CamanJS: Layers, Blend Modes, and Events
In the previous tutorial, you learned how to create an image editor using CamanJS which can apply basic filters like contrast, brightness, and noise to an image. CamanJS also has some other built-in filters like Nostalgia, Pinhole, Sunrise, etc., which we applied directly to the image.
In this tutorial, we will cover some more advanced features of the library like Layers, Blend Modes, and Events.
Layers in CamanJS
In the first tutorial, we only worked with a single layer which contained our image. All the filters that we applied only manipulated that main layer. CamanJS allows you to create multiple layers to enable you to edit your images in a more sophisticated manner. You can create nested layers, but they will always be applied on their parent layer like a stack.
Whenever you create a new layer and apply it on the parent layer, the default blend mode used will be normal. You can create a new layer on the canvas using the newLayer() method. When you create a new layer, you can also pass a callback function which will be useful if you intend to manipulate layer.
This function can be used for a lot of tasks like setting the blend mode for the new layer using the setBlendingMode() method. Similarly, you can control the opacity of the new layer using the opacity() method.
Any new layer that you create can be filled with a solid color using the fillColor() method. You can also copy the contents of the parent layer to the new layer using the copyParent() method. All the filters that we learned about in the previous tutorial can also be applied on the new layer that we are creating. For example, you can increase the brightness of the newly created layer by using this.filter.brightness(10).
Instead of copying the parent or filling the layer with a solid color, you also have the option to load any other image in the layer and overlay it on the parent. Just like the main image, you will be able to apply different filters to the overlaid image as well.
In the following code snippet, we have attached a click event handler to three buttons which will fill the new layer with a solid color, the parent layer, and an overlay image respectively.
$('#orange-btn').on('click', function (e) {
Caman("#canvas", function () {
this.newLayer(function () {
this.opacity(50);
this.fillColor('#ff9900');
});
this.render();
});
});
$('#parent-btn').on('click', function (e) {
Caman("#canvas", function () {
this.newLayer(function () {
this.opacity(50);
this.copyParent();
this.filter.brightness(20);
});
this.render();
});
});
$('#overlay-btn').on('click', function (e) {
var oImg = new Image();
oImg.src = "trees.png";
Caman("#canvas", function () {
this.newLayer(function () {
this.opacity(50);
this.overlayImage(oImg);
this.filter.brightness(20);
});
this.render();
});
});
Blend Modes in CamanJS
In the previous section, we kept the opacity of any new layer that we added to the canvas below 100. This was done because the new layer would otherwise hide the old layer completely. When you place one layer over another, CamanJS allows you to specify a blend mode which determines the final outcome after the placement. The blend mode is set to normal by default.
This means that any new layer that you add on the canvas will make the layer below it invisible. The library has ten blend modes in total. These are normal, multiply, screen, overlay, difference, addition, exclusion, softLight, exclusion, and darken.
As I mentioned earlier, the normal blend mode sets the final color to be equal to the color of the new layer. The multiply blend mode determines the final color of a pixel by multiplying the individual channels together and then dividing the result by 255. The difference and addition blend modes work in a similar manner, but they subtract and add the channels.
The darken blend mode sets the final color of a pixel to be equal to the lowest value of individual color channels. The lighten blend mode sets the final color of a pixel to be equal to the highest value of individual color channels. The exclusion blend mode is somewhat similar to difference, but it sets the contrast to a lower value. In the case of the screen blend mode, the final color is obtained by inverting the colors of each layer, multiplying them, and then again inverting the result.
The overlay blend mode acts like multiply if the bottom color is darker, and it acts like screen if the bottom color is lighter.
If you want the colors in different layers to interact in a different manner, CamanJS also lets you define your own blend modes. We will cover this in the next tutorial of the series.
Here is the JavaScript code to apply different blend modes on an image:
$('#multiply-btn').on('click', function (e) {
hexColor = $("#hex-color").val();
Caman("#canvas", function () {
this.revert(false);
this.newLayer(function () {
this.fillColor(hexColor);
this.setBlendingMode('multiply');
});
this.render();
});
});
$('#screen-btn').on('click', function (e) {
hexColor = $("#hex-color").val();
Caman("#canvas", function () {
this.revert(false);
this.newLayer(function () {
this.fillColor(hexColor);
this.setBlendingMode('screen');
});
this.render();
});
});
In the above code snippet, we get the Hex color value from an input field. This color is then applied on the new layer. You can write the code to apply other blend modes in a similar manner.
Try to specify a color of your choice in the input field, and then apply any of the blend modes by clicking on the respective button. I have applied the blend modes on a solid color in the example, but you can also apply them on an overlaid image from the previous section.
Events in CamanJS
If you uploaded any large image in either the demo of the first tutorial or the second tutorial, you might have noticed that the result of any applied filter or blend mode became evident after a long time.
Large images have a lot of pixels, and calculating the final value of different channels for each pixel after applying a specific blend mode can be very time-consuming. For example, when applying the multiply blend mode on an image with dimensions 1920*1080, the device will have to perform multiplication and division over 6 million times.
In such cases, you can use events to give users some indication about the progress of a filter or blend mode. CamanJS has five different events which can be used to execute specific callback functions at different stages. These five events are processStart, processComplete, renderFinished, blockStarted, and blockFinished.
The processStart and processComplete events are triggered after a single filter starts or finishes its rendering process. When all the filters that you specified have been applied on an image, the library fires the renderFinished event.
CamanJS divides large images into blocks before starting to manipulate them. The blockStarted and blockFinished events are fired after individual blocks of the image have been processed by the library.
In our example, we will only be using the processStart and renderFinished events to inform users about the progress of our image editing operation.
Caman.Event.listen("processStart", function (process) {
$(".process-message").text('Applying ' + process.name);
});
Caman.Event.listen("renderFinished", function () {
$(".process-message").text("Done!");
});
With the processStart and processFinish events, you get access to the name of the process currently operating on the image. The blockStarted and blockFinished events, on the other hand, give you access to information like the total number of blocks, the current block being processed, and the number of finished blocks.
Try clicking on any button in the demo below, and you will see the name of the process currently manipulating the image in the area below the canvas.
Final Thoughts
The first tutorial of the series showed you how to create a basic image editor with built-in filters from the CamanJS library. This tutorial showed you how to work with more than one layer and apply different filters and blend modes to each layer individually.
Since the image editing process can take a while for large images, we also learned how to indicate to users that the image editor is actually processing the image and not sitting idle.
In the next and final tutorial of the series, you will learn how to create your own blend modes and filters in CamanJS. If you have any questions related to this tutorial, feel free to let me know in the comments.
Thursday, March 29, 2018
Wednesday, March 28, 2018
Creating an Image Editor Using CamanJS: Applying Basic Filters
A while back, I wrote some tutorials which described how to apply different kinds of filters and blend modes to an image using just CSS. This could be very helpful in situations where you want to show the grayscale, blurred, or high-contrast version of the same image. Instead of creating four different images, you could just apply these effects to the original image using a few lines of CSS.
Using CSS filters and blend modes works nicely in most cases. However, CSS doesn't modify the pixels of the image itself. In other words, the filters and blend modes or any other effects are not permanent.
If someone downloads an image with CSS filters applied to it, they will get the original image and not the modified version. This can be a major setback if you were planning on creating an image editor for your users.
If you want the image modifications to be permanent and allow the user to download the modified image, you can use HTML5 canvas. The canvas element allows you to do a lot of things, including drawing lines and shapes, writing text, and rendering animations.
In this tutorial, we will focus on editing images loaded on the canvas. CSS3 already has built-in functionality to allow you to apply effects like contrast, brightness, and blurring directly. When working with HTML5 canvas, we will use a canvas manipulation library called CamanJS to edit the images.
The library supports basic effects like brightness, contrast, and saturation out of the box. This will save time and allow us to create more sophisticated filters based on these basic ones.
CamanJS Basics
The name of this library is based on the fact that it is used for doing (ca)nvas (man)ipulation in JavaScript(JS). Before you can start using different features of the library, you will have to include it in your project. This can be done either by downloading the library and hosting it yourself or by linking directly to a CDN.
There are two ways to use the library. The first option is to use the data-caman attribute with your image elements. This attribute can accept a combination of different CamanJS filters as its value. For example, if you want to increase the brightness of an image by 20 and the contrast by 10, you can use the following HTML:
<img src="path/to/image.jpg"
data-caman="brightness(20) contrast(10)">
Similarly, you can apply other filters like saturation, exposure, noise, sepia, etc. Besides the basic filters, CamanJS also gives you access to some more sophisticated filters out of the box. These filters can be applied to an image in a similar manner. To apply the sunrise filter, you can simply use the following HTML:
<img src="path/to/image.jpg"
data-caman="sunrise()">
Your second option for manipulating images is by calling Caman() with the id of the canvas where you have rendered the image and different filters that you want to apply to the rendered image.
Caman('#canvas-id', function () {
this.brightness(20);
this.contrast(10);
this.render();
});
In this series, we will be going the JavaScript way to create our image editor.
Implementing Upload and Download Functionality
You need to provide users with a way to upload the images they want to edit so that you can render them on the canvas for further manipulation. Once the users have made the changes, they should also be able to download the edited images. In this section, we will add these two functions to our image editor.
Let's begin with the HTML needed to add the canvas and upload/download buttons:
<div class="preview-wrapper">
<canvas id="canvas"></canvas>
<p class="process-message"></p>
</div>
<div class="editor-buttons">
<input type="file" id="upload-file" placeholder="Upload a Picture" />
<label for="upload-file">Upload a Picture</label>
<button id="download-btn">Download Image</button>
<br/>
</div>
Here is the code for implementing the basic image upload functionality:
var img = new Image();
var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
var fileName = '';
$("#upload-file").on("change", function(){
var file = document.querySelector('#upload-file').files[0];
var reader = new FileReader();
if (file) {
fileName = file.name;
reader.readAsDataURL(file);
}
reader.addEventListener("load", function () {
img = new Image();
img.src = reader.result;
img.onload = function () {
canvas.width = img.width;
canvas.height = img.height;
ctx.drawImage(img, 0, 0, img.width, img.height);
$("#canvas").removeAttr("data-caman-id");
}
}, false);
});
We begin by creating some variables to store the name of the image file selected by the user and the context for our canvas. After that, we write the code to get the image file from the file input after its change event is fired. The files selected by a user are stored in a FileList, and we can get the first file from the list using .files[0].
Once we have the file, we use a FileReader object to read the contents of the file selected by the user. The onload event for the FileReader is triggered after the selected file has been read successfully.
Inside the onload event handler for the FileReader object, we create an HTMLImageElement instance using the Image() constructor. The src attribute of the image is then set to the value of the result property of our FileReader.
Once the image has loaded successfully, we set the width and height of our canvas to be equal to the width and height of the image selected by the user. After that, we draw the image on the canvas and remove the data-caman-id attribute from the canvas.
The attribute is added automatically by CamanJS when setting up the canvas for editing an image. We need to remove it every time a user selects a new file in order to avoid any mixup between the old image file we were editing and the new file selected by the user.
Besides loading the image file in the canvas, we have also set the value of the fileName variable to be equal to the name of the file selected by the user. This will be useful when we are saving the edited image.
Users will now be able to upload different images in your image editor. Once they have edited the image, they would also like to download them. Let's write some code that will allow users to save the edited image file.
$('#download-btn').on('click', function (e) {
var fileExtension = fileName.slice(-4);
if (fileExtension == '.jpg' || fileExtension == '.png') {
var actualName = fileName.substring(0, fileName.length - 4);
}
download(canvas, actualName + '-edited.jpg');
});
function download(canvas, filename) {
var e;
var lnk = document.createElement('a');
lnk.download = filename;
lnk.href = canvas.toDataURL("image/jpeg", 0.8);
if (document.createEvent) {
e = document.createEvent("MouseEvents");
e.initMouseEvent("click", true, true, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null);
lnk.dispatchEvent(e);
}
else if (lnk.fireEvent) {
lnk.fireEvent("onclick");
}
}
We use the jQuery .on() method to execute a piece of code every time the click event is fired for the download button. This code removes the file extension from the name of the image file selected by the user and replaces it with the suffix -edited.jpg. This name is then passed to the download function along with a reference to the canvas where we rendered and edited the image.
The download function creates a link and sets its download attribute to filename. The href attribute contains the data URI for the edited image. After setting the value of these two attributes, we programmatically fire the click event for our newly created link. This click starts the download of the edited image.
Applying Built-in Filters
As I mentioned in the beginning of the tutorial, CamanJS comes with basic built-in filters. So you can directly apply brightness, contrast, sepia, saturation, exposure, noise, sharpen, vibrance, and hue. Some filters like brightness and contrast can have a negative as well as a positive value.
You can make the values as high or as low as you want, but a sensible choice would be to keep them between -100 and 100. For example, the image becomes white when you set the brightness to 100. So any value above 100 will be useless. Similarly, the image will become completely gray if you set the value of the contrast to -100.
Other filters like sepia, noise, sharpen, and blur will only accept a positive value. Keep in mind that the hue filter covers the full 360-degree circle, with values ranging from 0 to 100. The image will look exactly the same when you set the hue to 0 or 100.
Here is the HTML to add buttons for our image editor:
<div class="filter-buttons">
<div class="filter-group">
<button id="brightness-dec">-</button>
<span class="filter-name">Brightness</span>
<button id="brightness-inc">+</button>
</div>
<!-- More Such Buttons -->
<div class="filter-group">
<button id="gamma-dec" class="disabled">-</button>
<span class="filter-name">Gamma</span>
<button id="gamma-inc">+</button>
</div>
</div>
<div class="editor-buttons">
<input type="file" id="upload-file" placeholder="Upload a Picture" />
<label for="upload-file">Upload a Picture</label>
<button id="download-btn">Download Image</button>
<br/>
<button id="vintage-btn">Vintage</button>
<!-- More Such Buttons -->
<button id="lomo-btn">Lomo</button>
</div>
All the filters like brightness and contrast have been given increase and decrease buttons. However, the decrease button has been disabled for some filters like noise because they can't have a meaningful negative value.
We will apply the respective filters based on the button clicked with the help of the following JavaScript.
/* Similar Code for All Other Buttons */
$('#brightness-inc').on('click', function (e) {
Caman('#canvas', img, function () {
this.brightness(10).render();
});
});
$('#brightness-dec').on('click', function (e) {
Caman('#canvas', img, function () {
this.brightness(-10).render();
});
});
/* Similar Code for All Inbuilt Filters */
$('#nostalgia-btn').on('click', function (e) {
Caman('#canvas', img, function () {
this.nostalgia().render();
});
});
$('#majestic-btn').on('click', function (e) {
Caman('#canvas', img, function () {
this.herMajesty().render();
});
});
For the increase and decrease buttons, the strength of the filter is based on how its effect scales. For example, the brightness and contrast are increased by 10, but the value of gamma is only increased by 0.1 after each click.
The following CodePen demo shows the CamanJS image editor we created in action.
Some filters might take some time before you see their final outcome. In such cases, users might think that the filter is not working. We will be using events to keep readers updated about the progress of a filter. All this will be discussed in the next tutorial.
Final Thoughts
The first tutorial was meant to teach you how to create an image editor with basic image upload and download functionality which users can use to edit images. We used basic filters like noise, contrast, and brightness as well as some more complicated effects like Vintage and Nostalgia, which are built into the library.
In the next tutorial, you will learn about more CamanJS features like layers, blend modes, and events. And in the meantime, don't forget to review what we have available in the Envato Market for purchase, study, use, and review.
Tuesday, March 27, 2018
Monday, March 26, 2018
Creating an Image Editor Using CamanJS: Creating Custom Filters and Blend Modes
In the first tutorial of our CamanJS image editor series, we only used built-in filters to edit our images. This limited us to some basic effects like brightness, contrast, and 18 other more complicated filters with names like Vintage, Sunrise, etc. They were all easy to apply, but we were not in full control of the individual pixels of the image we wanted to edit.
In the second tutorial, we learned about layers and blend modes, which gave us more control over the images we were editing. For instance, you could add a new layer on the canvas, fill it with a color or image, and then place it over the parent layer with a blend mode applied to it. However, we were still not creating our own filters, and the blend modes we could apply were limited to the ones already provided by CamanJS.
The aim of this tutorial will be to teach you how to create your own blend modes and filters. We will also address some bugs present in the library and how you can patch them when using CamanJS in your own projects.
Creating New Blend Modes
By default, CamanJS offers ten blend modes. These are normal, multiply, screen, overlay, difference, addition, exclusion, softLight, lighten, and darken. The library also allows you to register your own blend modes. This way, you can control how the corresponding pixels of the current layer and parent layer mix together in order to produce the final result.
You can create a new blend mode using Caman.Blender.register("blend_mode", callback);. Here, blend_mode is the name that you want to use in order to identify the blend mode you are creating. The callback function accepts two parameters which contain the RGB values for different pixels on the current layer and corresponding pixels on the parent layer. The function returns an object with final values for the rgb channels.
Here is an example of a custom blend mode which sets the value of individual channels of a pixel to 255 if the value of that channel for the corresponding pixel in the parent layer is over 128. If the value is below 128, the final channel value is the result of subtracting the current layer channel value from the parent channel value. The name of this blend mode is maxrgb.
Caman.Blender.register("maxrgb", function(rgbaLayer, rgbaParent) {
return {
r: rgbaParent.r > 128 ? 255 : rgbaParent.r - rgbaLayer.r,
g: rgbaParent.g > 128 ? 255 : rgbaParent.g - rgbaLayer.g,
b: rgbaParent.b > 128 ? 255: rgbaParent.b - rgbaLayer.b
};
});
Let's create another blend mode in a similar manner. This time, the final channel values will be set to 0 if the channel value for the corresponding pixel in the parent layer is greater than 128. If the channel value for the parent layer is less than 128, the final result would be the addition of the channel values for the current layer and parent layer of the particular pixel. This blend mode has been named minrgb.
Caman.Blender.register("minrgb", function(rgbaLayer, rgbaParent) {
return {
r: rgbaParent.r < 128 ? rgbaParent.r + rgbaLayer.r : 0,
g: rgbaParent.g < 128 ? rgbaParent.g + rgbaLayer.r : 0,
b: rgbaParent.b < 128 ? rgbaParent.r + rgbaLayer.r : 0
};
});
You should try and create your own blend modes for practice.
Creating New Pixel-Based Filters
There are two broad categories of filters in CamanJS. You can either operate on the whole image one pixel at a time or you can modify an image using a convolution kernel. A convolution kernel is a matrix which determines the color of a certain pixel based on the pixels around it. In this section, we will focus on pixel-based filters. Kernel manipulations will be covered in the next section.
Pixel-based filters are given the value of RGB channels for one pixel at a time. The final RGB values for that particular pixel are not affected by the surrounding pixels. You can create your own filters using Caman.Filter.register("filter_name", callback);. Any filter that you create must call the process() method. This method accepts the filter name and a callback function as parameters.
The following code snippet shows you how to create a pixel-based filter which turns images greyscale. This is done by calculating the luminescence of each pixel and then setting the value of individual channels to be equal to the calculated luminescence.
Caman.Filter.register("grayscale", function () {
this.process("grayscale", function (rgba) {
var lumin = (0.2126 * rgba.r) + (0.7152 * rgba.g) + (0.0722 * rgba.b);
rgba.r = lumin;
rgba.g = lumin;
rgba.b = lumin;
});
return this;
});
You can create a threshold filter in a similar manner. This time, we will allow the users to pass a threshold value. If the luminosity of a particular pixel is above the user provided limit, that pixel will turn white. If the luminosity of a particular pixel is less than the user provided limit, that pixel will turn black.
Caman.Filter.register("threshold", function (limit) {
this.process("threshold", function (rgba) {
var lumin = (0.2126 * rgba.r) + (0.7152 * rgba.g) + (0.0722 * rgba.b);
rgba.r = lumin > limit ? 255 : 0;
rgba.g = lumin > limit ? 255 : 0;
rgba.b = lumin > limit ? 255 : 0;
});
return this;
});
As an exercise, you should try and create your own pixel-based filters which, for example, increase the value for a particular channel on all pixels.
Instead of manipulating the color of the current pixel, CamanJS also allows you to set the color for pixels at absolute and relative locations. Unfortunately, this behavior is a little buggy, so we will have to rewrite some methods. If you look at the source code of the library, you will notice that methods like getPixel() and putPixel() call the methods coordinatesToLocation() and locationToCoordinates() on this. However, these methods are not defined on the prototype but on the class itself.
Another issue with the library is that the putPixelRelative() method uses the variable name nowLoc instead of newLoc in two different places. You can get rid of both these issues by adding the following code inside your script.
Caman.Pixel.prototype.coordinatesToLocation = Caman.Pixel.coordinatesToLocation
Caman.Pixel.prototype.locationToCoordinates = Caman.Pixel.locationToCoordinates
Caman.Pixel.prototype.putPixelRelative = function (horiz, vert, rgba) {
var newLoc;
if (this.c == null) {
throw "Requires a CamanJS context";
}
newLoc = this.loc + (this.c.dimensions.width * 4 * (vert * -1)) + (4 * horiz);
if (newLoc > this.c.pixelData.length || newLoc < 0) {
return;
}
this.c.pixelData[newLoc] = rgba.r;
this.c.pixelData[newLoc + 1] = rgba.g;
this.c.pixelData[newLoc + 2] = rgba.b;
this.c.pixelData[newLoc + 3] = rgba.a;
return true;
};
After correcting the code, you should now be able to create a filter that relies on putPixelRelative() without any issues. Here is one such filter that I created.
Caman.Filter.register("erased", function (adjust) {
this.process("erased", function (rgba) {
if(Math.random() < 0.25) {
rgba.putPixelRelative(2, 2, {
r: 255,
g: 255,
b: 255,
a: 255
});
}
});
return this;
});
This filter randomly sets the value of pixels two rows up and two columns to the right of the current pixel to white. This erases parts of the image. Hence the name of the filter.
Creating New Kernel Manipulation Based Filters
As I mentioned earlier, CamanJS allows you to create custom filters where the color of the current pixel is determined by the pixels surrounding it. Basically, these filters go over each pixel in the image that you are editing. A pixel in the image will be surrounded by eight other pixels. The values of these nine pixels from the image are multiplied by the corresponding entries of the convolution matrix. All these products are then added together to get the final color value for the pixel. You can read about the process in more detail in the GIMP documentation.
Just like pixel-based filters, you can define your own kernel manipulation filters using Caman.Filter.register("filter_name", callback);. The only difference is that you will now call processKernel() inside the callback function.
Here is an example of creating an emboss filter using kernel manipulation.
Caman.Filter.register("emboss", function () {
this.processKernel("emboss", [
-2, -1, 0,
-1, 1, 1,
0, 1, 2
]);
});
The following CodePen demo will show all the filters that we created in this tutorial in action.
Final Thoughts
In this series, I have covered almost everything that CamanJS has to offer in terms of canvas-based image editing. You should now be able to use all the built-in filters, create new layers, apply blend modes on those layers, and define your own blend modes and filter functions.
You can also go through the guide on the CamanJS website in order to read about anything that I might have missed. I would also recommend that you read the source code of the library in order to learn more about image manipulation. This will also help you uncover any other bugs in the library.
Sunday, March 25, 2018
Saturday, March 24, 2018
Friday, March 23, 2018
Thursday, March 22, 2018
Wednesday, March 21, 2018
Tuesday, March 20, 2018
Monday, March 19, 2018
How to Get the Guitar Heard Properly in a Band: Part 2
In the previous tutorial I examined how the guitar competes with other instruments in a typical band set-up because of overlapping frequencies.
I also offered solutions, such as:
- Choose a guitar wisely, as wood, pickups, and so on have a bearing on being heard
- When working with another guitarist, go for the opposite of their set-up, such as guitar, amplifier and so on
- Point the speakers at your head
- Put distance between yourself and other instruments, such as drums and keyboards
In this tutorial I’ll explain the choosing and setting up an amplifier.
Before starting with the input, consider the output.
The Most Overlooked Part of the Sound
Whilst choosing the right guitar, amp and effects is important, the guitar speaker is often ignored. After all, speakers are speakers, right.
Wrong.
My epiphany occurred at a guitar show when a Celestion demonstrator switched between three speaker cabinets whilst playing. Despite all being from the same manufacturer, the difference from speaker to speaker was night and day.
If you’ve a nice amp, the accompanying speaker’s likely to be of a decent quality. If not, you should consider an upgrade. Typically £70 to £150, it’s a tonal improvement far cheaper than a new amp or guitar.
You could spend hours staring at frequency response graphs, or read the many arguments as to alnico versus ceramic magnets.
The simplest solution, however, is to find guitar sounds you like, then research the speakers that are being used.
Alternatively, try searching YouTube for guitar speaker comparison. Ensure you’re listening on decent speakers or headphones to really hear the differences.
Choosing an Amp
In an ideal world—amps are amps—they’d all do the same job. If that were true, however, there wouldn’t be the huge range of makes and models that exists currently.
It is true that some amps have different or pronounced characteristics.
Certain brands have become synonymous with particular styles or tones. For example, if I say Marshall, you’re unlikely to think of your favourite jazz guitarist.
The point I’m making here is, if you’ve a certain sound in mind, some amps are better suited than others, and that may help you get the right tone straight away. That said, don’t be swayed by brand image—if it sounds right to you, then it is.
Now to examine the major controls.
Amp Controls
These differ from one manufacturer to another, but, broadly speaking, you usually have the following:
- Gain/Pre Gain
- Volume/Post Gain
- Bass/Lo
- Mid (sometimes omitted)
- Treble/Hi
There can be other controls, such as Presence, but I'll concentrate on those listed.
Gain and Volume
Everyone knows what volume means—more is louder, less is quieter. The word gain’ is often associated with distortion. This is true, but only because excessive usage leads to it.
The best way to understand how they work is to view gain as the input signal, and volume as the output signal.
In simple terms, the sound is cleaner if you keep the gain low and the volume high.
Conversely, you’ll hit distortion sooner if you increase the gain whilst lowering the volume.
When playing a single channel amp, I’ll typically set the gain-to-volume ratio as 3:4. In other words, if my volume’s at 100%, the gain’s set to 75%.
Playing gently gives me a clean sound, but the gain ensures punch and robustness. Hitting the strings harder, the sound crunches up. For more dirt, an overdrive pedal set to low gain/high volume produces a fat yet articulate sound.
EQ
Short for equalisation, this refers to controls adjusting frequency levels within an audio signal. On the amp, this is covered by the Bass, Mid, and Treble controls. But these names, like the controls themselves, aren’t very precise.
Amp EQ
Whilst the amp’s EQ controls are more useful than a single tone knob, they’re hardly the surgical tools common to recording studios. Most lack a Q control, which governs the size of the alteration being made. Without this, you’ve no idea how many frequencies are being affected.
Furthermore, even when these controls are set flat—neither cutting nor boosting—a lot of very famous amps have a slight dip in that all-important mid-range. Boosting the Middle does little to counter this, so cutting them makes it a lot worse.
Mids are important.
Home on the Midrange
A guitar’s typical frequency response is 80-4500Hz, putting it in the following ranges:
- Bass (60-250Hz)
- Low Mids (250-500Hz)
- Mids (500-2000Hz)
- High Mids (2000-4000Hz)
- Presence (4000-6000Hz)
The guitar’s therefore considered to be a predominantly mid-range instrument. With this in mind, here’s something to avoid.
Don’t Smile
Beloved of metal bands, the infamous Smile EQ refers to mids cut, bass and treble boosted. As preserving the mid-range is crucial, this approach is a bad idea. Trust me, for every gigging guitarist cutting their mids, there’s a sound engineer boosting them.
Therefore, keep the mids up, but don’t go crazy as you could create a very nasal, cocked wah sound.
As for bass, use less than you might think. Adding treble is fine, but too much induces harshness.
Set the controls halfway, and adjust from there. A good rule of thumb is dialling each control up to the point where it’s obnoxious, and then back a bit.
Favourite Settings
You may have seen an amp with stickers around the controls. This is the set-and-forget approach, something I wouldn’t endorse, as the sound changes from room to room, venue to venue.
Conclusion
Achieving clarity in a band is having the right tools and understanding how they work. In terms of the amp:
- Research the one that suits you best
- A different speaker can completely change the sound
- Balance volume and gain according to needs
- Preserve the midrange
- Easy on the bass
- Go bright but not harsh
- Avoid Smile EQ
- Set EQ according to the room you’re in
In the next tutorial, I’ll demonstrate the effects that can improve clarity in a live environment.
Saturday, March 17, 2018
Friday, March 16, 2018
Getting Started With the Mojs Animation Library: The Burst Module
We started this series by learning how to animate HTML elements using mojs. In the second tutorial, we moved on to animation of built-in SVG shapes using the Shape module. The third tutorial covered more ways of animating SVG shapes using the ShapeSwirl and stagger modules.
Now, we will learn how to animate different SVG shapes in a burst formation using the Burst module. This tutorial will depend on concepts we covered in the previous three tutorials. If you have not already read them, I would suggest that you go through them first.
Creating Basic Burst Animations
The first thing that we need to do before we can create any burst animations is instantiate a Burst object. After that, we can just specify the values of different properties to control how the animation plays out. The names of a lot of properties in the Burst module are the same as the properties in the Shape module. However, these properties perform very different tasks in this case.
The left and right properties determine the initial position of the burst instead of particles inside it. Similarly, the x and y properties determine the shift of the whole burst instead of individual particles.
The radius of the circle formed by all the burst particles is controlled by the radius property. This is very different from the radius property of individual shapes, which determines the size of those shapes. In the case of a burst, the radius determines how much further apart the individual shapes in it are going to be.
The number of shapes or particles in a single burst can be specified using the count property. By default, there will be five particles in each burst that you create. All these particles are evenly spaced over the circumference of the burst. For example, if there are four particles, they will be placed at 90 degrees to each other. If there are three particles, they will be placed at 120 degrees.
If you don't want the burst particles to cover the whole 360 degrees, you can specify the portion that should be covered using the degree property. Any value above 0 is valid for this property. The specified number of degrees will be evenly distributed between all the particles. If the degree value is over 360, the shapes might overlap.
The angle specified using the angle property determines the angle of the whole burst. In this case, individual particles are not rotated around their own center but around the center of the burst. This is similar to how the earth revolves around the sun, which is different from the rotation of the earth on its own axis.
The scale property scales the value of all physical properties of the burst and in turn individual shapes. Just like other burst properties, all shapes in it would be scaled at once. Setting the burst scale to 3 will increase the radius of the whole burst as well as the size of individual shapes by 3.
In the following code snippet, we are creating five different bursts using the properties we just discussed.
var burstA = new mojs.Burst({
count: 20
});
var burstB = new mojs.Burst({
angle: {
0: 360
},
scale: {
1: 2
},
radius: 10
});
var burstC = new mojs.Burst({
angle: {
0: 360
},
scale: {
1: 2
},
radius: {
10: 100
}
});
var burstD = new mojs.Burst({
degree: 180,
radiusX: 10,
angle: -90,
scale: {
1: 2
},
radius: {
10: 100
}
});
var burstE = new mojs.Burst({
count: 20,
degree: 3600
});
You can see that burstA and burstE only differ in the number of degrees that they have to cover. Since the particles in burstA have to cover 360 degrees (the default value), they are placed 360/20 = 18 degrees apart. On the other hand, the particles in burstE are placed 3600/20 = 180 degrees apart. Starting from zero, the first particle is placed at 0 degrees, and the next is placed at 180 degrees.
The third particle is then placed at 360 degrees, which is basically equal to 0 degrees. The fourth particle is then placed at 540 degrees, but that is basically equal to 180 degrees. In other words, all the odd numbered particles are placed at 0 degrees, and all the even number particles are placed at 180 degrees. In the end, you only see two particles because all others overlap with the first two.
It is important to remember that you cannot directly control the duration, delay or easing function of the burst animations. The module determines all these values automatically based on the values of different children being animated.
Manipulating Individual Burst Particles
So far in this tutorial, all the particles in a burst had the same animation applied to them. Their angle, scale, radius, and position all changed by the same value. Moreover, we were not able to control the duration and delay of either the individual particles or the burst as a whole. The mojs Burst module does not have a set of properties which can directly change all these values. However, we can specify the animation value for individual particles, which in turn affects the burst animation.
All the particles in a burst animation are considered to be children of the original Burst object. Therefore, mojs allows us to control the animation of individual burst particles using a children property, which accepts an object as its value. You can use all the ShapeSwirl properties except x and y inside the children object. This makes sense because the individual particles in a burst animation have to appear at certain positions, and allowing us to randomly change the position of individual particles will change the configuration.
Any children property values that you don't specify will be set to the default provided by the ShapeSwirl module. In the following example, we are animating 20 different lines of our burst animation. This time, the angle property has been set on individual particles instead of the Burst object so that only the lines rotate around their center instead of the whole object. As we learned in the previous tutorial, all the ShapeSwirl objects scale down from 1 to 0 by default. That's why the lengths of the lines change from 40 to 0 in the animation.
var burstA = new mojs.Burst({
count: 20,
children: {
shape: 'line',
stroke: 'black',
radius: 20,
angle: {
0: 180
}
}
});
As I mentioned earlier, we can animate all the ShapeSwirl properties inside the burst animations. Each child in the animation can have its own set of properties. If only one value is provided, it will be applied on all the child particles. If the values are provided as an array, they will be applied sequentially, one particle at a time.
Here is the JavaScript code to create five different burst animations using all the concepts we have learned so far.
var burstA = new mojs.Burst({
count: 20,
angle: {
0: 180
},
radius: {
0: 100
},
children: {
shape: "polygon",
stroke: "black",
radius: 20,
angle: {
0: 360
},
duration: 4000
}
});
var burstB = new mojs.Burst({
count: 20,
angle: {
0: 180
},
radius: {
0: 100
},
children: {
shape: "polygon",
fill: ["yellow", "cyan", "orange"],
stroke: "black",
radius: 20,
scale: {
1: 2
},
duration: 2000
},
isShowEnd: false
});
var burstC = new mojs.Burst({
count: 20,
angle: {
0: -180
},
radius: {
0: 100
},
children: {
shape: "circle",
fill: ["red", "black", "blue"],
radius: {
10: "stagger(5, 1)"
}
}
});
var burstD = new mojs.Burst({
count: 6,
radius: {
0: 100
},
children: {
shape: "circle",
fill: ["red", "yellow", "blue"],
scale: {
1: "rand(1, 10)"
}
},
isShowEnd: false
});
var burstE = new mojs.Burst({
count: 6,
radius: {
0: 100
},
children: {
shape: "circle",
fill: ["red", "yellow", "blue"],
stroke: "black",
scale: {
1: "rand(1, 10)"
}
}
}).then({
angle: {
0: 360
},
radius: {
100: 0
},
scale: {
1: 0
}
});
In the first burst animation, the angle applied directly on the Burst object rotates the whole group around the center of the burst object. However, the angle applied inside the children property rotates all the triangles around their own centers. We also slowed down the burst animation by changing the animation duration for all the children to 4000ms.
In the second burst animation, the color of all the triangles is taken from the array passed to the fill property. We have specified only three fill colors, but the total number of triangles is 20. In such cases, mojs keeps cycling through the array elements and fills the triangles with the same three colors again and again.
In the fourth animation, we use rand strings, which we learned about in the previous tutorial, to randomly choose a scale value for all the child particles. We also set the value of isShowEnd property to false in order to hide the particles at the end of the animation.
In the fifth animation, we use the then() method from the Shape module tutorial to play another animation sequence after the first one finishes.
Final Thoughts
The aim of this series was to get you acquainted with the basics of the mojs animation library. Each tutorial focused on a single module and how you can use the properties in that module to create basic animations.
This last tutorial used the concepts from the previous tutorials to create slightly more complicated animations. Mojs is a very powerful animation library, and the final results you get depend on how creative you can get with all the properties, so keep experimenting.
If there is anything that you would like me to clarify in this tutorial, please let me know in the comments.
Thursday, March 15, 2018
Wednesday, March 14, 2018
Tuesday, March 13, 2018
Monday, March 12, 2018
Cleaning Up Your Data With Go: Part 2
Overview
This is part two out of two in a series on cleaning up data using Go. In part one, we covered the basic text facilities of Go and working with CSV files. In this tutorial, we'll dive into actual data cleaning.
We'll start by understanding the problem of messy data and coming up with a strategy, and then we'll look into verifying individual fields, fixing the data where possible, and deciding what to do about missing values.
Data Cleaning Strategy
A strategy for cleaning up data should dictate what to do when encountering invalid, messy, partial, or missing data. It should also determine what level of reporting is needed about the cleanup process.
The data we're focusing on here is tabular data, where each row is independent. There are no nested hierarchies or connections between different rows of data. A lot of real-world datasets have this nice property.
Remove
The simplest approach for dealing with invalid data is to remove it. If any field is missing or contains invalid data, just get rid of the whole row. This is very easy, and sometimes it is the right thing to do. If the problematic field is critical and you have no way to recover it then all you can do is drop the entire record.
Fix
The best solution is fixing the bad field. In some cases, it's easy to detect the problem and fix it. In the UFO sightings dataset, the state field can be one of the 52 states of the US.
If the value must be all uppercase and some rows contain lowercase letters, you can just make them uppercase.
Report
Reporting on invalid rows, either dropped or fixed, is important. The organization may decide to let people try to fix dropped data. It may be necessary to run fixed data by QA to ensure the automatic fixes didn't introduce invalid data.
Stats
Collecting statistics on the cleanup process is necessary to evaluate the quality of the source data and sometimes to determine if the cleaned-up data is even worth processing. The stats can include the number of dropped and fixed rows and the number of bad and missing fields for each column.
Live Cleanup
So far I've described a pre-processing approach for data cleanup. However, it is possible to perform cleanup during processing. Each row is checked just before it is processed. This is sometimes useful, if there is no point in pre-processing because no one can fix bad data ahead of time for later analysis or if the processing is time-sensitive.
In this scenario, the main purpose of the cleanup is to make sure bad data rows don't break the entire processing pipeline and can be skipped or fixed as necessary.
Verifying Fields
How do you go about verifying fields? You need to know exactly what type of data is supposed to be there and sometimes what values. Here are a few examples.
Verifying Numeric Fields
Numeric fields are very common in data sets. Beyond the type of number (integer, real, complex), some fields are more specialized. For example, a price field may require exactly two decimal points and be positive. Here is a function that checks if a string represents a price:
func validate_price(s string) bool {
parts := strings.Split(s,".")
if len(parts) != 2 {
return false
}
dollars, err := strconv.Atoi(parts[0])
if err != nil {
return false
}
if dollars < 0 {
return false
}
cents, err := strconv.Atoi(parts[1])
if err != nil {
return false
}
if cents < 0 || cents > 99 {
return false
}
return true
}
Verifying URL Fields
Sometimes you need to go above and beyond. If you need to verify that a URL is valid then there are two approaches:
- Parse the URL.
- Try and fetch it (or at least get the headers).
If you only care if the URL is well-formed then the first approach works. But if you want to make sure the URL actually points to a real destination, you need to use the second approach. Since the second approach is a superset of the first approach, let's just use it:
func validate_url(url string) bool {
_, err := http.Head(url)
return err == nil
}
Verifying Custom Format Fields
If the values must respect a custom format, you can usually either match it using simple strings functions like Split() or in more complex cases use regular expressions. For example, if your dataset contains social security numbers (I hope not) in the format XXX-XX-XXXX then you can split by "-" and ensure there are three tokens where the first is three digits long, the second is two digits long, and the third is four digits long. But it's more concise to use a regex like ^\d{3}-?\d{2}-?\d{4}$.
Fixing Invalid Values
Fixing invalid values is not a trivial thing. If your fixing method is incorrect, you can end up with corrupt data. You should consider carefully the importance of the field, the range of possible valid values, and how confident you are that you can really fix any invalid value automatically.
Fixing Case
This is a pretty safe fix. If a text field is supposed to be all uppercase, you can fix it without risking much, because which characters were originally lowercase is not an important piece of information. There is no need to write special code as the strings package has a ToUpper() function. There are also ToLower() and even ToTitle() and ToTitleSpecific() functions to capitalize text properly.
Stripping Unwanted Characters
Another common easy fix is removing leading and trailing whitespace. You'll be surprised how many people add spaces or new lines when entering data. The strings package has a selection of TrimXXX() functions that can take care of most situations:
- Trim()
- TrimFunc()
- TrimLeft()
- TrimLeftFunc()
- TrimPrefix()
- TrimRight()
- TrimRightFunc()
- TrimSpace()
- TrimSuffix()
Dropping Invalid Characters
In some cases, it's OK to drop invalid characters. I recommend only doing it for non-critical and optional fields. For example, you may have a description or notes field that contains free text, and you want to make sure it doesn't contain certain symbols like quotes or double quotes. Here is how to do it:
func remove_quotes(s string) string {
var b bytes.Buffer
for _, r := range (s) {
if r != '"' && r != '\'' {
b.WriteRune(r)
}
}
return b.String()
}
func main() {
original := `'quotes' and "double quotes".`
clean := remove_quotes(original)
fmt.Println(original)
fmt.Println(clean)
}
Output:
'quotes' and "double quotes".
quotes and double quotes.
Fixing Numeric Values
Numeric values are often easy to fix. If you require a precision of two decimal digits, you can truncate or round additional digits. In the same fashion, it's easy to convert integers to floating-point numbers.
Sometimes, there is a range of valid values, and you can bring too large or too small numbers to fit the range. The following function takes a string and a range of integers and returns a string that represents an integer within the range. Too large values become the maximum value, and too small become the minimum value.
func fit_into_range(s string, min int, max int) string {
n, _ := strconv.Atoi(s)
if n < min {
n = min
} else if n > max {
n = max
} else {
return s
}
return strconv.Itoa(n)
}
func main() {
fmt.Println(fit_into_range("15", 10, 20))
fmt.Println(fit_into_range("-15", 10, 20))
fmt.Println(fit_into_range("55", 10, 20))
}
Output:
15
10
20
Fixing URL Values
URLs can often be fixed safely by trying different schemes ("http" or "https") or adding or dropping "www" sub-domains. Combining the options with trying to fetch the candidates can give you confidence that the fix was correct.
Dealing With Missing Values
Missing values are very common when ingesting real-world data. If the missing value is required, there are two primary ways to handle it (without rejecting the row altogether)—use default values or recover the value from an alternative source.
Applying Default Values
Default values are helpful because the processing code doesn't have to check if a value is present or not. The data cleaning code ensures that there is always a value in place. In many cases, the default is so common that it is also a helper for data input where you don't have to enter the same default value again and again.
Using Alternative Data
This approach is a little more involved. The idea is to consult another data source that has the requested information. For example, if you have a user's email, but the first and last name is missing, you may consult your user database and extract the user's name. This saves the processing code from accessing the DB or even being aware of this dependency.
Putting Everything Together
Let's clean up a little dataset of products. The fields are:
| Column Name | Column Description |
|---|---|
| Id | PRD-XXXX-XXXX (where X is a digit) |
| Name | up to 40 characters long |
| Price | fixed precision numeric field (two decimal points) |
| Description | up to 500 characters long (optional) |
Here is the dataset in a readable form (whitespace will be trimmed during cleanup):
const data = `
Id, Name, Price, Description
PRD-1234-0000, Airzooka, 9.99, Shoots air at people
PRD-1234-0017, Pink Onesie, 34.55,
PRD-1234-666, Oh oh, 18.18, Invalid product id
PRD-1234-7777, Oh oh 2, , Missing price
prd-1234-8888, PostIt!, 13.13, Fixable: lowercase id
`
The first two products are valid. The third product, "PRD-1234-666", is missing a digit in its id. The next product, "PRD-1234-7777", is missing a price. The last product, "prd-1234-8888", has an invalid product id, but it can be safely fixed (make it uppercase).
The following code will clean up the data, fix what can be fixed, drop the rows that can't be fixed, and produce a clean dataset and a report that can be used to manually correct the invalid data.
To verify the product id and the price, I'll use regular expressions. Here are the two helper functions:
func verifyProductId(s string) bool {
matched, _ := regexp.MatchString(`^PRD-\d{4}-\d{4}$`, s)
return matched
}
func verifyProductPrice(s string) bool {
matched, _ := regexp.MatchString(`^\d+\.\d\d$`, s)
return matched
}
Once the data is cleaned up and all the invalid rows of data have been dropped, the following function will write the clean data to a new CSV file called "clean.csv" and print it to the screen.
func writeCleanData(cleanData []string) {
f, _ := os.Create("clean.csv")
w := bufio.NewWriter(f)
fmt.Println("Clean data:")
defer w.Flush()
for _, line := range cleanData {
fmt.Println(line)
w.WriteString(line)
w.WriteString("\n")
}
}
The main() function does most of the work. It iterates over the original dataset, eliminates redundant whitespace, fixes what it can, keeps track of dropped data rows, writes the clean data to file, and finally reports on the dropped lines.
func main() {
cleanData := []string{"Id,Name,Price,Description"}
dropped := []string{}
// Clean up data
all_lines := strings.Split(data, "\n")
for _, line := range all_lines {
fields := strings.Split(line, ",")
if len(fields) != 4 {
continue
}
// Strip all leading and trailing spaces from each field
for i, f := range fields {
fields[i] = strings.TrimSpace(f)
}
// Automatic fix (no need to check)
id := strings.ToUpper(fields[0])
if !verifyProductId(id) {
dropped = append(dropped, line)
continue
}
name := fields[1]
// Product names can't be empty
if name == "" {
dropped = append(dropped, line)
continue
}
// Truncate name at 40 characters (runes)
if len([]rune(name)) > 40 {
name = string([]rune(name)[:40])
}
price := fields[2]
if !verifyProductPrice(price) {
dropped = append(dropped, line)
continue
}
description := fields[3]
// Truncate description at 500 characters (runes)
if len([]rune(name)) > 500 {
name = string([]rune(name)[:500])
}
cleanLine := strings.Join([]string{id,
name,
price,
description}, ",")
cleanData = append(cleanData, cleanLine)
}
writeCleanData(cleanData)
// Report
fmt.Println("Dropped lines:")
for _, s := range dropped {
fmt.Println(s)
}
}
Conclusion
Go has well-designed packages for text processing. Unlike in most languages, the string object is really just a slice of bytes. All the string handling logic is in separate packages such as "strings" and "strconv".
In the second part of the tutorial, we utilized a lot of concepts to accomplish a common real-world task of cleaning up a CSV formatted dataset before analysis.
Saturday, March 10, 2018
Friday, March 9, 2018
Cleaning Up Your Data With Go: Part 1
Overview
One of the most important aspects of any application is validating its input. The most basic approach is just failing if the input doesn't satisfy the requirements. However, in many cases this is not enough. In many systems the data collection is separate from data analysis. It could be a survey or an old dataset.
In these cases, it is necessary to go over the entire dataset before analysis, detect invalid or missing data, fix what can be fixed, and flag or remove data that can't be salvaged. It is also useful to provide statistics about the quality of the data and what kinds of errors were encountered.
In this two-part series you'll learn how to use Go's text facilities, slice and dice CSV files, and ensure your data is spotlessly clean. In part one, we'll focus on the foundation of text processing in Go—bytes, runes, and strings—as well as working with CSV files.
Text in Go
Before we dive into data cleaning, let's start with the foundation of text in Go. The building blocks are bytes, runes, and strings. Let's see what each one represents and what the relationships are between them.
Bytes
Bytes are 8-bit numbers. Each byte can represent one of a possible 256 values (2 to the power of 8). Each character in the ASCII character set can be represented by a single byte. But bytes are not characters. The reason is that Go as a modern language supports Unicode, where there are way more than 256 separate characters. Enter runes.
Runes
A rune in Go is another name for the int32 type. This means that each rune can represent more than four billion separate values (2 to the power of 32), which is good enough to cover the entire Unicode character set.
In the following code you can see that the rune '∆' (alt-J on the Mac) is just an int32. To print the character it represents to the screen, I have to convert it to a string.
package main
import (
"fmt"
)
func main() {
r := '∆'
fmt.Println(r)
fmt.Println(int32(r))
fmt.Println(string(r))
}
Output:
8710
8710
∆
Unicode is complicated. A rune officially represents a Unicode code point. Unicode characters are usually represented by a single Unicode code point, but sometimes more than one.
Strings
Strings are officially just read-only slices of bytes. If you index a string, you get a byte back:
func main() {
s := "abc"
for i := 0; i < len(s); i++ {
fmt.Println(s[i])
}
}
Output:
97
98
99
String literals are a sequence of UTF-8 characters enclosed in double quotes. They may contain escape sequences, which are a backslash followed by an ASCII character such as \n (newline) or \t (tab). They have special meanings. Here is the full list:
\a U+0007 alert or bell \b U+0008 backspace \f U+000C form feed \n U+000A line feed or newline \r U+000D carriage return \t U+0009 horizontal tab \v U+000b vertical tab \\ U+005c backslash \' U+0027 single quote (valid only within rune literals) \" U+0022 double quote (valid only within string literals)
Sometimes you may want to store literal bytes directly in a string, regardless of escape sequences. You could escape each backslash, but that's tedious. A much better approach is to use raw strings that are enclosed in backticks.
Here is an example of a string with a \t (tab) escape sequence, which is represented once as is, then with the backslash escape, and then as a raw string:
func main() {
s1 := "1\t2"
s2 := "1\\t2"
s3 := `1\t2`
fmt.Println(s1)
fmt.Println(s2)
fmt.Println(s3)
}
Output:
1 2
1\t2
1\t2
While strings are slices of bytes, when you iterate over a string with a for-range statement, you get a rune in each iteration. This means you may get one or more bytes. This is easy to see with the for-range index. Here is a crazy example. The Hebrew word "שלום" means "Hello" (and peace). Hebrew is also written right to left. I'll construct a string that mixes the Hebrew word with its English translation.
Then, I'll print it rune by rune, including the byte index of each rune within the string. As you'll see, each Hebrew rune takes two bytes, while the English characters take one byte, so the total length of this string is 16 bytes, even though it has four Hebrew characters, three symbols, and five English characters (12 characters). Also, the Hebrew characters will be displayed from right to left:
func main() {
hello := "שלום = hello"
fmt.Println("length:", len(hello))
for i, r := range(hello) {
fmt.Println(i, string(r))
}
}
Output:
length: 16
0 ש
2 ל
4 ו
6 ם
8
9 =
10
11 h
12 e
13 l
14 l
15 o
All these nuances can be extremely important when you have a dataset to clean up with weird quotes and a mix of Unicode characters and symbols.
When printing strings and byte slices, there are several format specifiers that work the same on both. The %s format prints the bytes as is, %x prints two lowercase hexadecimal characters per byte, %X prints two uppercase hexadecimal characters per byte, and %q prints a double quoted string escaped with go syntax.
To escape the % sign inside a format string specifier, just double it. To separate the bytes when using %x or %X, you can add a space, as in "% x" and "% X". Here is the demo:
func main() {
s := "שלום"
fmt.Printf("%%s format: %s\n", s)
fmt.Printf("%%x format: %x\n", s)
fmt.Printf("%%X format: %X\n", s)
fmt.Printf("%% x format: % x\n", s)
fmt.Printf("%% X format: % X\n", s)
fmt.Printf("%%q format: %q\n", s)
}
Output:
%s format: שלום
%x format: d7a9d79cd795d79d
%X format: D7A9D79CD795D79D
% x format: d7 a9 d7 9c d7 95 d7 9d
% X format: D7 A9 D7 9C D7 95 D7 9D
%q format: "שלום"
Reading and Writing CSV Files
Data can arrive in many ways and formats. One of the most common formats is CSV (comma-separated values). CSV data is very efficient. The files typically have a header line with the name of the fields or columns and rows of data where each row contains a value per field, separated by commas.
Here is a little snippet from a UFO sightings dataset (really). The first row (header) contains the column names, and the other lines contain the data. You can see that often the "Colors Reported" column is empty:
City,Colors Reported,Shape Reported,State,Time Ithaca,,TRIANGLE,NY,6/1/1930 22:00 Willingboro,,OTHER,NJ,6/30/1930 20:00 Holyoke,,OVAL,CO,2/15/1931 14:00 Abilene,,DISK,KS,6/1/1931 13:00 New York Worlds Fair,,LIGHT,NY,4/18/1933 19:00 Valley City,,DISK,ND,9/15/1934 15:30 Crater Lake,,CIRCLE,CA,6/15/1935 0:00 Alma,,DISK,MI,7/15/1936 0:00 Eklutna,,CIGAR,AK,10/15/1936 17:00 Hubbard,,CYLINDER,OR,6/15/1937 0:00 Fontana,,LIGHT,CA,8/15/1937 21:00 Waterloo,,FIREBALL,AL,6/1/1939 20:00 Belton,RED,SPHERE,SC,6/30/1939 20:00
Writing this chunk of CSV data to a file involves some string operations as well as working with files. Before we dive into the main logic, here are the mandatory parts: the package definition, the imports, and the data string (note the use of const).
package main
import (
"os"
"strings"
"bufio"
)
data := `
City,Colors Reported,Shape Reported,State,Time
Ithaca,,TRIANGLE,NY,6/1/1930 22:00
Willingboro,,OTHER,NJ,6/30/1930 20:00
Holyoke,,OVAL,CO,2/15/1931 14:00
Abilene,,DISK,KS,6/1/1931 13:00
New York Worlds Fair,,LIGHT,NY,4/18/1933 19:00
Valley City,,DISK,ND,9/15/1934 15:30
Crater Lake,,CIRCLE,CA,6/15/1935 0:00
Alma,,DISK,MI,7/15/1936 0:00
Eklutna,,CIGAR,AK,10/15/1936 17:00
Hubbard,,CYLINDER,OR,6/15/1937 0:00
Fontana,,LIGHT,CA,8/15/1937 21:00
Waterloo,,FIREBALL,AL,6/1/1939 20:00
Belton,RED,SPHERE,SC,6/30/1939 20:00
`
The main() function creates a file called "ufo-sightings.csv", checks that there is no error, and then creates a buffered writer w. The defer call in the next line, which flushes the contents of the buffer to the file, is executed at the end of the function. That is the meaning of defer. Then, it uses the Split() function of the strings package to break the data strings into individual lines.
Then, inside the for-loop, the leading and trailing whitespace is trimmed from each line. Empty lines are skipped, and non-empty lines are written to the buffer, followed by a newline character. That's it. The buffer will be flushed to the file in the end.
func main() {
f, err := os.Create("ufo-sightings.csv")
if err != nil {
panic(e)
}
w := bufio.NewWriter(f)
defer w.Flush()
lines := strings.Split(data, "\n")
for _, line := range lines {
line := strings.Trim(line, " ")
if line == "" {
continue
}
w.WriteString(line)
w.WriteString("\n")
}
}
Reading from the file is pretty simple:
package main
import (
"fmt"
"io/ioutil"
)
func main() {
data, err := ioutil.ReadFile("ufo-sightings.csv")
if err != nil {
panic(err)
}
fmt.Println(string(data))
}
Conclusion
Go has strong facilities to deal with text of all shapes and encodings. In this part of the series, we looked into the basics of text representation in Go, text processing using the strings package, and dealing with CSV files.
In part two, we will put what we've learned into practice to clean up messy data in preparation for analysis.
Thursday, March 8, 2018
Wednesday, March 7, 2018
Demystifying Python Recursion
Most complex tasks in Python can be broken down into simpler subtasks. Recursion helps to achieve this, hence making the code clean and neat. This tutorial will introduce recursion, the benefits of recursion, and how to use it in Python programming.
What Is Recursion?
Recursion is a method of solving a problem with the solutions to smaller instances of the same problem. This approach can be applied to many types of challenges in programming.
The Benefits of Using Recursion
Some of the benefits of using recursion are:
- Recursion adds simplicity when writing code, hence making it easier to debug.
- Recursion reduces the amount of time taken by an algorithm to run as a function of the length of the input.
- Recursion is also preferred when solving very complex problems, especially problems on tree-based structures, because it performs better.
Introduction to the Python Recursive Function
Although recursion seems like a complicated procedure, it's not all that complicated. In layman's terms, assume you have two rectangles A and B. If you add them together, they form a rectangle C. This is in itself a recursive procedure. We have used smaller instances of a rectangle to define itself, and if we were to write a Python function, it would be as follows:
def rectangle(a,b):
return a+b
Since a recursive function calls itself, there needs to be a rule or a breakpoint at which the process or loop would terminate. Such a condition is known as a base condition. A base condition is a requirement in every recursive program, otherwise the procedure would result in an infinite loop.
The second requirement is the recursive case when the function calls itself.
Let's look at an example:
In this example, you will write a factorial function that takes an integer (positive) as an input. The factorial of a number is obtained by multiplying the number by all positive integers below it. For example, factorial(3) = 3 x 2 x 1, factorial(2) = 2 x 1, and factorial(0) = 1.
The first thing to do is to define our base case, which will be factorial(0) = 1.
As you can see above, there is a relationship between each consecutive factorial scenario. You should notice that factorial(4) = 4 x factorial(3). Similarly, factorial(5) = 5 x factorial(4).
The second part will be writing a function that calls itself.
Now that we have simplified it, the resulting function will be:
def factorial(n):
if(n == 0):
#Define our base case?
return 1
else:
return n*factorial(n-1)
print(factorial(5))
#result
# 120
The solution if n==0 is:
def factorial(n):
if(n == 0):
#Define our base case?
return 1
else:
return n*factorial(n-1)
print(factorial(0))
#result
# 0
Now that you know how to write recursive functions, let's look at several case studies that will solidify your understanding of recursion.
Case Study 1: Fibonacci
In a Fibonacci sequence, each number is the sum of the two preceding numbers, such as: 1 + 1 = 2; 1 + 2 = 3; 2 + 3 = 5; 3 + 5 = 8. The Fibonacci sequence has been applied in many areas, and the most common is in predicting price action in the stock market by forex traders.
The Fibonacci sequence starts with 0 and 1. The first number in a Fibonacci sequence is 0, the second number is 1, and the third term in the sequence is 0 + 1 = 1. The fourth is 1 + 1 = 2 and so on.
In order to come up with a recursive function, you need to have two base cases, i.e. 0 and 1. You can then translate the adding pattern into the else case.
The resulting function will be:
def fibonacci(n):
if(n == 1):
#define Base case 1
return 0+1
elif(n == 2):
#define Base case 1
return 1+2
else:
return fibonacci(n) + fibonacci(n-1)
print(fibonacci(5))
#result
#
Case Study 2: Reversing a String
In this example, you will write a function that takes a string as input and then returns the reverse of the string.
The first thing to do is to define our base case, which will check if the string is equal to 0 and, if so, will return the string itself.
The second step is to recursively call the reverse function to slice the part of the string excluding the first character and then concatenate the first character to the end of the sliced string.
The resulting function is as shown below:
def reverse(a):
if len(a) == 0:
return a
else:
return reverse(a[1:]) + a[0]
print(reverse("Python is a very easy language to learn"))
# result
#nrael ot egaugnal ysae yrev a si nohtyP
Case study 3: Sum of Elements
In this example, you will write a function that takes an array as input and then returns the sum of the elements in the list.
The first thing to do is to define our base case, which will check if the size of the list is zero, and return 0 if True.
The second step returns the element and a call to the function sum() minus one element of the list.
The solution is as shown below:
def sum_of_numbers(l):
if len(l) == 1:
return 0
else:
return l[0] + sum(l[1:])
a =[5,7,3,8,10]
print(sum(a))
# result
# 33
The solution for an empty list is as shown below:
def sum_of_numbers(l):
if len(l) == 1:
return 0
else:
return l[0] + sum(l[1:])
b =[]
print(sum(b))
# result
# 0
Conclusion
This tutorial has covered what is necessary to use recursion to solve complex programs in Python. It's also important to note that recursion has its own limitations:
- Recursion takes a lot of stack space, hence making it a bit slow to maintain the program.
- Recursion functions require more space and time to execute.
Remember, don’t hesitate to see what we have available for sale and for study in the Envato Market, and please ask any questions and provide your valuable feedback using the feed below.
Tuesday, March 6, 2018
Solving Player Frustration: Techniques for Random Number Generation
If you strike up a conversation with an RPG fan, it won’t take long to hear a rant about randomized results and loot—and how frustrating they can be. Many gamers have made this irritation known, and while some developers have been creating innovative solutions, many are still forcing us through infuriating tests of perseverance.
There’s a better way. By altering how we as developers utilize random numbers and their generators, we are able to create engaging experiences that push for that “perfect” amount of difficulty without pushing players over the edge. But before we get into that, let’s go over some basics of Random Number Generators (or RNGs for short).
The Random Number Generator and Its Use
Random numbers are all around us, being used to add variation to our software. In general, the major uses of RNGs are to represent chaotic events, show volatility, or behave as an artificial limiter.
You likely interact with random numbers, or the results of their actions, every day. They’re used in scientific trials, video games, animations, art, and nearly every application on your computer. For example, an RNG is likely implemented in the basic animations on your phone.
Now that we’ve talked about what an RNG is, let’s take a look at its implementation and how it can improve our games.
The Standard Random Number Generator
Almost every programming language uses a Standard RNG in basic functions. It works by returning a random value between two numbers. Standard RNGs can be implemented in dozens of different ways across different systems, but they all have generally the same effect: return a randomized number where each value in the range has the same chance of being returned.
For games, these are commonly used to simulate rolling dice. Ideally, they should only be used in situations where each result is desired to occur an equal number of times.
If you want to experiment with rarity or different rates of randomization, this next method is more suitable for you.
Weighted Random Numbers and Rarity Slotting
This type of RNG is the basis for any RPG with item rarity. Specifically, when you need a randomized result but want some to occur with less frequency than others. In most probability classes, this is commonly represented with a bag of marbles. With weighted RNGs, your bag might have three blue marbles and one red one. Since we only want one marble, we’ll either get a red one or a blue one, but it’s much more likely for it to be blue.
Why would weighted randomization be important? Let’s use SimCity’s in-game events as an example. If each event was selected using non-weighted methods, then the potential for each event to occur is statistically the same. That makes it just as likely for you to get a proposal for a new casino as to experience an in-game earthquake. By adding weighting, we can ensure that these events happen in a proportional amount that preserves gameplay.
Its Forms and Uses
Grouping of the Same Items
In many computer science courses or books, this method is often referred to as a ‘bag’. The name is pretty on the nose, using classes or objects to create a virtual representation of a literal bag.
It works basically like this: there is a container that objects can be placed into where they are stored, a function for placing an object into the ‘bag’, and a function for randomly selecting an item from the ‘bag’. To refer back to our marble example, this means that you would treat your bag as containing one blue marble, one blue marble, one blue marble, and one red marble.
Utilizing this method of randomization, we can roughly determine the rate at which an outcome occurs to help homogenize each player’s experience. If we were to simplify results on a scale from ‘Very Bad’ to ‘Very Good’, we’ve now made it much more viable that a player will experience an unnecessary string of unwanted results (such as receiving the ‘Very Bad’ result 20 times in a row).
However, it is still statistically possible to receive a series of bad results, just increasingly less so. We’ll take a look at a method that goes slightly further to reduce unwanted results shortly.
Here’s a quick pseudocode example of what a bag class might look like:
Class Bag {
//Keep an array of all the items that are in the bag
Array itemsInBag;
//Fill the bag with items when its created
Constructor (Array startingItems) {
itemsInBag = startingItems;
}
//add an item to the bag by passing the object (then just push it onto the array)
Function addItem (Object item) {
itemsInBag.push(item);
}
//To get a random item return, use a built-in random function to grab an item from the array
Function getRandomItem () {
return(itemsInBag[random(0,itemsInBag.length-1)]);
}
}
Rarity Slotting Implementation
Similar to the grouping implementation from before, rarity slotting is a method of standardization to determine rates (typically to make the process of game design and player reward easier to maintain).
Instead of individually determining the rate of every item in a game, you’ll create a representative rarity—where the rate of a ‘Common’ might represent a 20 in X chance of a certain outcome, whereas ‘Rare’ might represent a 1 in X chance.
This method doesn’t alter much in the actual function of the bag itself, but rather can be used to increase efficiency on the end of the developer, allowing an exponentially large number of items to be quickly assigned a statistical chance.
In addition, rarity slotting is useful in shaping the perception of a player, by easily allowing them to understand how often an event might occur without eliminating their immersion through number crunching.
Here’s a simple example of how we might add rarity slotting to our bag:
Class Bag {
//Keep an array of all the items that are in the bag
Array itemsInBag;
//add an item to the bag by passing the object
Function addItem (Object item) {
//keep track of the looping related to rarity slots
Int timesToAdd;
//Check the rarity variable on the item
//(but first create that rarity variable in the item class,
//preferably with an enumerated type)
Switch(item.rarity) {
Case 'common':
timesToAdd = 5;
Case 'uncommon':
timesToAdd = 3;
Case 'rare':
timesToAdd = 1;
}
//Add instances of the item to the bag according to rarity
While (timesToAdd >0)
{
itemsInBag.push(item);
timesToAdd--;
}
}
}
Variable Rate Random Numbers
We’ve now talked about some of the most common ways to deal with randomization in games, so let’s delve into a more advanced one. The concept of using variable rates starts similarly to the bag from before: we have a set number of outcomes, and we know how often we want them to occur. The difference with this implementation is that we want to adjust the potential for outcomes as they happen.
Why would we want to do this? Take, for example, games with a collectable aspect. If you have ten possible outcomes for the item you receive, with nine being “common” and one being “rare”, then your chances are pretty straightforward: 90% of the time, a player will get the common, and 10% of the time they’ll get the rare. The issue comes when we take multiple draws into account.
Let’s look at your chances of drawing a series of common results:
- On your first draw, there’s a 90% chance of drawing a common.
- At two draws, there’s an 81% chance of having drawn all commons.
- At 10 draws, there’s still a 35% chance of all commons.
- At 20 draws, there’s a 12% chance of all commons.
While the initial ratio of 9:1 seemed to be an ideal rate at first, it only ended up representing the average result, and left 1 in 10 players spending twice as long as intended to get that rare. Furthermore, 4% of players would spend three times as long to get the rare, and an unlucky 1.5% would spend four times as long.
How Variable Rates Solve This Issue
The solution is implementing a rarity range on our objects. You do so by defining both a maximum and minimum rarity for each object (or rarity slot, if you’d like to combine it with the previous example). For example, let’s give our common item a minimum rarity value of 1, with a maximum of 9. The rare will have a minimum and maximum value of 1.
Now, with the scenario from before, we’ll have ten items, and nine of them are one instance of a common, while one of them is a rare. On the first draw, there is a 90% chance of getting the common. With variable rates now, after that common is drawn, we’re going to lower its rarity value by 1.
This makes our next draw have a total of nine items, eight of which are common, giving an 89% chance of drawing a common. After each common result, the rarity of that item drops, making it more likely to pull the rare until we cap out with two items in the bag, one common and one rare.
Whereas before there was a 35% chance of drawing 10 commons in a row, now there is only a 5% chance. For the outlier results, such as drawing 20 commons in a row, the chances are now reduced to 0.5%, and even further down the line. This creates a more consistent outcome for our players, and prevents those edge cases where a player repeatedly has a bad result.
Building a Variable Rate Class
The most basic implementation of variable rate would be to remove an item from the bag, rather than just returning it, like this:
Class Bag {
//Keep an array of all the items that are in the bag
Array itemsInBag;
//Fill the bag with items when its created
Constructor (Array startingItems) {
itemsInBag = startingItems;
}
//add an item to the bag by passing the object (then just push it onto the array)
Function addItem (Object item) {
itemsInBag.push(item);
}
Function getRandomItem () {
//pick a random item from the bag
Var currentItem = itemsInBag[random(0,itemsInBag.length-1)];
//reduce the number of instances of that item if it's above the minimum
If (instancesOf (currentItem, itemsInBag) > currentItem.minimumRarity) {
itemsInBag.remove(currentItem);
}
return(currentItem);
}
}
While such a simple version brings with itself a few issues (such as the bag eventually reaching a state of standard randomization), it represents the minor changes that can help to stabilize the results of randomization.
Expansions on the Idea
While this covers the basic idea of variable rates, there are still quite a few things to consider for your own implementations:
-
Removing items from the bag helps to create consistent results, but eventually returns to the issues of standard randomization. How could we shape functions to allow both increases and decreases of items to prevent this?
-
What happens when we are dealing with thousands or millions of items? Utilizing a bag filled with bags could be a solution for this. For example, creating a bag for each rarity (all of the common items in one bag, rares in another) and placing each of those into slots within a large bag can provide a wide number of new possibilities for manipulation.
The Case for Less Tedious Random Numbers
Many games are still using standard random number generation to create difficulty. By doing so, a system is created where half of player experiences fall on either side of the intended. If left unchecked, this creates the potential for edge cases of repeat bad experiences to happen at an unintended amount.
By limiting the ranges of how far the results can stray, a more cohesive user experience is ensured, letting a larger number of players enjoy your game without ongoing tedium.
Wrapping Up
Random number generation is a staple of good game design. Make sure that you’re double-checking your statistics and implementing the best kind of generation for each scenario to enhance the player experience.
Do you love another method that I didn’t cover? Have questions about random number generation in your own game’s design? Leave me a comment below, and I’ll do my best to get back to you.
