Sunday, December 31, 2017
Saturday, December 30, 2017
Friday, December 29, 2017
How to Send Emails in Laravel
In this article, we're going to explore the Mail API in the Laravel web framework. Laravel takes advantage of the popular SwiftMailer library, which is easy to use and comes with a variety of email drivers to choose from. In the latter stages of the article, we'll go through an in-depth demonstration of the concepts discussed in the first half of the article.
Setting Up the Prerequisites
Laravel implements a wrapper on top of the SwiftMailer library that makes email management very easy to configure and use at the same time. You can find the default mail settings at config/mail.php.
<?php
return [
/*
|--------------------------------------------------------------------------
| Mail Driver
|--------------------------------------------------------------------------
|
| Laravel supports both SMTP and PHP's "mail" function as drivers for the
| sending of e-mail. You may specify which one you're using throughout
| your application here. By default, Laravel is setup for SMTP mail.
|
| Supported: "smtp", "sendmail", "mailgun", "mandrill", "ses",
| "sparkpost", "log", "array"
|
*/
'driver' => env('MAIL_DRIVER', 'sendmail'),
/*
|--------------------------------------------------------------------------
| SMTP Host Address
|--------------------------------------------------------------------------
|
| Here you may provide the host address of the SMTP server used by your
| applications. A default option is provided that is compatible with
| the Mailgun mail service which will provide reliable deliveries.
|
*/
'host' => env('MAIL_HOST', 'smtp.mailgun.org'),
/*
|--------------------------------------------------------------------------
| SMTP Host Port
|--------------------------------------------------------------------------
|
| This is the SMTP port used by your application to deliver e-mails to
| users of the application. Like the host we have set this value to
| stay compatible with the Mailgun e-mail application by default.
|
*/
'port' => env('MAIL_PORT', 587),
/*
|--------------------------------------------------------------------------
| Global "From" Address
|--------------------------------------------------------------------------
|
| You may wish for all e-mails sent by your application to be sent from
| the same address. Here, you may specify a name and address that is
| used globally for all e-mails that are sent by your application.
|
*/
'from' => [
'address' => env('MAIL_FROM_ADDRESS', 'hello@example.com'),
'name' => env('MAIL_FROM_NAME', 'Example'),
],
/*
|--------------------------------------------------------------------------
| E-Mail Encryption Protocol
|--------------------------------------------------------------------------
|
| Here you may specify the encryption protocol that should be used when
| the application send e-mail messages. A sensible default using the
| transport layer security protocol should provide great security.
|
*/
'encryption' => env('MAIL_ENCRYPTION', 'tls'),
/*
|--------------------------------------------------------------------------
| SMTP Server Username
|--------------------------------------------------------------------------
|
| If your SMTP server requires a username for authentication, you should
| set it here. This will get used to authenticate with your server on
| connection. You may also set the "password" value below this one.
|
*/
'username' => env('MAIL_USERNAME'),
'password' => env('MAIL_PASSWORD'),
/*
|--------------------------------------------------------------------------
| Sendmail System Path
|--------------------------------------------------------------------------
|
| When using the "sendmail" driver to send e-mails, we will need to know
| the path to where Sendmail lives on this server. A default path has
| been provided here, which will work well on most of your systems.
|
*/
'sendmail' => '/usr/sbin/sendmail -bs',
/*
|--------------------------------------------------------------------------
| Markdown Mail Settings
|--------------------------------------------------------------------------
|
| If you are using Markdown based email rendering, you may configure your
| theme and component paths here, allowing you to customize the design
| of the emails. Or, you may simply stick with the Laravel defaults!
|
*/
'markdown' => [
'theme' => 'default',
'paths' => [
resource_path('views/vendor/mail'),
],
],
];
When it comes to sending mails, Laravel supports different drivers to choose from. As you can see, the default MAIL_DRIVER is set to smtp.
If you are going to use the smtp driver to send mails then you're also required to set other related settings like MAIL_HOST, MAIL_PORT, MAIL_ENCRYPTION, MAIL_USERNAME, and MAIL_PASSWORD.
On the other hand, if you are going to use the sendmail driver, then you want to make sure that the sendmail system path is set to the correct value in the config/mail.php file.
You can also set the from address that will be used while sending mails under the from key. And finally, if you want to use Markdown-based email rendering, you can set those settings under the markdown key.
The cherry on the top is that you could also use third-party email service providers like Mailgun, Mandrill, SES, and SparkPost. If you are using one of those services, you need to make sure that you set the corresponding settings in the config/services.php file.
So that was a basic introduction to the mail API related settings in Laravel. From the next section onwards, we'll go through a custom example that shows you how to send emails.
Create the Mailable Class
In this section, we'll create the mailable class, which will be used to send emails. The mailable class is responsible for sending emails using a mailer that's configured in the config/mail.php file. In fact, Laravel already provides an artisan command that allows us to create a base template.
php artisan make:mail DemoEmail
That should create a blank email template at app/Mail/DemoEmail.php, as shown in the following snippet.
<?php
namespace App\Mail;
use Illuminate\Bus\Queueable;
use Illuminate\Mail\Mailable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Contracts\Queue\ShouldQueue;
class DemoEmail extends Mailable
{
use Queueable, SerializesModels;
/**
* Create a new message instance.
*
* @return void
*/
public function __construct()
{
//
}
/**
* Build the message.
*
* @return $this
*/
public function build()
{
return $this->view('view.name');
}
}
Let's replace the contents of that file with the following.
<?php
namespace App\Mail;
use Illuminate\Bus\Queueable;
use Illuminate\Mail\Mailable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Contracts\Queue\ShouldQueue;
class DemoEmail extends Mailable
{
use Queueable, SerializesModels;
/**
* The demo object instance.
*
* @var Demo
*/
public $demo;
/**
* Create a new message instance.
*
* @return void
*/
public function __construct($demo)
{
$this->demo = $demo;
}
/**
* Build the message.
*
* @return $this
*/
public function build()
{
return $this->from('sender@example.com')
->view('mails.demo')
->text('mails.demo_plain')
->with(
[
'testVarOne' => '1',
'testVarTwo' => '2',
])
->attach(public_path('/images').'/demo.jpg', [
'as' => 'demo.jpg',
'mime' => 'image/jpeg',
]);
}
}
There are two important methods the mailable class generally implements—__construct and build. The __construct method is used to initialize objects that you're supposed to use in the email template. On the other hand, the build method is used to initialize more email-specific values like from, view template, attachments and similar.
In our case, we've passed the $demo object as a constructor argument, and it's assigned to the demo public property.
In the build method, we've initialized an email-specific configuration.
- The
fromis used to set an email address that'll be used as a from address. - Using the
viewmethod, you can set the email template that will be used while sending an email using this mailable. In our case, we've set it tomails.demo, and it means that you need to create a view template file atresources/views/mails/demo.blade.php. - Next, the
textmethod is used to set up the plain text version of an email template. - As we've just discussed, the
__constructmethod is used to set up objects that'll be used in the email template, you can also use thewithmethod that allows you to set the view data of a message. - Next, we've used the
attachmethod to attach an image with a message.
Of course, we need to create email templates that we're supposed to use while sending emails. Go ahead and create a file resources/views/mails/demo.blade.php as shown in the following snippet.
Hello <i></i>, <p>This is a demo email for testing purposes! Also, it's the HTML version.</p> <p><u>Demo object values:</u></p> <div> <p><b>Demo One:</b> </p> <p><b>Demo Two:</b> </p> </div> <p><u>Values passed by With method:</u></p> <div> <p><b>testVarOne:</b> </p> <p><b>testVarTwo:</b> </p> </div> Thank You, <br/> <i></i>
Also, let's create the plain text version of that file at resources/views/mails/demo_plain.blade.php.
Hello , This is a demo email for testing purposes! Also, it's the HTML version. Demo object values: Demo One: Demo Two: Values passed by With method: testVarOne: testVarOne: Thank You,
So that was the mailable class at your disposal, and we've not done yet as we need to use the Mail facade to actually send mails. In the very next section, we'll explore how you can use the Mail Facade to send emails using the DemoEmail Mailable class that was just created in this section.
Wrapping Up
In this section, we'll create an example to demonstrate how you can use the Mailable class that was created in the last section.
Let's create a controller file at app/Http/Controllers/MailController.php with the following contents.
<?php
namespace App\Http\Controllers;
use App\Http\Controllers\Controller;
use App\Mail\DemoEmail;
use Illuminate\Support\Facades\Mail;
class MailController extends Controller
{
public function send()
{
$objDemo = new \stdClass();
$objDemo->demo_one = 'Demo One Value';
$objDemo->demo_two = 'Demo Two Value';
$objDemo->sender = 'SenderUserName';
$objDemo->receiver = 'ReceiverUserName';
Mail::to("receiver@example.com")->send(new DemoEmail($objDemo));
}
}
It's important to note that we've included the Illuminate\Support\Facades\Mail Facade that will be used to send an email. In the send method, the following statement is responsible for sending an email by initialising the App\Mail\DemoEmail Mailable in the first place.
Mail::to("receiver@example.com")->send(new DemoEmail($objDemo));
The to method of the Illuminate\Support\Facades\Mail Facade returns an instance of the \Illuminate\Mail\PendingMail class, which already contains an appropriate mailer configured in the config/mail.php file.
And finally, we use the send method of the \Illuminate\Mail\PendingMail class that sends an actual email.
To test it, let's add an associated route in the routes/web.php file.
// Email related routes
Route::get('mail/send', 'MailController@send');
And with that in place, you can run the http://ift.tt/2BSrKyg URL to see if it works as expected.
On the other hand, if you want to test your email templates quickly, without sending actual emails, there's a provision in Laravel that allows you to log all outgoing emails.
To achieve that, you need to set the value of MAIL_DRIVER to log in the config/mail.php file. Next, you could run the aforementioned URL and inspect the log file to check if the email template was logged there.
If everything goes fine, you should see an email being logged to the storage/logs/laravel.log file.
That's pretty much it as far as the mail feature is concerned in Laravel, and that concludes this article as well.
Conclusion
Today, we went through the mail API that comes built into Laravel, and it also supports a variety of drivers as well.
Starting with basic concepts, we implemented the mailable class that is an essential element in the mail API in Laravel as we moved on. At the end, we also tested the mailable class by creating a custom controller to see if it actually works.
If you're just getting started with Laravel or looking to expand your knowledge, site, or application with extensions, we have a variety of things you can study in Envato Market.
I would love to know your feedback in the form of queries and comments using the feed below!
Thursday, December 28, 2017
Wednesday, December 27, 2017
Tuesday, December 26, 2017
Components in Vue.js
Introduction
Vue.js components are important to understand as you work with Vue.js. In this tutorial we are going to dig into Vue.js components, understanding the basics and applying them to an application. Let's get started.
What Are Components?
Components enable us to break the complexity of an application into tiny chunks. For example, a typical web application will have sections like header, sidebar, content, and footer.
Vue.js allows us to break each of these sections into separate modular code called components. These components can be extended, and then attached to the application you are working on. Components are a great way of reusing your code throughout your application.
Let's say you have a blog application, and you would like to display a list of blog posts. Using a Vue component, you can do:
<blog-post></blog-post>
Vue handles the rest.
Create a simple HTML page that mounts a Vue instance to your DOM element. You will use this to learn about components. Here is what the sample HTML page should look like:
<!DOCTYPE html>
<html>
<head>
<title>VueJs Components</title>
</head>
<body>
<!-- Div where Vue instance will be mounted -->
<div id="app"></div>
<!-- Vue.js is included in your page via CDN -->
<script src="https://unpkg.com/vue"></script>
<script>
// A new Vue instance is created and mounted to your div element
new Vue({
el: '#app',
data: {
domain: 'Tutsplus'
},
template: '<p>Welcome to </p>
});
</script>
</body>
</html>
In the above, you created a simple Vue instance, with no component factor in the code. Now if you want the welcome message to appear twice, how do you do it? Try getting that to work.
Your guess might be to have the div where the Vue instance is mounted appear twice. That will not work. Try changing it from id to class so you have this:
<!DOCTYPE html>
<html>
<head>
<title>VueJs Components</title>
</head>
<body>
<!-- Div where Vue instance will be mounted -->
<div class="app"></div>
<div class="app"></div>
<!-- Vue.js is included in your page via CDN -->
<script src="https://unpkg.com/vue"></script>
<script>
// A new Vue instance is created and mounted to your div element
new Vue({
el: '.app',
data: {
domain: 'Tutsplus'
},
template: '<p>Welcome to </p>
});
</script>
</body>
</html>
It still will not work!
The only way to get around this is to create a component. How do you create a component?
Components are created using the Vue.component() constructor. This constructor takes two parameters: the name of your component (which can also be called the tag name), and an object containing options for your component.
Let's create a component using what you have above.
Vue.component('welcome-message', {
data: function() {
return {
domain: 'Tutsplus'
}
},
template: '<p>Welcome to </p>'
})
In the above, the component name is called welcome-message. Your component can have whatever name you choose. Yet it is important that the name does not interfere with any HTML tags, as you will not want to override it.
The options object passed to the constructor contains the data and template. When creating components, your data should be a function, as you can see above. The data being held should then be returned as an object.
In situations where there are no data to pass, you can pass only the template like so:
Vue.component('welcome-message', {
template: '<p>Welcome to Tutsplus</p>'
})
With that done, you can use your component in your application by calling it like a regular HTML element using the name you passed to the constructor. It is called like this: <welcome-message></welcome-message>.
To output the template more than once, you call the component as many times as you want—as I did below.
<!DOCTYPE html>
<html>
<head>
<title>VueJs Components</title>
</head>
<body>
<!-- Div where Vue instance will be mounted -->
<div id="app">
<welcome-message></welcome-message>
<welcome-message></welcome-message>
<welcome-message></welcome-message>
<welcome-message></welcome-message>
</div>
<!-- Vue.js is included in your page via CDN -->
<script src="https://unpkg.com/vue"></script>
<script>
Vue.component('welcome-message', {
data: function() {
return {
domain: 'Tutsplus'
}
},
template: '<p>Welcome to </p>'
})
// A new Vue instance is created and mounted to your div element
new Vue({
el: '#app'
});
</script>
</body>
</html>
From the above, this will display the welcome message four times.
Store Data in Components
Above I mentioned that data has to be a function, and the information it holds has to be returned as an object. Why is it like that?
When returned data is not an object, the components that make use of that data share the same source: shared data. Thus a change in data for one component affects the other component. This is not the same when the data is returned as an object.
It is important to see how this works practically. First, let's see a situation where data is returned as an object.
<!DOCTYPE html>
<html>
<head>
<title>VueJs Components</title>
</head>
<body>
<!-- Div where Vue instance will be mounted -->
<div id="app">
<welcome-message></welcome-message>
<welcome-message></welcome-message>
</div>
<!-- Vue.js is included in your page via CDN -->
<script src="https://unpkg.com/vue"></script>
<script>
var data = { name: 'Henry' }
Vue.component('welcome-message', {
data: function() {
return data
},
template: '<p>Hello , welcome to TutsPlus (<button @click="changeName">Change Name</button>)</p>',
methods :{
changeName: function() {
this.name = 'Mark'
}
}
})
// A new Vue instance is created and mounted to your div element
new Vue({
el: '#app'
});
</script>
</body>
</html>
Can you guess what is happening above?
There are two components, and both components share the same data source as the data is not returned as an object. How do you prove I am right? When you view the above page from your browser, you will see that a change in one component results in a change in the data of the other component. How is it supposed to be?
Like this:
<!DOCTYPE html>
<html>
<head>
<title>VueJs Components</title>
</head>
<body>
<!-- Div where Vue instance will be mounted -->
<div id="app">
<welcome-message></welcome-message>
<welcome-message></welcome-message>
</div>
<!-- Vue.js is included in your page via CDN -->
<script src="https://unpkg.com/vue"></script>
<script>
Vue.component('welcome-message', {
data: function() {
return {
name: 'Henry'
}
},
template: '<p>Hello , welcome to TutsPlus (<button @click="changeName">Change Name</button>)</p>',
methods :{
changeName: function() {
this.name = 'Mark'
}
}
})
// A new Vue instance is created and mounted to your div element
new Vue({
el: '#app'
});
</script>
</body>
</html>
Here the data is returned as an object, and a change in one component does not affect the other. The function is executed for the individual component. When building your applications, it is important you do not forget this.
Creating and Using Components
Using the lessons learned thus far, let us implement this in a new Vue.js project started from scratch using vue-cli. If you do not have vue-cli installed on your machine, you can do that by running:
npm install -g vue-cli
Start your new Vue.js project:
vue init webpack vue-component-app
Navigate to your application, install the dependencies, and run your dev server using the commands below.
cd vue-component-app npm install npm run dev
First, you'll rename the HelloWorld component that was created when you initialized your application to Hello.vue. You'll then register this component as a global component to be used in your application.
So your Hello component should look like this.
#src/components/Hello.vue
<template>
<div class="hello">
<p>Welcome to TutsPlus </p>
<hr>
<button @click="changeName">Change Display Name</button>
</div>
</template>
<script>
export default {
name: 'Hello',
data () {
return {
name: 'Henry'
}
},
methods: {
changeName () {
this.name = 'Mark'
}
}
}
</script>
<!-- Add "scoped" attribute to limit CSS to this component only -->
<style scoped>
h1, h2 {
font-weight: normal;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
a {
color: #42b983;
}
</style>
Nothing serious is happening here. You have your welcome text displaying the welcome message and a name which is passed as data. When the button underneath the welcome message gets clicked, the changeName method is called. This changes the name from Henry to Mark.
To use this component globally it has to be registered. Can you guess where that will be done? If you said main.js, you are absolutely correct.
To register a component, you import it and then use the Vue.component() constructor to set it up. Try that on your own.
I bet you got that to work. Here is how things should look in your main.js file.
#src/main.js
// The Vue build version to load with the `import` command
// (runtime-only or standalone) has been set in webpack.base.conf with an alias.
import Vue from 'vue'
import App from './App'
import Home from './components/Hello'
Vue.config.productionTip = false
Vue.component('display-name', Home)
/* eslint-disable no-new */
new Vue({
el: '#app',
template: '<App/>',
components: { App }
})
Nothing new here except the line that imports your Hello component. The component is then registered using the constructor. Finally, for this part, the component needs to be displayed using the component name you entered. In this case, the component is display-name. This will be done in your App.vue file.
Open up src/App.vue and make it look like this.
#src/App.vue
<template>
<div id="app">
<display-name/>
</div>
</template>
<script>
export default {
}
</script>
<style>
#app {
font-family: 'Avenir', Helvetica, Arial, sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-align: center;
color: #2c3e50;
margin-top: 60px;
}
</style>
With your server on, point your browser to http://localhost:8080. Click on the button and the name should change.
Let's see how to use a component locally.
Create a file called Detail.vue in your components directory. This component will not do anything special—it will be used in the Hello component.
Make your Detail.vue file look like this.
#src/components/Detail.vue
<template>
<div class="hello">
<p>This component is imported locally.</p>
</div>
</template>
<script>
export default {
name: 'Detail'
}
</script>
<!-- Add "scoped" attribute to limit CSS to this component only -->
<style scoped>
h1, h2 {
font-weight: normal;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
a {
color: #42b983;
}
</style>
To use it in the Hello component, you start by importing it, as you did with the Hello component. Next, you register it, but this time you will not make use of the Vue.component() constructor.
You register it using the components object inside your exports. The name of the component which will be used as the element tag has to be passed as a value to the object. With that done, you can now use the element tag to output the component.
To understand all that, here is how the Hello component should look.
#src/components/Hello.vue
<template>
<div class="hello">
<p>Welcome to TutsPlus </p>
<hr>
<button @click="changeName">Change Display Name</button>
<!-- Detail component is outputted using the name it was registered with -->
<Detail/>
</div>
</template>
<script>
// Importation of Detail component is done
import Detail from './Detail'
export default {
name: 'HelloWorld',
data () {
return {
name: 'Henry'
}
},
methods: {
changeName () {
this.name = 'Mark'
}
},
/**
* Detail component gets registered locally.
* This component can only be used inside the Hello component
* The value passed is the name of the component
*/
components: {
Detail
}
}
</script>
<!-- Add "scoped" attribute to limit CSS to this component only -->
<style scoped>
h1, h2 {
font-weight: normal;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
a {
color: #42b983;
}
</style>
Refresh your page to see the new page.
Scoping Component Styles
Vue.js allows you to give global and local styling for your components. What do I mean? There might be scenarios where you want certain elements in a component to be styled differently from their counterparts in another component. Vue.js has got you covered on that.
A good example is available in the little app you just built. The styles in your App.vue are global; how is that possible? Open your App.vue, and the style section looks like this.
<style>
#app {
font-family: 'Avenir', Helvetica, Arial, sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
text-align: center;
color: #2c3e50;
margin-top: 60px;
}
</style>
This is different from that of Detail.vue, which looks like this.
<style scoped>
h1, h2 {
font-weight: normal;
}
ul {
list-style-type: none;
padding: 0;
}
li {
display: inline-block;
margin: 0 10px;
}
a {
color: #42b983;
}
</style>
Adding scoped to the style tag makes all the difference. Try editing one of the component styles by removing scoped, and you will see how that works out.
Conclusion
That was a long one, and I trust you enjoyed it.
Now you know how to work effectively with components. You understand how to construct a component in an existing application. When working with vue-cli, you can also create and use components—locally and globally. When you want to use a particular style for a component, you saw how to do that using scoped.
You can now go on to build a complex Vue.js application that uses components. Understand that Vue.js allows you to reuse code—your headers, footer, login panel, and search bar can be used as components.
Monday, December 25, 2017
Friday, December 22, 2017
Thursday, December 21, 2017
Wednesday, December 20, 2017
5 Tips for Stakeholder “Buy-in”
UX designers don’t work alone; they need the support of their engineering or business counterparts. Here are some tips on how to smoothly navigate collaboration and get buy-in on your ideas!
What is “Buy-in”?
If you Google “buy-in” you’ll most likely find that it’s a trading term, referring to the purchase of a controlling number of a company’s shares.
But that’s not what this article is about. We’re talking about people “buying into” an idea or concept. If a team member or stakeholder has “bought in”, then they agree with and accept your suggestions.
But how do we get buy-in?
1. Include Stakeholders in Early in the Process
Including the major stakeholders into your design process early on sets the expectation that they are equal team members rather than gatekeepers. They may also bring a unique perspective to the constraints and edge cases that designers need to incorporate.
Don’t neglect including partners early on in your product journey, as this is an excellent opportunity to educate them on the design process.
2. Speak to Tangible Ideas
One advantage designers have is the ability to quickly make lo-fidelity mock-ups. A well-known adage is that “a picture is worth a thousand words”. This holds true for conversations where a team is in disagreement.
Sketch out your ideas quickly rather than spending endless hours discussing them with hand-waving. Something tangible is easier to have conversations over and eventually test than words alone. Don’t spend time polishing your designs early on in the process when you should be exploring and iterating quickly.
3. Have Empathy for Trade-offs
When engaging your engineers, consider what implications your design options have for their time and resource investment. If what you are proposing is beyond the time they have to deliver, they will likely push back on your designs. Having a general sense of what takes more or less engineering efforts will allow you to speak to what is a priority in their minds.
Note: This doesn’t mean to always cater to what is easiest to build–rather knowing what key experiences are worth the time investment as there are trade-offs to be made if resources are scarce.
4. Convince With Data
When you aren’t able to convince your team of something you strongly believe is the right direction, bring data with customer evidence to support your point.
Re-framing your point of view in terms of real users or business success is a value that most would prioritize above subjective opinions.
5. Focus on the Problem
When working in a team with many different personalities, it is easy to get caught up in the team dynamic and politics that distract from the project’s progress. Instead, focus back on the problem at hand. As a UX designer, your purpose is to bring a human-centered point of view to every problem you encounter.
Conclusion
Make your ideas more tangible and test them with real users early and often. Then you can get feedback to iterate again–equipped with more knowledge, even with complex problems, information directs you toward making better design decisions that (hopefully) everyone can agree on.
Tuesday, December 19, 2017
Monday, December 18, 2017
Deferring Tasks in Laravel Using Queues
In this article, we're going to explore the Queue API in the Laravel web framework. It allows you to defer resource-intensive tasks during script execution to enhance the overall end user experience. After introducing the basic terminology, I'll demonstrate it by implementing a real-world example.
Page load time is an important aspect of any successful website, and one should not overlook the importance of that as it affects the SEO of the site and the overall end user experience as well. More often than not, you end up needing to debug web pages with long page load times. Of course, there are different approaches you could use to rectify this issue.
Upon investigation, you often realize that there are certain code blocks causing a delay in the page execution. The next thing you could try is identifying blocks that can be deferred for processing and that have no real impact on the end result of the current page. That should really improve the overall web page speed as we've eliminated code blocks that were causing a delay.
Today, we're going to explore a similar concept in the context of the Laravel web framework. In fact, Laravel already provides a useful built-in API that allows us to defer the processing of tasks—the Queue API. Without wasting much of your time, I'll go ahead and discuss the basic elements of the Queue API.
Drivers, Connections, Queues, and Jobs
The basic purpose of the Queue API is to run jobs that are added in a queue. Next, the queue could belong to a specific connection, and that connection may belong to a specific queue driver configured with that connection itself. Let's briefly try to understand what I've just said.
Queue Drivers
In the same way you would have used a different driver for your database connection, you could also choose from a variety of different queue drivers. The Queue API supports different adapters like database, beanstalkd, sqs, and redis.
The queue driver is just a place that is used to store queue-related information. So if you're using a database queue driver, for example, the new job will be added in the jobs table in the database. On the other hand, if you've configured redis as the default queue driver, the job will be added to the redis server.
The Queue API also provides two special queue drivers for testing purposes—sync and null. The sync queue driver is used to execute a queue job immediately, while the null queue driver is used to skip a job so that it won't be executed at all.
Connections
When you configure the Queue API for the first time, you need to specify a default connection that should be used for default queue processing. At the very least, the connection is expected to provide the following information:
- the queue driver that will be used
- the queue driver's specific configuration values
- the default queue name in which the job will be added
Queues
When you add any job into a queue, it'll be added into the default queue. In fact, that should be fine in most cases, unless you have jobs that need to be given higher priority over other jobs. In that case, you could create a queue named high and place the higher priority jobs in that particular queue.
When you run a queue worker that processes queued jobs, you could optionally pass the --queue parameter, which allows you to list queue names in the order in which they need to be processed. For example, if you specify --queue=high,default, it will first process jobs in the high queue, and once it's completed it fetches jobs in the default queue.
Jobs
A job in the Queue API is a task that's deferred from the main execution flow. For example, if you want to create a thumbnail when the user uploads an image from the front-end, you could create a new job that handles the thumbnail processing. In this way, you could defer the task of thumbnail processing from the main execution flow.
That was a basic introduction to the Queue API terminology. From the next section onwards, we'll explore how to create a custom queue job and run it by using a Laravel queue worker.
Create Your First Queue Job
By now, you should feel confident about queue jobs. From this section onwards, we're going to implement a real-world example that demonstrates the concept of queue jobs in Laravel.
More often than not, you end up in the situation where you need to create different thumbnail versions of an image uploaded by a user. In most cases, the developer tries to process it in real time so that different versions of images are created right away when the user uploads an image.
It seems to be a reasonable approach if you're going to create a couple of versions and it doesn't take too much time in the first place. On the other hand, if you're dealing with an application that requires heavy processing and thus eats up more resources, real-time processing could end up in a bad user experience.
The obvious option that pops up in your mind in the first place is to defer processing of the thumbnail generation as late as possible. The simplest approach you could implement in this specific scenario is to set a cron job that triggers processing at regular intervals, and you should be fine.
A much better approach, on the other hand, is to defer and push the task into a queue, and let the queue worker process it when it gets a chance to do so. In a production environment, the queue worker is a daemon script that's always running and processing tasks in a queue. The obvious benefit of this approach is a much better end user experience, and you don't have to wait for the cron run as the job will be processed as soon as possible.
I guess that's enough theory to get started with an actual implementation.
In our case, we're going to use the database queue driver, and it requires us to create the jobs table in the database. The jobs table holds all the jobs that need to be processed in the next queue worker run.
Before we go ahead and create the jobs table, let's change the default queue configuration from sync to database in the config/queue.php file.
...
...
/*
|--------------------------------------------------------------------------
| Default Queue Driver
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for each one. Here you may set the default queue driver.
|
| Supported: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/
'default' => env('QUEUE_DRIVER', 'database'),
...
...
In fact, Laravel already provides an artisan command that helps us to create the jobs table. Run the following command in the root of your Laravel application, and it should create the necessary database migration that creates the jobs table.
$php artisan queue:table
The migration file that's generated at database/migrations/YYYY_MM_DD_HHMMSS_create_jobs_table.php should look like this:
<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateJobsTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('jobs', function (Blueprint $table) {
$table->bigIncrements('id');
$table->string('queue');
$table->longText('payload');
$table->unsignedTinyInteger('attempts');
$table->unsignedInteger('reserved_at')->nullable();
$table->unsignedInteger('available_at');
$table->unsignedInteger('created_at');
$table->index(['queue', 'reserved_at']);
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::dropIfExists('jobs');
}
}
Next, let's run the migrate command so that it actually creates the jobs table in a database.
php artisan migrate
That's it as far as the jobs migration is concerned.
Next, let's create the Image model that will be used to manage images uploaded by the end user. The image model also requires an associated database table, so we'll use the --migrate option while creating the Image model.
php artisan make:model Image --migration
The above command should create the Image model class and an associated database migration as well.
The Image model class should look like this:
<?php
// app/Image.php
namespace App;
use Illuminate\Database\Eloquent\Model;
class Image extends Model
{
//
}
And the database migration file should be created at database/migrations/YYYY_MM_DD_HHMMSS_create_images_table.php. We also want to store the original path of the image uploaded by the end user. Let's revise the code of the Image database migration file to look like the following.
<?php
// database/migrations/YYYY_MM_DD_HHMMSS_create_images_table.php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateImagesTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('images', function (Blueprint $table) {
$table->increments('id');
$table->timestamps();
$table->string('org_path');
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::dropIfExists('images');
}
}
As you can see, we've added the $table->string('org_path') column to store the path of the original image. Next, you just need to run the migrate command to actually create that table in the database.
$php artisan migrate
And that's it as far as the Image model is concerned.
Next, let's create an actual queue job that's responsible for processing image thumbnails. For the thumbnail processing, we're going to use a very popular image processing library—Intervention Image.
To install the Intervention Image library, go ahead and run the following command at the root of your application.
$php composer.phar require intervention/image
Now, it's time to create the Job class, and we'll use an artisan command to do that.
$php artisan make:job ProcessImageThumbnails
That should create the Job class template at app/Jobs/ProcessImageThumbnails.php. Let's replace the contents of that file with the following.
<?php
// app/Jobs/ProcessImageThumbnails.php
namespace App\Jobs;
use App\Image as ImageModel;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\DB;
class ProcessImageThumbnails implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $image;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct(ImageModel $image)
{
$this->image = $image;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
// access the model in the queue for processing
$image = $this->image;
$full_image_path = public_path($image->org_path);
$resized_image_path = public_path('thumbs' . DIRECTORY_SEPARATOR . $image->org_path);
// create image thumbs from the original image
$img = \Image::make($full_image_path)->resize(300, 200);
$img->save($resized_image_path);
}
}
When the queue worker starts processing any job, it looks for the handle method. So it's the handle method that holds the main logic of your job.
In our case, we need to create a thumbnail of an image uploaded by the user. The code of the handle method is pretty straightforward—we retrieve an image from the ImageModel model and create a thumbnail using the Intervention Image library. Of course, we need to pass the corresponding Image model when we dispatch our job, and we'll see it in a moment.
To test our newly created job, we'll create a simple upload form that allows the user to upload an image. Of course, we won't create image thumbnails right away; we'll defer that task so that it could be processed by the queue worker.
Let's create a controller file at app/Http/Controllers/ImageController.php as shown below.
<?php
namespace App\Http\Controllers;
use App\Image;
use App\Jobs\ProcessImageThumbnails;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Redirect;
use App\Http\Controllers\Controller;
use Validator;
class ImageController extends Controller
{
/**
* Show Upload Form
*
* @param Request $request
* @return Response
*/
public function index(Request $request)
{
return view('upload_form');
}
/**
* Upload Image
*
* @param Request $request
* @return Response
*/
public function upload(Request $request)
{
// upload image
$this->validate($request, [
'demo_image' => 'required|image|mimes:jpeg,png,jpg,gif,svg|max:2048',
]);
$image = $request->file('demo_image');
$input['demo_image'] = time().'.'.$image->getClientOriginalExtension();
$destinationPath = public_path('/images');
$image->move($destinationPath, $input['demo_image']);
// make db entry of that image
$image = new Image;
$image->org_path = 'images' . DIRECTORY_SEPARATOR . $input['demo_image'];
$image->save();
// defer the processing of the image thumbnails
ProcessImageThumbnails::dispatch($image);
return Redirect::to('image/index')->with('message', 'Image uploaded successfully!');
}
}
Let's create an associated view file at resources/views/upload_form.blade.php.
<!DOCTYPE html>
<html lang="">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="csrf-token" content="" />
<title>Laravel</title>
<!-- Fonts -->
<link href="http://ift.tt/2dHeAcp" rel="stylesheet" type="text/css">
<!-- Styles -->
<style>
html, body {
background-color: #fff;
color: #636b6f;
font-family: 'Raleway', sans-serif;
font-weight: 100;
height: 100vh;
margin: 0;
}
.full-height {
height: 100vh;
}
.flex-center {
align-items: center;
display: flex;
justify-content: center;
}
.position-ref {
position: relative;
}
.top-right {
position: absolute;
right: 10px;
top: 18px;
}
.content {
text-align: center;
}
.title {
font-size: 84px;
}
.links > a {
color: #636b6f;
padding: 0 25px;
font-size: 12px;
font-weight: 600;
letter-spacing: .1rem;
text-decoration: none;
text-transform: uppercase;
}
.m-b-md {
margin-bottom: 30px;
}
.alert {
color: red;
font-weight: bold;
margin: 10px;
}
.success {
color: blue;
font-weight: bold;
margin: 10px;
}
</style>
</head>
<body>
<div class="flex-center position-ref full-height">
@if (Route::has('login'))
<div class="top-right links">
@if (Auth::check())
<a href="">Home</a>
@else
<a href="">Login</a>
<a href="">Register</a>
@endif
</div>
@endif
<div class="content">
<div class="m-b-md">
<h1 class="title">Demo Upload Form</h1>
@if ($errors->any())
<div class="alert alert-danger">
<ul>
@foreach ($errors->all() as $error)
<li></li>
@endforeach
</ul>
</div>
@endif
@if (session('message'))
<div class="success">
</div>
@endif
<form method="post" action="" enctype="multipart/form-data">
<div>
<input type="file" name="demo_image" />
</div>
<br/>
<div>
<input type="hidden" name="_token" value="">
<input type="submit" value="Upload Image"/>
</div>
</form>
</div>
</div>
</div>
</body>
</html>
Finally, let's add routes for the index and upload actions in the routes/web.php file.
Route::get('image/index', 'ImageController@index');
Route::post('image/upload', 'ImageController@upload');
In the ImageController controller, the index method is used to render an upload form.
public function index(Request $request)
{
return view('upload_form');
}
When the user submits a form, the upload method is invoked.
public function upload(Request $request)
{
// upload image
$this->validate($request, [
'demo_image' => 'required|image|mimes:jpeg,png,jpg,gif,svg|max:2048',
]);
$image = $request->file('demo_image');
$input['demo_image'] = time().'.'.$image->getClientOriginalExtension();
$destinationPath = public_path('/images');
$image->move($destinationPath, $input['demo_image']);
// make db entry of that image
$image = new Image;
$image->org_path = 'images' . DIRECTORY_SEPARATOR . $input['demo_image'];
$image->save();
// defer the processing of the image thumbnails
ProcessImageThumbnails::dispatch($image);
return Redirect::to('image/index')->with('message', 'Image uploaded successfully!');
}
At the beginning of the upload method, you'll notice the usual file upload code that moves the uploaded file to the public/images directory. Next, we insert a database record using the App/Image model.
Finally, we use the ProcessImageThumbnails job to defer the thumbnail processing task. It's important to note that it's the dispatch method that's used to defer a task. At the end, the user is redirected to the upload page with a success message.
At this point in time, the job is added to the jobs table for processing. Let's confirm it by issuing the following query.
mysql> select * FROM lvl_jobs;
| 1 | default | {"displayName":"App\\Jobs\\ProcessImageThumbnails","job":"Illuminate\\Queue\\CallQueuedHandler@call","maxTries":null,"timeout":null,"data":{"commandName":"App\\Jobs\\ProcessImageThumbnails","command":"O:31:\"App\\Jobs\\ProcessImageThumbnails\":5:{s:8:\"\u0000*\u0000image\";O:45:\"Illuminate\\Contracts\\Database\\ModelIdentifier\":2:{s:5:\"class\";s:9:\"App\\Image\";s:2:\"id\";i:2;}s:6:\"\u0000*\u0000job\";N;s:10:\"connection\";N;s:5:\"queue\";N;s:5:\"delay\";N;}"}} | 0 | NULL | 1510219099 | 1510219099 |
You must be wondering, what does it take to process a job then? Don't worry—that's what we're going to discuss in the very next section.
Queue Worker
The job of the Laravel queue worker is to process jobs that are queued up for processing. In fact, there's an artisan command that helps us to start the queue worker process.
$php artisan queue:work
As soon as you run that command, it processes pending jobs. In our case, it should process the ProcessImageThumbnails job that was queued up when the user uploaded an image earlier.
$php artisan queue:work [YYYY-MM-DD HHMMSS] Processing: App\Jobs\ProcessImageThumbnails [YYYY-MM-DD HHMMSS] Processed: App\Jobs\ProcessImageThumbnails
You would have noticed that when you start a queue worker, it keeps running until you kill it manually or close the terminal. In fact, it's waiting for the next job to be processed. As soon as there's a new job in the queue, it'll be processed right away if the queue worker is running.
Of course, we can't keep it running that way, so we need to find a way for the queue worker to run permanently in the background.
To our rescue, there are several process management tools out there you could choose from. To name a few, here's a list:
- Circus
- daemontools
- Monit
- Supervisor
- Upstart
You should choose a tool that you're comfortable with to manage the Laravel queue worker. Basically, we want to make sure that the queue worker should run indefinitely so that it processes queued jobs right away.
So that's the Queue API at your disposal. You can use it in your day-to-day development to defer time-consuming tasks for the betterment of the end user experience.
Conclusion
In this article, we discussed the Queue API in Laravel, which is really helpful should you wish to defer processing of resource-consuming tasks.
We started with a basic introduction to the Queue API, which involved a discussion of connections, queues, and jobs. In the second half of the article, we created a custom queue job that demonstrated how you could use the Queue API in the real world.
For those of you who are either just getting started with Laravel or looking to expand your knowledge, site, or application with extensions, we have a variety of things you can study in Envato Market.
Feel free to use the feedback form below to post your queries and suggestions.
Friday, December 15, 2017
Testing Data-Intensive Code With Go, Part 5
Overview
This is part five out of five in a tutorial series on testing data-intensive code. In part four, I covered remote data stores, using shared test databases, using production data snapshots, and generating your own test data. In this tutorial, I'll go over fuzz testing, testing your cache, testing data integrity, testing idempotency, and missing data.
Fuzz Testing
The idea of fuzz testing is to overwhelm the system with lots of random input. Instead of trying to think of input that will cover all cases, which can be difficult and/or very labor intensive, you let chance do it for you. It is conceptually similar to random data generation, but the intention here is to generate random or semi-random inputs rather than persistent data.
When Is Fuzz Testing Useful?
Fuzz testing is useful in particular for finding security and performance problems when unexpected inputs cause crashes or memory leaks. But it can also help ensure that all invalid inputs are detected early and are rejected properly by the system.
Consider, for example, input that comes in the form of deeply nested JSON documents (very common in web APIs). Trying to generate manually a comprehensive list of test cases is both error-prone and a lot of work. But fuzz testing is the perfect technique.
Using Fuzz Testing
There are several libraries you can use for fuzz testing. My favorite is gofuzz from Google. Here is a simple example that automatically generates 200 unique objects of a struct with several fields, including a nested struct.
import (
"fmt"
"http://ift.tt/1uxljqG"
)
func SimpleFuzzing() {
type SomeType struct {
A string
B string
C int
D struct {
E float64
}
}
f := fuzz.New()
uniqueObject := SomeType{}
uniqueObjects := map[SomeType]int{}
for i := 0; i < 200; i++ {
f.Fuzz(&object)
uniqueObjects[object]++
}
fmt.Printf("Got %v unique objects.\n", len(uniqueObjects))
// Output:
// Got 200 unique objects.
}
Testing Your Cache
Pretty much every complex system that deals with a lot of data has a cache, or more likely several levels of hierarchical caches. As the saying goes, there are only two difficult things in computer science: naming things, cache invalidation, and off by one errors.
Jokes aside, managing your caching strategy and implementation can complicate your data access but have a tremendous impact on your data access cost and performance. Testing your cache can't be done from the outside because your interface hides where the data comes from, and the cache mechanism is an implementation detail.
Let's see how to test the cache behavior of the Songify hybrid data layer.
Cache Hits and Misses
Caches live and die by their hit/miss performance. The basic functionality of a cache is that if requested data is available in the cache (a hit) then it will be fetched from the cache and not from the primary data store. In the original design of the HybridDataLayer, the cache access was done through private methods.
Go visibility rules make it impossible to call them directly or replace them from another package. To enable cache testing, I'll change those methods to public functions. This is fine because the actual application code operates through the DataLayer interface, which doesn't expose those methods.
The test code, however, will be able to replace these public functions as needed. First, let's add a method to get access to the Redis client, so we can manipulate the cache:
func (m *HybridDataLayer) GetRedis() *redis.Client {
return m.redis
}
Next I'll change the getSongByUser_DB() methods to a public function variable. Now, in the test, I can replace the GetSongsByUser_DB() variable with a function that keeps track of how many times it was called and then forwards it to the original function. That allows us to verify if a call to GetSongsByUser() fetched the songs from the cache or from the DB.
Let's break it down piece by piece. First, we get the data layer (that also clears the DB and redis), create a user, and add a song. The AddSong() method also populates redis.
func TestGetSongsByUser_Cache(t *testing.T) {
now := time.Now()
u := User{Name: "Gigi",
Email: "gg@gg.com",
RegisteredAt: now, LastLogin: now}
dl, err := getDataLayer()
if err != nil {
t.Error("Failed to create hybrid data layer")
}
err = dl.CreateUser(u)
if err != nil {
t.Error("Failed to create user")
}
lm, err := NewSongManager(u, dl)
if err != nil {
t.Error("NewSongManager() returned 'nil'")
}
err = lm.AddSong(testSong, nil)
if err != nil {
t.Error("AddSong() failed")
}
This is the cool part. I keep the original function and define a new instrumented function that increments the local callCount variable (it's all in a closure) and calls the original function. Then, I assign the instrumented function to the variable GetSongsByUser_DB. From now on, every call by the hybrid data layer to GetSongsByUser_DB() will go to the instrumented function.
callCount := 0
originalFunc := GetSongsByUser_DB
instrumentedFunc := func(m *HybridDataLayer,
email string,
songs *[]Song) (err error) {
callCount += 1
return originalFunc(m, email, songs)
}
GetSongsByUser_DB = instrumentedFunc
At this point, we're ready to actually test the cache operation. First, the test calls the GetSongsByUser() of the SongManager that forwards it to the hybrid data layer. The cache is supposed to be populated for this user we just added. So the expected result is that our instrumented function will not be called, and the callCount will remain at zero.
_, err = lm.GetSongsByUser(u)
if err != nil {
t.Error("GetSongsByUser() failed")
}
// Verify the DB wasn't accessed because cache should be
// populated by AddSong()
if callCount > 0 {
t.Error(`GetSongsByUser_DB() called when it
shouldn't have`)
}
The last test case is to ensure that if the user's data is not in the cache, it will be fetched properly from the DB. The test accomplishes it by flushing Redis (clearing all its data) and making another call to GetSongsByUser(). This time, the instrumented function will be called, and the test verifies that the callCount is equal to 1. Finally, the original GetSongsByUser_DB() function is restored.
// Clear the cache
dl.GetRedis().FlushDB()
// Get the songs again, now it's should go to the DB
// because the cache is empty
_, err = lm.GetSongsByUser(u)
if err != nil {
t.Error("GetSongsByUser() failed")
}
// Verify the DB was accessed because the cache is empty
if callCount != 1 {
t.Error(`GetSongsByUser_DB() wasn't called once
as it should have`)
}
GetSongsByUser_DB = originalFunc
}
Cache Invalidation
Our cache is very basic and doesn't do any invalidation. This works pretty well as long as all songs are added through the AddSong() method that takes care of updating Redis. If we add more operations like removing songs or deleting users then these operations should take care of updating Redis accordingly.
This very simple cache will work even if we have a distributed system where multiple independent machines can run our Songify service—as long as all the instances work with the same DB and Redis instances.
However, if the DB and cache can get out of sync due to maintenance operations or other tools and applications changing our data then we need to come up with an invalidation and refresh policy for the cache. It can be tested using the same techniques—replace target functions or directly access the DB and Redis in your test to verify the state.
LRU Caches
Usually, you can't just let the cache grow infinitely. A common scheme to keep the most useful data in the cache is LRU caches (least recently used). The oldest data gets bumped from the cache when it reaches capacity.
Testing it involves setting the capacity to a relatively small number during the test, exceeding the capacity, and ensuring that the oldest data is not in the cache anymore and accessing it requires DB access.
Testing Your Data Integrity
Your system is only as good as your data integrity. If you have corrupted data or missing data then you're in bad shape. In real-world systems, it's difficult to maintain perfect data integrity. Schema and formats change, data is ingested through channels that might not check for all the constraints, bugs let in bad data, admins attempt manual fixes, backups and restores might be unreliable.
Given this harsh reality, you should test your system's data integrity. Testing data integrity is different than regular automated tests after each code change. The reason is that data can go bad even if the code didn't change. You definitely want to run data integrity checks after code changes that might alter data storage or representation, but also run them periodically.
Testing Constraints
Constraints are the foundation of your data modeling. If you use a relational DB then you can define some constraints at the SQL level and let the DB enforce them. Nullness, length of text fields, uniqueness and 1-N relationships can be defined easily. But SQL can't check all the constraints.
For example, in Desongcious, there is a N-N relationship between users and songs. Each song must be associated with at least one user. There is no good way to enforce this in SQL (well, you can have a foreign key from song to user and have the song point to one of the users associated with it). Another constraint may be that each user may have at most 500 songs. Again, there is no way to represent it in SQL. If you use NoSQL data stores then usually there is even less support for declaring and validating constraints at the data store level.
That leaves you with a couple of options:
- Ensure that access to data goes only through vetted interfaces and tools that enforce all the constraints.
- Periodically scan your data, hunt constraint violations, and fix them.
Testing Idempotency
Idempotency means that performing the same operation multiple times in a row will have the same effect as performing it once.
For example, setting the variable x to 5 is idempotent. You can set x to 5 one time or a million times. It will still be 5. However, incrementing X by 1 is not idempotent. Every consecutive increment changes its value. Idempotency is a very desirable property in distributed systems with temporary network partitions and recovery protocols that retry sending a message multiple times if there is no immediate response.
If you design idempotency into your data access code, you should test it. This is typically very easy. For each idempotent operation you extend to perform the operation twice or more in a row and verify there are no errors and the state remains the same.
Note that idempotent design may sometimes hide errors. Consider deleting a record from a DB. It is an idempotent operation. After you delete a record, the record doesn't exist in the system anymore, and trying to delete it again will not bring it back. That means that trying to delete a non-existent record is a valid operation. But it might mask the fact that the wrong record key was passed by the caller. If you return an error message then it's not idempotent.
Testing Data Migrations
Data migrations can be very risky operations. Sometimes you run a script over all your data or critical parts of your data and perform some serious surgery. You should be ready with plan B in case something goes wrong (e.g. go back to the original data and figure out what went wrong).
In many cases, data migration can be a slow and costly operation that may require two systems side by side for the duration of the migration. I participated in several data migrations that took several days or even weeks. When facing a massive data migration, it's worth it to invest the time and test the migration itself on a small (but representative) subset of your data and then verify that the newly migrated data is valid and the system can work with it.
Testing Missing Data
Missing data is an interesting problem. Sometimes missing data will violate your data integrity (e.g. a song whose user is missing), and sometimes it's just missing (e.g. someone removes a user and all their songs).
If the missing data causes a data integrity problem then you'll detect it in your data integrity tests. However, if some data is just missing then there is no easy way to detect it. If the data never made it into persistent storage then maybe there is a trace in the logs or other temporary stores.
Depending how much of a risk missing data is, you may write some tests that deliberately remove some data from your system and verify the system behaves as expected.
Conclusion
Testing data-intensive code requires deliberate planning and an understanding of your quality requirements. You can test at several levels of abstraction, and your choices will affect how thorough and comprehensive your tests are, how many aspects of your actual data layer you test, how fast your tests run, and how easy it is to modify your tests when the data layer changes.
There is no single correct answer. You need to find your sweet spot along the spectrum from super comprehensive, slow and labor-intensive tests to fast, lightweight tests.
Manipulating HTML5 Canvas Using Konva: Part 5, Events
If you have been following this series from the beginning, you should now be very comfortable with shapes, groups, and layers. You should also be able to easily draw some basic and complex shapes on the canvas using Konva. If you plan on using Konva to create some interactive app or games, learning how to bind events to different shapes on the stage is the next logical step.
In this tutorial, you will learn how to bind events to any shape using Konva. You will also learn about event delegation and propagation. Sometimes, you need might need to control the hit region of a shape as well as fire events programmatically. We will be discussing these two topics as well.
Binding Events to a Shape
You can bind different events to any shape created using Konva with the help of the on() method. All you have to do is pass the name of the event as the first parameter and the function to be executed when the event occurs as the second parameter. You can use Konva to detect mouseup, mousedown, mouseenter, mouseleave, mouseover, mousemove, click, and dblclick. Additionally, Konva allows you to detect wheel, dragstart, dragmove, and dragend events.
Here is an example that detects mousedown and mouseleave events on a regular polygon (hexagon). Similarly, the smaller circle is bound to the mouseover and mouseup events and the larger circle is bound to the mouseenter, mouseleave, and mousemove events.
var canvasWidth = 600;
var canvasHeight = 400;
var stage = new Konva.Stage({
container: "example",
width: canvasWidth,
height: canvasHeight
});
var layerA = new Konva.Layer();
var polyA = new Konva.RegularPolygon({
x: 125,
y: 125,
sides: 6,
radius: 80,
fill: "yellow",
stroke: "black",
strokeWidth: 5
});
var circA = new Konva.Circle({
x: 275,
y: 225,
height: 100,
fill: "orange",
stroke: "black"
});
var circB = new Konva.Circle({
x: 475,
y: 275,
radius: 100,
fill: "red",
stroke: "black"
});
layerA.add(polyA, circA, circB);
stage.add(layerA);
polyA.on("mousedown", function() {
polyA.sides(polyA.sides() + 1);
layerA.draw();
});
polyA.on("mouseleave", function() {
var totalSides = polyA.sides();
if(totalSides > 3) {
polyA.sides(polyA.sides() - 1);
}
layerA.draw();
});
circA.on("mouseover", function() {
circA.strokeWidth(10);
layerA.draw();
});
circA.on("mouseup", function() {
circA.strokeWidth(5);
layerA.draw();
});
circB.on("mouseenter", function() {
stage.container().style.cursor = "crosshair";
});
circB.on("mouseleave", function() {
stage.container().style.cursor = "default";
});
circB.on("mousemove", function() {
var pointerPos = stage.getPointerPosition();
var r = pointerPos.x % 255;
var g = pointerPos.y % 255;
circB.fill("rgb(" + r + ", " + g + ", 100)");
layerA.draw();
});
If a user presses any mouse button while the cursor is inside the regular polygon, we increase the number of sides of the polygon by 1. The sides() method can be used without a parameter to get the number of sides for a polygon or used with one parameter to set the number of sides for a polygon. You can also get the number of sides using getSides() and set the number of sides using setSides(). The sides of the polygon are reduced by one whenever the mouse cursor leaves the polygon.
For the smaller circle, the mouseover event is used to set the stroke width value to 10. The mouseup event changes the stroke width value to 5. Keep in mind that the mouseup event has to occur inside the circle itself. For example, the stroke width will not change to 5 if you press the mouse button inside the circle and then release it only after the cursor is outside the circle.
In the case of the larger circle, we are using the mousemove event to change its fill color. We are also changing the cursor of the larger circle using stage.container().style.cursor whenever the cursor moves in and out of the circle.
One important thing that you should keep in mind is that you have to call the draw() method on the respective layer if the event listeners for any shape resulted in a change of attributes like fill color, stroke width, etc. Otherwise, the changes won't be reflected on the canvas.
You don't have to bind one event at a time to a shape. You can also pass in a space delimited string containing multiple event types to the on() method. This will bind all the events listed in the string to that particular shape.
Konva also supports corresponding mobile versions of all these events. For example, you can register touchstart, touchmove, touchend, tap, dbltap, dragstart, dragmove, and dragend using Konva on mobile devices.
You can also fire any of these events for a particular shape or shapes using the fire() method. Similarly, Konva allows you to fire custom events like throwStones.
Removing Event Listeners
You can remove any event listeners attached to a shape with the help of the off() method in Konva. You just have to specify the event name that you don't want to listen to.
You can also create multiple event bindings for a single shape. For example, let's say you have a circle and you want to increase the radius of the circle every time the mouse cursor goes over it, up to a certain limit. You might also want to change the fill color of the circle on every mouseover event.
One option is to do both these tasks inside a single mouseover event listener and stop updating the radius later. Another option is to create two mouseover event listeners with different namespaces to identify them. This way, you will be able to increase the radius and change the fill color independently.
circA.on("mouseover.radius", function() {
var curRadius = circA.radius();
if(curRadius < 150) {
circA.radius(curRadius + 5);
layerA.draw();
} else {
circA.off('mouseover.radius');
}
});
circA.on("mouseover.fillcolor", function() {
var h = Math.floor(Math.random()*360);
var color = "hsl(" + h + ", 60%, 60%)";
circA.fill(color);
layerA.draw();
});
You should note that I have added layerA.draw() inside both listeners. If you fail to add it inside the mouseover.fillcolor listener, the color will stop updating as soon as the radius becomes 150.
Instead of removing one event listener at a time, you can also stop listening to all events bound to a shape using the setListening() method. You can pass true and false to this method in order to turn event listeners on and off. Keep in mind that you will also have to redraw the hit graph of the affected layer by calling the drawHit() method right after you call setListening().
Event Delegation and Propagation
Instead of binding events directly to all the shapes present on a layer, you can also bind an event to the layer itself. After that, you can determine which shape triggered the event using the target property of the event object. This way, Konva allows you to effectively delegate events from the parent to its children.
Let's say you are listening to click events on a circle drawn on a layer in Konva. The same click event propagates to the containing group as well as the containing layer. This may or may not be the intended behavior. If you want to prevent the event from bubbling up inside a shape to the containing layer, you can set the cancelBubble property for the event object to true.
var canvasWidth = 600;
var canvasHeight = 400;
var stage = new Konva.Stage({
container: "example",
width: canvasWidth,
height: canvasHeight
});
var layerA = new Konva.Layer();
var circA = new Konva.Circle({
x: 300,
y: 200,
height: 100,
fill: "orange",
stroke: "black",
name: "Orange Circle"
});
var starA = new Konva.Star({
x: 125,
y: 125,
innerRadius: 25,
outerRadius: 75,
rotation: 90,
fill: "blue",
stroke: "black",
name: "Blue Star"
});
var ringA = new Konva.Ring({
x: 475,
y: 275,
innerRadius: 25,
outerRadius: 75,
fill: "brown",
stroke: "black",
name: "Brown Ring"
});
var textA = new Konva.Text({
text: "",
fontFamily: "Calibri",
fontSize: 24,
fill: "black",
x: 10,
y: 10
});
layerA.add(circA, starA, ringA, textA);
stage.add(layerA);
layerA.on("click", function(e) {
var shapeName = e.target.attrs.name;
textA.setText(shapeName);
layerA.draw();
});
I have used the name property to assign a name to each of our shapes. The setText() method is then used to change the text inside textA to the name of the shape we just clicked.
Custom Hit Regions
In the above example, the ring registered a click on it when the click occurred between the inner and outer circle. What if you wanted to register the click inside the smaller circle as well? Konva allows you to define custom hit regions using the hitFunc property. This property accepts a function as its value, and this function is used to draw the custom hit region.
The following example shows you how to create custom hit regions. You should now be able to click in the area between the star spikes and still register a click. With the help of custom hit regions, you can make sure that your users don't have to click on exact locations to register a click event. This can result in a better user experience when dealing with smaller or more complex shapes.
var starA = new Konva.Star({
x: 125,
y: 125,
innerRadius: 25,
outerRadius: 75,
rotation: 90,
fill: "blue",
stroke: "black",
name: "Blue Star",
hitFunc: function(context) {
context.beginPath();
context.arc(0, 0, this.getOuterRadius(), 0, Math.PI * 2, true);
context.closePath();
context.fillStrokeShape(this);
}
});
var ringA = new Konva.Ring({
x: 475,
y: 275,
innerRadius: 25,
outerRadius: 75,
fill: "brown",
stroke: "black",
name: "Brown Ring",
hitFunc: function(context) {
context.beginPath();
context.arc(0, 0, this.getOuterRadius(), 0, Math.PI * 2, true);
context.closePath();
context.fillStrokeShape(this);
}
});
Final Thoughts
In this tutorial, we have covered different mobile and desktop events that you can bind to any shape in Konva. You can attach these events one at a time or many of them at once. Konva also allows you to fire your own custom events programmatically using the fire() method. The last section of the tutorial showed you how to define your own hit regions in order to detect hits on an area that could be larger or smaller than the original shape.
Combining the knowledge of this tutorial with others in the series, you should now be able to draw a variety of shapes on the canvas and allow your users to interact with them.
If you have any questions related to this tutorial, feel free to let me know in the comments.
Thursday, December 14, 2017
Manipulating HTML5 Canvas Using Konva: Part 4, Styling
In the second tutorial of the series, you learned how to draw some basic shapes like rectangles, circles, and regular polygons using Konva. The third tutorial covered how you can use Konva to draw some more complex shapes like stars and rings as well as sprites on the canvas.
In this tutorial, we will go one step further and learn how to apply different styling to shapes by changing their fill and stroke values. You will also learn how to control the opacity of a shape and apply shadows to it. In the final sections, you will learn how to use blend modes to specify what the final result should look like if different shapes overlap.
Applying Fill and Stroke
We have been using the fill and stroke properties from the first tutorial of the series. However, we have only used them to fill shapes with a solid color so far. You can also fill a shape with gradients (both linear and radial) as well as images. The same goes for different strokes applied to a shape.
There are two ways of applying a fill to different shapes. You can either set the fill value using the fill property when a shape is first created in Konva, or you can use the fill() method to apply a fill dynamically in response to some events like hover, button click, etc.
When filling an element with a solid color, you can specify a value for the fill property and it will work just fine. When using a linear gradient to fill the inside of a shape, you need to specify valid values for a lot of other properties like fillLinearGradientStartPoint, fillLinearGradientEndPoint, and fillLinearGradientColorStops. The first two properties accept objects that specify the x and y co-ordinates of the start and end points of a gradient. You can also specify the x and y values separately using the fillLinearGradientStartPointX, fillLinearGradientStartPointY, fillLinearGradientEndPointX, and fillLinearGradientEndPointY properties.
Radial gradients also have the same set of properties, but the word Linear is replaced with Radial. Two additional properties related to radial gradients are fillRadialGradientStartRadius and fillRadialGradientEndRadius.
var canvasWidth = 600;
var canvasHeight = 400;
var stage = new Konva.Stage({
container: "example",
width: canvasWidth,
height: canvasHeight
});
var layerA = new Konva.Layer();
var rectA = new Konva.Rect({
x: 25,
y: 25,
width: 200,
height: 150,
fillLinearGradientStartPoint: {
x: 0,
y: 0
},
fillLinearGradientEndPoint: {
x: 200,
y: 150
},
fillLinearGradientColorStops: [0, 'blue', 1, 'yellow'],
stroke: "black"
});
var rectB = new Konva.Rect({
x: 375,
y: 25,
width: 200,
height: 150,
fillLinearGradientStartPoint: {
x: 0,
y: 50
},
fillLinearGradientEndPoint: {
x: 100,
y: -50
},
fillLinearGradientColorStops: [0, 'green', 0.1, 'yellow', 0.5, 'red', 0.9, 'black'],
stroke: "black"
});
var rectC = new Konva.Rect({
x: 25,
y: 200,
width: 200,
height: 150,
fillRadialGradientStartRadius: 0,
fillRadialGradientEndRadius: 220,
fillRadialGradientColorStops: [0, 'green', 0.5, 'yellow', 0.75, 'red', 0.9, 'black'],
stroke: "black"
});
var rectD = new Konva.Rect({
x: 375,
y: 200,
width: 200,
height: 150,
fillRadialGradientStartRadius: 0,
fillRadialGradientEndRadius: 150,
fillRadialGradientStartPoint: {
x: 100,
y: 75
},
fillRadialGradientEndPoint: {
x: 100,
y: 75
},
fillRadialGradientColorStops: [0, 'blue', 0.5, 'yellow', 0.9, 'green'],
stroke: "black"
});
layerA.add(rectA, rectB, rectC, rectD);
stage.add(layerA);
When not specified, the start and end point of a radial gradient are assumed to be 0,0. This is why the radial gradient in the third rectangle originates from the top left corner. Also remember that the start and end points are specified relative to the shape itself.
Just like the fill, you can set a value for the stroke color and stroke width using the stroke and strokeWidth properties when a shape is first instantiated. You can also dynamically set both these values using the stroke() and strokewidth() methods.
Creating Shadows in Konva
You can apply shadows to any shapes created using Konva with the help of four different properties called shadowColor, shadowOffset, shadowBlur, and shadowOpacity. The shadowOffset property accepts an object with x and y components as its value, but you can also use shadowOffsetX and shadowOffsetY to specify the x and y co-ordinates separately. You also have the option of enabling and disabling the shadows for any particular shape using the shadowEnabled property.
You can control the opacity of the shape itself using the opacity property. Please note that a fully transparent object will not cast any shadow. Similarly, if you have set the fill color of a shape to transparent, only the shadow of its stroke will be rendered on the canvas.
var canvasWidth = 600;
var canvasHeight = 400;
var stage = new Konva.Stage({
container: "example",
width: canvasWidth,
height: canvasHeight
});
var layerA = new Konva.Layer();
var rectA = new Konva.Rect({
x: 25,
y: 25,
width: 200,
height: 150,
cornerRadius: 5,
fill: "orange",
opacity: 0.5,
shadowColor: "black",
shadowOffset: {
x: -10,
y: 10
},
shadowBlur: 10,
stroke: "black"
});
var starA = new Konva.Star({
x: 400,
y: 200,
numPoints: 10,
innerRadius: 50,
outerRadius: 150,
fill: "transparent",
stroke: "black",
strokeWidth: 5,
shadowColor: "red",
shadowOffset: {
x: 5,
y: 5
},
shadowBlur: 0
});
layerA.add(rectA, starA);
stage.add(layerA);
Setting the shadowBlur property to 0 makes the shadow as sharp as the original shape itself. Setting this value too high will make the shadow lose the original shape; you will only see a dark patch on the canvas.
I would like to point out that you can also create text shadows with the same set of properties once you have instantiated a Konva.Text() object.
Applying Blend Modes
So far in the series, any overlap between shapes hid the bottom shape completely. The only way to keep the bottom shape visible was to make all the shapes over it partially transparent.
Sometimes, you might want the final result after the overlap of different shapes to follow certain rules. For example, it is possible to only show the lighter or darker color in cases where the shapes overlap.
Konva allows you to specify some values to determine how the colors of overlapping shapes should blend together using the globalCompositeOperation property. You can read the documentation on MDN to learn about the property and all its possible values in more detail.
In the following example, I have applied a different blend mode to each of the rectangles placed at the corner of the central rectangle.
var canvasWidth = 600;
var canvasHeight = 400;
var stage = new Konva.Stage({
container: "example",
width: canvasWidth,
height: canvasHeight
});
var layerA = new Konva.Layer();
var rectCenter = new Konva.Rect({
x: 225,
y: 125,
width: 150,
height: 150,
fill: "rgb(255, 100, 0)"
});
var rectA = new Konva.Rect({
x: 125,
y: 25,
width: 150,
height: 150,
fill: "rgb(0, 200, 100)",
globalCompositeOperation: "lighten"
});
var rectB = new Konva.Rect({
x: 325,
y: 25,
width: 150,
height: 150,
fill: "rgb(0, 200, 100)",
globalCompositeOperation: "darken"
});
var rectC = new Konva.Rect({
x: 125,
y: 225,
width: 150,
height: 150,
fill: "rgb(0, 200, 100)",
globalCompositeOperation: "hue"
});
var rectD = new Konva.Rect({
x: 325,
y: 225,
width: 150,
height: 150,
fill: "rgb(0, 255, 0)",
globalCompositeOperation: "xor"
});
layerA.add(rectCenter, rectA, rectB, rectC, rectD);
stage.add(layerA);
The color of the top-left rectangle is rgb(0, 200, 100), and the color of the central rectangle is rgb(255, 100, 0). When the lighten blend mode is applied, the rgb components of both the colors are compared individually, and the higher values for each component are used to get the final color. In our case, the final color for the top-left corner becomes rgb(255, 200, 100).
When the darken blend mode is applied, the rgb components of both the colors are compared individually, and the lower values for each component are used to get the final color. In our case, the final color for the top-right corner becomes rgb(0, 100, 0).
When the hue blend mode is applied, the luma and chroma of the bottom color are combined with the hue of the top color. This is why the final color still remains green but gets lighter. When the xor blend mode is applied, the final color becomes transparent at all the places of overlap.
Final Thoughts
In this tutorial, we learned how to fill a shape with linear or radial gradients instead of solid colors. We also learned how to apply shadows to different shapes and make them partially transparent using the opacity property. The final section showed you how to use blend modes in order to change the final color after two shapes overlap.
If you have any questions related to this tutorial, please let me know in the comments. The next and final tutorial of the series will teach you how to bind events to different shapes in Konva.
