Friday, June 30, 2017
Creating a Blogging App Using React, Part 2: User Sign-Up
In the first part of this tutorial series, you saw how to implement the sign-in functionality. In this part, you'll learn how to implement the sign-up functionality and modify the sign-in functionality to check for valid users from MongoDB.
Getting Started
Let's get started by cloning the source code from the first part of the tutorial.
git clone http://ift.tt/2spGDCR
Once the directory has been cloned, navigate to the project directory and install the required dependencies.
cd ReactBlogApp-SignIn npm install
Start the Node.js server and you will have the application running at http://localhost:7777/index.html#/.
Setting Up the Back End
For this application, you'll be using MongoDB as the back end. Follow the instructions in the MongoDB official documentation to install MongoDB on Ubuntu. Once you have MongoDB installed, you'll need a connector to connect MongoDB and Node.js. Install the MongoDB Node.js driver using the Node Package Manager (or npm):
npm install mongodb
Once you have the driver installed, you should be able to require the driver in the application.
Create a file called user.js where you'll keep the user-related stuff. Inside the user.js file, require the MongoDB client-related dependencies.
var MongoClient = require('mongodb').MongoClient;
You'll be using a library called assert to check the returned response. Include assert in the user.js file.
var assert = require('assert');
Let's name our database Blog in MongoDB, so our database URL is as shown:
var url = 'mongodb://localhost:27017/Blog';
Inside the user.js file, create and export a function called signup.
module.exports = {
signup: function(){
// Code will be here
}
}
Using the MongoDB client, try to connect to the database. Once connected, you'll log the connected message in the terminal.
module.exports = {
signup: function(name, email, password){
MongoClient.connect(url, function(err, db) {
console.log('connected')
});
}
}
Setting Up the Sign-Up Event
Once you have set up the MongoDB back end, let's implement the sign-up event. Inside the main.jsx page, include the on-change event for the name, email and password input text boxes in the signup class.
handleNameChange(e){
this.setState({name:e.target.value})
}
handleEmailChange(e){
this.setState({email:e.target.value})
}
handlePasswordChange(e){
this.setState({password:e.target.value})
}
Bind the above event changes in the class constructor.
constructor(props) {
super(props);
this.handleNameChange = this.handleNameChange.bind(this);
this.handleEmailChange = this.handleEmailChange.bind(this);
this.handlePasswordChange = this.handlePasswordChange.bind(this);
}
Define the state variables inside the signup class constructor.
this.state = {
name:'',
email:'',
password:''
};
Define the signup method inside the signup class. Inside the signup method, using the axios library, make a post method call to the signup method in the user.js file.
signUp(){
axios.post('/signup', {
name: this.state.name,
email: this.state.email,
password: this.state.password
})
.then(function (response) {
console.log(response);
})
.catch(function (error) {
console.log(error);
});
}
Inside the signup function in the user.js file, you'll implement the database insert.
Add the /signup request handler in the app.js file as shown to handle the sign-up click event. Inside the /signup request handler function, make a call to the user.signup method.
app.post('/signup', function (req, res) {
user.signup('','','')
console.log(res);
})
Require the user.js file inside the app.js file.
var user = require('./user')
Save the above changes and restart the server. Point your browser to http://localhost:7777/index.html#/signup and you should have the sign-up page. Click on the Sign Up button and you will have the connected message in the terminal.
Save User Details in MongoDB
To save user details in the Blog database, you'll create a collection called user. Inside the user collection, you'll keep all the user details such as name, email address, and password. The MongoClient.connect returns a db parameter using which you can insert an entry in the user collection.
You'll make use of the insertOne method to insert a single record in the user collection. Modify the code in the signup method in user.js as shown below:
db.collection('user').insertOne( {
"name": name,
"email": email,
"password": password
},function(err, result){
assert.equal(err, null);
console.log("Saved the user sign up details.");
});
Here is the complete user.js code:
var MongoClient = require('mongodb').MongoClient;
var assert = require('assert');
var url = 'mongodb://localhost:27017/Blog';
module.exports = {
signup: function(name, email, password){
MongoClient.connect(url, function(err, db) {
db.collection('user').insertOne( {
"name": name,
"email": email,
"password": password
},function(err, result){
assert.equal(err, null);
console.log("Saved the user sign up details.");
});
});
}
}
Modify the /signup request handler in the app.js file to pass in the name, email and password to the user.js signup method.
app.post('/signup', function (req, res) {
var name=req.body.name;
var email=req.body.email;
var password=req.body.password;
if(name && email && password){
user.signup(name, email, password)
}
else{
res.send('Failure');
}
})
Save the above changes and restart the server. Point your browser to http://localhost:7777/index.html#/signup. Fill the user sign-up details and click the sign-up button. You will have the Saved the user sign up details. message in the server terminal. Log in to the MongoDB shell and check the user collection in the Blog database. To find the user details, enter the following command in the MongoDB shell:
db.user.find()
The above command will display the user details in JSON format.
{
"name": "roy",
"email": "royagasthyan@gmail.com",
"password": "test",
"_id": ObjectId("58f622f50cb9b32905f1cb4b")
}
Implementing User Sign-In Check
In the first part of the tutorial, you hard-coded the user sign-in check since the user sign-up hasn't been implemented. Let's modify the hard-coded sign-in check and look into the MongoDB database for valid user sign-ins.
Create a function called validateSignIn in the user.js file.
validateSignIn: function(username, password,callback){
}
Inside the validateSignIn function, using the MongoDB client you'll connect to the Blog database and query the user table for a user with the specified username and password. You'll make use of the findOne method to query the user collection.
db.collection('user').findOne( { email : username ,password: password },function(err, result){
});
Check the returned result for null in case the entry is not found.
if(result==null){
callback(false)
}
else{
callback(true)
}
As seen in the above code, if no entry is found, false is returned in the callback. If an entry is found, true is returned in the callback.
Here is the complete validateSignIn method:
validateSignIn: function(username, password,callback){
MongoClient.connect(url, function(err, db){
db.collection('user').findOne( { email : username ,password: password
},function(err, result){
if(result==null){
callback(false)
}
else{
callback(true)
}
});
});
}
In the /signin method in the app.js file, you'll make a call to the validateSignIn method. In the callback function, you'll check for the response. If true, it will indicate a valid sign-in, else an invalid sign-in. Here is how it looks:
app.post('/signin', function (req, res) {
var user_name=req.body.email;
var password=req.body.password;
user.validateSignIn(user_name,password,function(result){
if(result){
res.send('Success')
}
else{
res.send('Wrong username password')
}
});
Save the above changes and restart the server. Point your browser to http://localhost:7777/index.html#/. Enter a valid username and password and you will have a success message logged in the browser console. On entering an invalid username and password, it would display an error message.
Wrapping It Up
In this part of the tutorial, you saw how to implement the user sign-up process. You saw how to create the sign-up view and pass the data from the React user interface to Node.js and then save it in the MongoDB. You also modified the user sign-in functionality to check for valid user sign-in from the MongoDB database.
In the next part of the tutorial, you'll implement the add post and display post page functionality.
Source code from this tutorial is available on GitHub.
Do let us know your thoughts or any suggestions in the comments below.
Thursday, June 29, 2017
Wednesday, June 28, 2017
Tuesday, June 27, 2017
Monday, June 26, 2017
Saturday, June 24, 2017
Friday, June 23, 2017
Securing iOS Data at Rest: Encryption
In this post, we'll look at advanced uses of encryption for user data in iOS apps. We'll start with a high-level look at AES encryption, and then go on to look at some examples of how to implement AES encryption in Swift.
In the last post, you learned how to store data using the keychain, which is good for small pieces of information such as keys, passwords, and certificates.
- iOS SDKSecuring iOS Data at Rest: The Keychain
If you are storing a large amount of custom data that you want to be available only after the user or device authenticates, then it's better to encrypt the data using an encryption framework. For example, you may have an app that can archive private chat messages saved by the user or private photos taken by the user, or which can store the user's financial details. In these cases, you would probably want to use encryption.
There are two common flows in applications for encrypting and decrypting data from iOS apps. Either the user is presented with a password screen, or the application is authenticated with a server which returns a key to decrypt the data.
It's never a good idea to reinvent the wheel when it comes to encryption. Therefore, we are going to use the AES standard provided by the iOS Common Crypto library.
AES
AES is a standard that encrypts data given a key. The same key used to encrypt the data is used to decrypt the data. There are different key sizes, and AES256 (256 bits) is the preferred length to be used with sensitive data.
RNCryptor is a popular encryption wrapper for iOS that supports AES. RNCryptor is a great choice because it gets you up and running very quickly without having to worry about the underlying details. It is also open source so that security researchers can analyze and audit the code.
On the other hand, if your app deals with very sensitive information and you think your application will be targeted and cracked, you may want to write your own solution. The reason for this is that when many apps use the same code, it can make the hacker's job easier, allowing them to write a cracking app that finds common patterns in the code and applies patches to them.
Keep in mind, though, that writing your own solution only slows down an attacker and prevents automated attacks. The protection you are getting from your own implementation is that a hacker will need to spend time and dedication on cracking your app alone.
Whether you choose a third-party solution or choose to roll your own, it's important to be knowledgeable about how encryption systems work. That way, you can decide if a particular framework you want to use is really secure. Therefore, the rest of this tutorial will focus on writing your own custom solution. With the knowledge you'll learn from this tutorial, you'll be able to tell if you're using a particular framework securely.
We'll start with the creation of a secret key that will be used to encrypt your data.
Create a Key
A very common error in AES encryption is to use a user's password directly as the encryption key. What if the user decides to use a common or weak password? How do we force users to use a key that is random and strong enough (has enough entropy) for encryption and then have them remember it?
The solution is key stretching. Key stretching derives a key from a password by hashing it many times over with a salt. The salt is just a sequence of random data, and it is a common mistake to omit this salt—the salt gives the key its vitally important entropy, and without the salt, the same key would be derived if the same password was used by someone else.
Without the salt, a dictionary of words could be used to deduce common keys, which could then be used to attack user data. This is called a "dictionary attack". Tables with common keys that correspond to unsalted passwords are used for this purpose. They're called "rainbow tables".
Another pitfall when creating a salt is to use a random number generating function that was not designed for security. An example is the rand() function in C, which can be accessed from Swift. This output can end up being very predictable!
To create a secure salt, we will use the function SecRandomCopyBytes to create cryptographically secure random bytes—which is to say, numbers that are difficult to predict.
To use the code, you'll need to add the following into your bridging header:
#import <CommonCrypto/CommonCrypto.h>
Here is the start of the code that creates a salt. We will add to this code as we go along:
var salt = Data(count: 8)
salt.withUnsafeMutableBytes { (saltBytes: UnsafeMutablePointer<UInt8>) -> Void in
let saltStatus = SecRandomCopyBytes(kSecRandomDefault, salt.count, saltBytes)
//...
Now we are ready to do key stretching. Fortunately, we already have a function at our disposal to do the actual stretching: the Password-Based Key Derivation Function (PBKDF2). PBKDF2 performs a function many times over to derive the key; increasing the number of iterations expands the time it would take to operate on a set of keys during a brute force attack. It is recommended to use PBKDF2 to generate your key.
var setupSuccess = true
var key = Data(repeating:0, count:kCCKeySizeAES256)
var salt = Data(count: 8)
salt.withUnsafeMutableBytes { (saltBytes: UnsafeMutablePointer<UInt8>) -> Void in
let saltStatus = SecRandomCopyBytes(kSecRandomDefault, salt.count, saltBytes)
if saltStatus == errSecSuccess
{
let passwordData = password.data(using:String.Encoding.utf8)!
key.withUnsafeMutableBytes { (keyBytes : UnsafeMutablePointer<UInt8>) in
let derivationStatus = CCKeyDerivationPBKDF(CCPBKDFAlgorithm(kCCPBKDF2), password, passwordData.count, saltBytes, salt.count, CCPseudoRandomAlgorithm(kCCPRFHmacAlgSHA512), 14271, keyBytes, key.count)
if derivationStatus != Int32(kCCSuccess)
{
setupSuccess = false
}
}
}
else
{
setupSuccess = false
}
}
Server-Side Key
You may be wondering now about the cases where you don't want to require users to provide a password within your app. Perhaps they are already authenticating with a single sign-on scheme. In this case, have your server generate an AES 256-bit (32 byte) key using a secure generator. The key should be different for different users or devices. On authenticating with your server, you can pass the server a device or user ID over a secure connection, and it can send the corresponding key back.
This scheme has a major difference. If the key is coming from the server, the entity that controls that server has the capacity to be able to read the encrypted data if the device or data were ever obtained. There is also the potential for the key to be leaked or exposed at a later time.
On the other hand, if the key is derived from something only the user knows—the user's password—then only the user can decrypt that data. If you are protecting information such as private financial data, only the user should be able to unlock the data. If that information is known to the entity anyway, it may be acceptable to have the server unlock the content via a server-side key.
Modes and IVs
Now that we have a key, let's encrypt some data. There are different modes of encryption, but we'll be using the recommended mode: cipher block chaining (CBC). This operates on our data one block at a time.
A common pitfall with CBC is the fact that each next unencrypted block of data is XOR’d with the previous encrypted block to make the encryption stronger. The problem here is that the first block is never as unique as all the others. If a message to be encrypted were to start off the same as another message to be encrypted, the beginning encrypted output would be the same, and that would give an attacker a clue to figuring out what the message might be.
To get around this potential weakness, we'll start the data to be saved with what is called an initialization vector (IV): a block of random bytes. The IV will be XOR’d with the first block of user data, and since each block depends on all blocks processed up until that point, it will ensure that the entire message will be uniquely encrypted, even if it has the same data as another message. In other words, identical messages encrypted with the same key will not produce identical results. So while salts and IVs are considered public, they should not be sequential or reused.
We will use the same secure SecRandomCopyBytes function to create the IV.
var iv = Data.init(count: kCCBlockSizeAES128)
iv.withUnsafeMutableBytes { (ivBytes : UnsafeMutablePointer<UInt8>) in
let ivStatus = SecRandomCopyBytes(kSecRandomDefault, kCCBlockSizeAES128, ivBytes)
if ivStatus != errSecSuccess
{
setupSuccess = false
}
}
Putting It All Together
To complete our example, we'll use the CCCrypt function with either kCCEncrypt or kCCDecrypt. Because we are using a block cipher, if the message doesn’t fit nicely into a multiple of the block size, we will need to tell the function to automatically add padding to the end.
As usual in encryption, it is best to follow established standards. In this case, the standard PKCS7 defines how to pad the data. We tell our encryption function to use this standard by supplying the KCCOptionPKCS7Padding option. Putting it all together, here is the full code to encrypt and decrypt a string.
class func encryptData(_ clearTextData : Data, withPassword password : String) -> Dictionary<String, Data>
{
var setupSuccess = true
var outDictionary = Dictionary<String, Data>.init()
var key = Data(repeating:0, count:kCCKeySizeAES256)
var salt = Data(count: 8)
salt.withUnsafeMutableBytes { (saltBytes: UnsafeMutablePointer<UInt8>) -> Void in
let saltStatus = SecRandomCopyBytes(kSecRandomDefault, salt.count, saltBytes)
if saltStatus == errSecSuccess
{
let passwordData = password.data(using:String.Encoding.utf8)!
key.withUnsafeMutableBytes { (keyBytes : UnsafeMutablePointer<UInt8>) in
let derivationStatus = CCKeyDerivationPBKDF(CCPBKDFAlgorithm(kCCPBKDF2), password, passwordData.count, saltBytes, salt.count, CCPseudoRandomAlgorithm(kCCPRFHmacAlgSHA512), 14271, keyBytes, key.count)
if derivationStatus != Int32(kCCSuccess)
{
setupSuccess = false
}
}
}
else
{
setupSuccess = false
}
}
var iv = Data.init(count: kCCBlockSizeAES128)
iv.withUnsafeMutableBytes { (ivBytes : UnsafeMutablePointer<UInt8>) in
let ivStatus = SecRandomCopyBytes(kSecRandomDefault, kCCBlockSizeAES128, ivBytes)
if ivStatus != errSecSuccess
{
setupSuccess = false
}
}
if (setupSuccess)
{
var numberOfBytesEncrypted : size_t = 0
let size = clearTextData.count + kCCBlockSizeAES128
var encrypted = Data.init(count: size)
let cryptStatus = iv.withUnsafeBytes {ivBytes in
encrypted.withUnsafeMutableBytes {encryptedBytes in
clearTextData.withUnsafeBytes {clearTextBytes in
key.withUnsafeBytes {keyBytes in
CCCrypt(CCOperation(kCCEncrypt),
CCAlgorithm(kCCAlgorithmAES),
CCOptions(kCCOptionPKCS7Padding + kCCModeCBC),
keyBytes,
key.count,
ivBytes,
clearTextBytes,
clearTextData.count,
encryptedBytes,
size,
&numberOfBytesEncrypted)
}
}
}
}
if cryptStatus == Int32(kCCSuccess)
{
encrypted.count = numberOfBytesEncrypted
outDictionary["EncryptionData"] = encrypted
outDictionary["EncryptionIV"] = iv
outDictionary["EncryptionSalt"] = salt
}
}
return outDictionary;
}
And here is the decryption code:
class func decryp(fromDictionary dictionary : Dictionary<String, Data>, withPassword password : String) -> Data
{
var setupSuccess = true
let encrypted = dictionary["EncryptionData"]
let iv = dictionary["EncryptionIV"]
let salt = dictionary["EncryptionSalt"]
var key = Data(repeating:0, count:kCCKeySizeAES256)
salt?.withUnsafeBytes { (saltBytes: UnsafePointer<UInt8>) -> Void in
let passwordData = password.data(using:String.Encoding.utf8)!
key.withUnsafeMutableBytes { (keyBytes : UnsafeMutablePointer<UInt8>) in
let derivationStatus = CCKeyDerivationPBKDF(CCPBKDFAlgorithm(kCCPBKDF2), password, passwordData.count, saltBytes, salt!.count, CCPseudoRandomAlgorithm(kCCPRFHmacAlgSHA512), 14271, keyBytes, key.count)
if derivationStatus != Int32(kCCSuccess)
{
setupSuccess = false
}
}
}
var decryptSuccess = false
let size = (encrypted?.count)! + kCCBlockSizeAES128
var clearTextData = Data.init(count: size)
if (setupSuccess)
{
var numberOfBytesDecrypted : size_t = 0
let cryptStatus = iv?.withUnsafeBytes {ivBytes in
clearTextData.withUnsafeMutableBytes {clearTextBytes in
encrypted?.withUnsafeBytes {encryptedBytes in
key.withUnsafeBytes {keyBytes in
CCCrypt(CCOperation(kCCDecrypt),
CCAlgorithm(kCCAlgorithmAES128),
CCOptions(kCCOptionPKCS7Padding + kCCModeCBC),
keyBytes,
key.count,
ivBytes,
encryptedBytes,
(encrypted?.count)!,
clearTextBytes,
size,
&numberOfBytesDecrypted)
}
}
}
}
if cryptStatus! == Int32(kCCSuccess)
{
clearTextData.count = numberOfBytesDecrypted
decryptSuccess = true
}
}
return decryptSuccess ? clearTextData : Data.init(count: 0)
}
Finally, here is a test to ensure that data is decrypted correctly after encryption:
class func encryptionTest()
{
let clearTextData = "some clear text to encrypt".data(using:String.Encoding.utf8)!
let dictionary = encryptData(clearTextData, withPassword: "123456")
let decrypted = decryp(fromDictionary: dictionary, withPassword: "123456")
let decryptedString = String(data: decrypted, encoding: String.Encoding.utf8)
print("decrypted cleartext result - ", decryptedString ?? "Error: Could not convert data to string")
}
In our example, we package all the necessary information and return it as a Dictionary so that all the pieces can later be used to successfully decrypt the data. You only need to store the IV and salt, either in the keychain or on your server.
Conclusion
This completes the three-part series on securing data at rest. We have seen how to properly store passwords, sensitive pieces of information, and large amounts of user data. These techniques are the baseline for protecting stored user information in your app.
It is a huge risk when a user's device is lost or stolen, especially with recent exploits to gain access to a locked device. While many system vulnerabilities are patched with a software update, the device itself is only as secure as the user's passcode and version of iOS. Therefore it is up to the developer of each app to provide strong protection of sensitive data being stored.
All of the topics covered so far make use of Apple's frameworks. I will leave an idea with you to think about. What happens when Apple's encryption library gets attacked?
When one commonly used security architecture is compromised, all of the apps that rely on it are also compromised. Any of iOS's dynamically linked libraries, especially on jailbroken devices, can be patched and swapped with malicious ones.
However, a static library that is bundled with the binary of your app is protected from this kind of attack because if you try and patch it, you end up changing the app binary. This will break the code signature of the app, preventing it from being launched. If you imported and used, for example, OpenSSL for your encryption, your app would not be vulnerable to a widespread Apple API attack. You can compile OpenSSL yourself and statically link it into your app.
So there is always more to learn, and the future of app security on iOS is always evolving. The iOS security architecture even now supports cryptographic devices and smart cards! In closing, you now know the best practices for securing data at rest, so it's up to you to follow them!
In the meantime, check out some of our other content about iOS app development and app security.
- SecurityHow to Hack Your Own App
- iOSGo Further With Swift: Animation, Networking, and Custom Controls
- SwiftSwift From Scratch: Delegation and Properties
How to Work With Elixir Comprehensions
Elixir is a very young programming language (emerged in 2011), but it is gaining popularity. I was initially interested in this language because when using it you can look at some common tasks programmers usually solve from a different angle. For instance, you can find out how to iterate over collections without the for cycle, or how to organize your code without classes.
Elixir has some very interesting and powerful features that may be hard to get your head around if you came from the OOP world. However, after some time it all starts to make sense, and you see how expressive the functional code can be. Comprehensions are one such feature, and this article I will explain how to work with them.
Comprehensions and Mapping
Generally speaking, a list comprehension is a special construct that allows you to create a new list based on existing ones. This concept is found in languages like Haskell and Clojure. Erlang also presents it and, therefore, Elixir has comprehensions as well.
You might ask how comprehensions are different from the map/2 function, which also takes a collection and produces a new one? That would be a fair question! Well, in the simplest case, comprehensions do pretty much the same thing. Take a look at this example:
defmodule MyModule do
def do_something(list) do
list |>
Enum.map(fn(el) -> el * 2 end)
end
end
MyModule.do_something([1,2,3]) |> IO.inspect # => [2,4,6]
Here I am simply taking a list with three numbers and producing a new list with all the numbers multiplied by 2. The map call can be further simplified as Enum.map( &(&1 * 2) ).
The do_something/1 function can now be rewritten using a comprehension:
def do_something(list) do
for el <- list, do: el * 2
end
This is what a basic comprehension looks like and, in my opinion, the code is a bit more elegant than in the first example. Here, once again, we take each element from the list and multiply it by 2. The el <- list part is called a generator, and it explains how exactly you wish to extract the values from your collection.
Note that we are not forced to pass a list to the do_something/1 function—the code will work with anything that is enumerable:
defmodule MyModule do
def do_something(collection) do
for el <- collection, do: el * 2
end
end
MyModule.do_something((1..3)) |> IO.inspect
In this example, I am passing a range as an argument.
Comprehensions work with binstrings as well. The syntax is slightly different as you need to enclose your generator with << and >>. Let's demonstrate this by crafting a very simple function to "decipher" a string protected with a Caesar cipher. The idea is simple: we replace each letter in the word with a letter a fixed number of positions down the alphabet. I'll shift by 1 position for simplicity:
defmodule MyModule do
def decipher(cipher) do
for << char <- cipher >>, do: char - 1
end
end
MyModule.decipher("fmjyjs") |>
IO.inspect
# => 'elixir'
This is looking pretty much the same as the previous example except for the << and >> parts. We take a code of each character in a string, decrement it by one, and construct a string back. So the ciphered message was "elixir"!
But still, there is more than that. Another useful feature of comprehensions is the ability to filter out some elements.
Comprehensions and Filtering
Let's further extend our initial example. I am going to pass a range of integers from 1 to 20, take only the elements that are even, and multiply them by 2:
defmodule MyModule do
require Integer
def do_something(collection) do
collection |>
Stream.filter( &Integer.is_even/1 ) |>
Enum.map( &(&1 * 2) )
end
end
MyModule.do_something( (1..20) ) |> IO.inspect
Here I had to require the Integer module to be able to use the is_even/1 macro. Also, I am using Stream to optimize the code a bit and prevent the iteration from being performed twice.
Now let's rewrite this example with a comprehension again:
def do_something(collection) do
for el <- collection, Integer.is_even(el), do: el * 2
end
So, as you see, for can accept an optional filter to skip some elements from the collection.
You are not limited to only one filter, so the following code is legit as well:
def do_something(collection) do
for el <- collection, Integer.is_even(el), el < 10, do: el * 2
end
It will take all even numbers less than 10. Just don't forget to delimit filters with commas.
The filters will be evaluated for each element of the collection, and if evaluation returns true, the block is executed. Otherwise, a new element is taken. What's interesting is that generators can also be used to filter out elements by using when:
def do_something(collection) do
for el when el < 10 <- collection, Integer.is_even(el), do: el * 2
end
This is very similar to what we do when writing guard clauses:
def do_something(x) when is_number(x) do
# ...
end
Comprehensions With Multiple Collections
Now suppose we have not one but two collections at once, and we'd like to produce a new collection. For example, take all even numbers from the first collection and odd from the second one, and then multiply them:
defmodule MyModule do
require Integer
def do_something(collection1, collection2) do
for el1 <- collection1, el2 <- collection2, Integer.is_even(el1), Integer.is_odd(el2),
do: el1 * el2
end
end
MyModule.do_something( (1..20), (5..10) ) |> IO.inspect
This example illustrates that comprehensions may work with more than one collection at once. The first even number from collection1 will be taken and multiplied by each odd number from collection2. Next, the second even integer from collection1 will be taken and multiplied, and so on. The result will be:
[10, 14, 18, 20, 28, 36, 30, 42, 54, 40, 56, 72, 50, 70, 90, 60, 84, 108, 70, 98, 126, 80, 112, 144, 90, 126, 162, 100, 140, 180]
What's more, the resulting values are not required to be integers. For instance, you may return a tuple containing integers from the first and the second collections:
defmodule MyModule do
require Integer
def do_something(collection1, collection2) do
for el1 <- collection1, el2 <- collection2, Integer.is_even(el1), Integer.is_odd(el2),
do: {el1,el2}
end
end
MyModule.do_something( (1..20), (5..10) ) |> IO.inspect
# => [{2, 5}, {2, 7}, {2, 9}, {4, 5}...]
Comprehensions With the "Into" Option
Up to this point, the final result of our comprehension was always a list. This is, actually, not mandatory either. You can specify an into parameter that accepts a collection to contain the resulting value.
This parameter accepts any structure that implements the Collectable protocol, so for example we may generate a map like this:
defmodule MyModule do
require Integer
def do_something(collection1, collection2) do
for el1 <- collection1, el2 <- collection2,
Integer.is_even(el1), Integer.is_odd(el2),
into: Map.new,
do: {el1,el2}
end
end
MyModule.do_something( (1..20), (5..10) ) |> IO.inspect
# => %{2 => 9, 4 => 9, 6 => 9...}
Here I simply said into: Map.new, which can be also replaced with into: %{}. By returning the {el1, el2} tuple, we basically set the first element as a key and the second as the value.
This example is not particularly useful, however, so let's generate a map with a number as a key and its square as a value:
defmodule MyModule do
def do_something(collection) do
for el <- collection, into: Map.new,
do: {el, :math.sqrt(el)}
end
end
squares = MyModule.do_something( (1..20) ) |> IO.inspect
# => %{1 => 1.0, 2 => 1.4142135623730951, 3 => 1.7320508075688772,...}
squares[3] |> IO.puts
# => 1.7320508075688772
In this example I am using Erlang's :math module directly, as, after all, all modules' names are atoms. Now you can easily find the square for any number from 1 to 20.
Comprehensions and Pattern Matching
The last thing to mention is that you can perform pattern matching in comprehensions as well. In some cases it may come in pretty handy.
Suppose we have a map containing employees' names and their raw salaries:
%{"Joe" => 50, "Bill" => 40, "Alice" => 45, "Jim" => 30}
I want to generate a new map where the names are downcased and converted to atoms, and salaries are calculated using a tax rate:
defmodule MyModule do
@tax 0.13
def format_employee_data(collection) do
for {name, salary} <- collection, into: Map.new,
do: {format_name(name), salary - salary * @tax}
end
defp format_name(name) do
name |>
String.downcase |>
String.to_atom
end
end
MyModule.format_employee_data( %{"Joe" => 50, "Bill" => 40, "Alice" => 45, "Jim" => 30} ) |>
IO.inspect
# => %{alice: 39.15, bill: 34.8, jim: 26.1, joe: 43.5}
In this example we define a module attribute @tax with an arbitrary number. Then I deconstruct the data in the comprehension using {name, salary} <- collection. Lastly, format the name and calculate the salary as needed, and store the result in the new map. Quite simple yet expressive.
Conclusion
In this article we have seen how to use Elixir comprehensions. You may need some time to get accustomed to them. This construct is really neat and in some situations can fit in much better than functions like map and filter. You can find some more examples in Elixir's official docs and the getting started guide.
Hopefully, you've found this tutorial useful and interesting! Thank you for staying with me, and see you soon.
Thursday, June 22, 2017
Using Celery With Django for Background Task Processing
Web applications usually start out simple but can become quite complex, and most of them quickly exceed the responsibility of only responding to HTTP requests.
When that happens, one must make a distinction between what has to happen instantly (usually in the HTTP request lifecycle) and what can happen eventually. Why is that? Well, because when your application becomes overloaded with traffic, simple things like this make the difference.
Operations in a web application can be classified as critical or request-time operations and background tasks, the ones that happen outside request time. These map to the ones described above:
- needs to happen instantly: request-time operations
- needs to happen eventually: background tasks
Request-time operations can be done on a single request/response cycle without worrying that the operation will time out or that the user might have a bad experience. Common examples include CRUD (Create, Read, Update, Delete) database operations and user management (Login/Logout routines).
Background tasks are different as they are usually quite time-consuming and are prone to failure, mostly due to external dependencies. Some common scenarios among complex web applications include:
- sending confirmation or activity emails
- daily crawling and scraping some information from various sources and storing them
- performing data analysis
- deleting unneeded resources
- exporting documents/photos in various formats
Background tasks are the main focus of this tutorial. The most common programming pattern used for this scenario is the Producer Consumer Architecture.
In simple terms, this architecture can be described like this:
- Producers create data or tasks.
- Tasks are put into a queue that is referred to as the task queue.
- Consumers are responsible for consuming the data or running the tasks.
Usually, the consumers retrieve tasks from the queue in a first-in-first-out (FIFO) fashion or according to their priorities. The consumers are also referred to as workers, and that is the term we will be using throughout, as it is consistent with the terminology used by the technologies discussed.
What kind of tasks can be processed in the background? Tasks that:
- are not essential for the basic functionality of the web application
- can't be run in the request/response cycle since they are slow (I/O intensive, etc.)
- depend on external resources that might not be available or not behave as expected
- might need to be retried at least once
- have to be executed on a schedule
Celery is the de facto choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. It supports various technologies for the task queue and various paradigms for the workers.
In this tutorial, we're going to create a Django toy web application (dealing with real-world scenarios) that uses background task processing.
Setting Things Up
Assuming you are already familiar with Python package management and virtual environments, let's install Django:
$ pip install Django
I've decided to build yet another blogging application. The focus of the application will be on simplicity. A user can simply create an account and without too much fuss can create a post and publish it to the platform.
Set up the quick_publisher Django project:
$ django-admin startproject quick_publisher
Let's get the app started:
$ cd quick_publisher $ ./manage.py startapp main
When starting a new Django project, I like to create a main application that contains, among other things, a custom user model. More often than not, I encounter limitations of the default Django User model. Having a custom User model gives us the benefit of flexibility.
# main/models.py
from django.db import models
from django.contrib.auth.models import AbstractBaseUser, PermissionsMixin, BaseUserManager
class UserAccountManager(BaseUserManager):
use_in_migrations = True
def _create_user(self, email, password, **extra_fields):
if not email:
raise ValueError('Email address must be provided')
if not password:
raise ValueError('Password must be provided')
email = self.normalize_email(email)
user = self.model(email=email, **extra_fields)
user.set_password(password)
user.save(using=self._db)
return user
def create_user(self, email=None, password=None, **extra_fields):
return self._create_user(email, password, **extra_fields)
def create_superuser(self, email, password, **extra_fields):
extra_fields['is_staff'] = True
extra_fields['is_superuser'] = True
return self._create_user(email, password, **extra_fields)
class User(AbstractBaseUser, PermissionsMixin):
REQUIRED_FIELDS = []
USERNAME_FIELD = 'email'
objects = UserAccountManager()
email = models.EmailField('email', unique=True, blank=False, null=False)
full_name = models.CharField('full name', blank=True, null=True, max_length=400)
is_staff = models.BooleanField('staff status', default=False)
is_active = models.BooleanField('active', default=True)
def get_short_name(self):
return self.email
def get_full_name(self):
return self.email
def __unicode__(self):
return self.email
Make sure to check out the Django documentation if you are not familiar with how custom user models work.
Now we need to tell Django to use this User model instead of the default one. Add this line to the quick_publisher/settings.py file:
AUTH_USER_MODEL = 'main.User'
We also need to add the main application to the INSTALLED_APPS list in the quick_publisher/settings.py file. We can now create the migrations, apply them, and create a superuser to be able to log in to the Django admin panel:
$ ./manage.py makemigrations main $ ./manage.py migrate $ ./manage.py createsuperuser
Let's now create a separate Django application that's responsible for posts:
$ ./manage.py startapp publish
Let's define a simple Post model in publisher/models.py:
from django.db import models
from django.utils import timezone
from django.contrib.auth import get_user_model
class Post(models.Model):
author = models.ForeignKey(get_user_model())
created = models.DateTimeField('Created Date', default=timezone.now)
title = models.CharField('Title', max_length=200)
content = models.TextField('Content')
slug = models.SlugField('Slug')
def __str__(self):
return '"%s" by %s' % (self.title, self.author)
Hooking the Post model with the Django admin is done in the publisher/admin.py file like this:
from django.contrib import admin
from .models import Post
@admin.register(Post)
class PostAdmin(admin.ModelAdmin):
pass
Finally, let's hook the publisher application with our project by adding it to the INSTALLED_APPS list.
We can now run the server and head over to http://localhost:8000/admin/ and create our first posts so that we have something to play with:
$ ./manage.py runserver
I trust you've done your homework and you've created the posts.
Let's move on. The next obvious step is to create a way to view the published posts.
# publisher/views.py
from django.http import Http404
from django.shortcuts import render
from .models import Post
def view_post(request, slug):
try:
post = Post.objects.get(slug=slug)
except Post.DoesNotExist:
raise Http404("Poll does not exist")
return render(request, 'post.html', context={'post': post})
Let's associate our new view with an URL in: quick_publisher/urls.py
# quick_publisher/urls.py
from django.conf.urls import url
from django.contrib import admin
from publisher.views import view_post
urlpatterns = [
url(r'^admin/', admin.site.urls),
url(r'^(?P<slug>[a-zA-Z0-9\-]+)', view_post, name='view_post')
]
Finally, let's create the template that renders the post in: publisher/templates/post.html
<!DOCTYPE html>
<html>
<head lang="en">
<meta charset="UTF-8">
<title></title>
</head>
<body>
<h1></h1>
<p></p>
<p>Published by on </p>
</body>
</html>
We can now head to http://localhost:8000/the-slug-of-the-post-you-created/ in the browser. It's not exactly a miracle of web design, but making good-looking posts is beyond the scope of this tutorial.
Sending Confirmation Emails
Here's the classic scenario:
- You create an account on a platform.
- You provide an email address to be uniquely identified on the platform.
- The platform checks you are indeed the owner of the email address by sending an email with a confirmation link.
- Until you perform the verification, you are not able to (fully) use the platform.
Let's add an is_verified flag and the verification_uuid on the User model:
# main/models.py
import uuid
class User(AbstractBaseUser, PermissionsMixin):
REQUIRED_FIELDS = []
USERNAME_FIELD = 'email'
objects = UserAccountManager()
email = models.EmailField('email', unique=True, blank=False, null=False)
full_name = models.CharField('full name', blank=True, null=True, max_length=400)
is_staff = models.BooleanField('staff status', default=False)
is_active = models.BooleanField('active', default=True)
is_verified = models.BooleanField('verified', default=False) # Add the `is_verified` flag
verification_uuid = models.UUIDField('Unique Verification UUID', default=uuid.uuid4)
def get_short_name(self):
return self.email
def get_full_name(self):
return self.email
def __unicode__(self):
return self.email
Let's use this occasion to add the User model to the admin:
from django.contrib import admin
from .models import User
@admin.register(User)
class UserAdmin(admin.ModelAdmin):
pass
Let's make the changes reflect in the database:
$ ./manage.py makemigrations $ ./manage.py migrate
We now need to write a piece of code that sends an email when a user instance is created. This is what Django signals are for, and this is a perfect occasion to touch this subject.
Signals are fired before/after certain events occur in the application. We can define callback functions that are triggered automatically when the signals are fired. To make a callback trigger, we must first connect it to a signal.
We're going to create a callback that will be triggered after a User model has been created. We'll add this code after the User model definition in: main/models.py
from django.db.models import signals
from django.core.mail import send_mail
def user_post_save(sender, instance, signal, *args, **kwargs):
if not instance.is_verified:
# Send verification email
send_mail(
'Verify your QuickPublisher account',
'Follow this link to verify your account: '
'http://localhost:8000%s' % reverse('verify', kwargs={'uuid': str(instance.verification_uuid)}),
'from@quickpublisher.dev',
[instance.email],
fail_silently=False,
)
signals.post_save.connect(user_post_save, sender=User)
What we've done here is we've defined a user_post_save function and connected it to the post_save signal (one that is triggered after a model has been saved) sent by the User model.
Django doesn't just send emails out on its own; it needs to be tied to an email service. For the sake of simplicity, you can add your Gmail credentials in quick_publisher/settings.py, or you can add your favourite email provider.
Here's what Gmail configuration looks like:
EMAIL_USE_TLS = True EMAIL_HOST = 'smtp.gmail.com' EMAIL_HOST_USER = '<YOUR_GMAIL_USERNAME>@gmail.com' EMAIL_HOST_PASSWORD = '<YOUR_GMAIL_PASSWORD>' EMAIL_PORT = 587
To test things out, go into the admin panel and create a new user with a valid email address you can quickly check. If all went well, you'll receive an email with a verification link. The verification routine is not ready yet.
Here's how to verify the account:
# main/views.py
from django.http import Http404
from django.shortcuts import render, redirect
from .models import User
def home(request):
return render(request, 'home.html')
def verify(request, uuid):
try:
user = User.objects.get(verification_uuid=uuid, is_verified=False)
except User.DoesNotExist:
raise Http404("User does not exist or is already verified")
user.is_verified = True
user.save()
return redirect('home')
Hook the views up in: quick_publisher/urls.py
# quick_publisher/urls.py
from django.conf.urls import url
from django.contrib import admin
from publisher.views import view_post
from main.views import home, verify
urlpatterns = [
url(r'^$', home, name='home'),
url(r'^admin/', admin.site.urls),
url(r'^verify/(?P<uuid>[a-z0-9\-]+)/', verify, name='verify'),
url(r'^(?P<slug>[a-zA-Z0-9\-]+)', view_post, name='view_post')
]
Also, remember to create a home.html file under main/templates/home.html. It will be rendered by the home view.
Try to run the entire scenario all over again. If all is well, you'll receive an email with a valid verification URL. If you'll follow the URL and then check in the admin, you can see how the account has been verified.
Sending Emails Asynchronously
Here's the problem with what we've done so far. You might have noticed that creating a user is a bit slow. That's because Django sends the verification email inside the request time.
This is how it works: we send the user data to the Django application. The application creates a User model and then creates a connection to Gmail (or another service you selected). Django waits for the response, and only then does it return a response to our browser.
Here is where Celery comes in. First, make sure it is installed:
$ pip install Celery
We now need to create a Celery application in our Django application:
# quick_publisher/celery.py
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'quick_publisher.settings')
app = Celery('quick_publisher')
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
Celery is a task queue. It receives tasks from our Django application, and it will run them in the background. Celery needs to be paired with other services that act as brokers.
Brokers intermediate the sending of messages between the web application and Celery. In this tutorial, we'll be using Redis. Redis is easy to install, and we can easily get started with it without too much fuss.
You can install Redis by following the instructions on the Redis Quick Start page. You'll need to install the Redis Python library, pip install redis, and the bundle necessary for using Redis and Celery: pip install celery[redis].
Start the Redis server in a separate console like this: $ redis-server
Let's add the Celery/Redis related configs into quick_publisher/settings.py:
# REDIS related settings
REDIS_HOST = 'localhost'
REDIS_PORT = '6379'
BROKER_URL = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}
CELERY_RESULT_BACKEND = 'redis://' + REDIS_HOST + ':' + REDIS_PORT + '/0'
Before anything can be run in Celery, it must be declared as a task.
Here's how to do this:
# main/tasks.py
import logging
from django.urls import reverse
from django.core.mail import send_mail
from django.contrib.auth import get_user_model
from quick_publisher.celery import app
@app.task
def send_verification_email(user_id):
UserModel = get_user_model()
try:
user = UserModel.objects.get(pk=user_id)
send_mail(
'Verify your QuickPublisher account',
'Follow this link to verify your account: '
'http://localhost:8000%s' % reverse('verify', kwargs={'uuid': str(user.verification_uuid)}),
'from@quickpublisher.dev',
[user.email],
fail_silently=False,
)
except UserModel.DoesNotExist:
logging.warning("Tried to send verification email to non-existing user '%s'" % user_id)
What we've done here is this: we moved the sending verification email functionality in another file called tasks.py.
A few notes:
- The name of the file is important. Celery goes through all the apps in
INSTALLED_APPSand registers the tasks intasks.pyfiles. - Notice how we decorated the
send_verification_emailfunction with@app.task. This tells Celery this is a task that will be run in the task queue. - Notice how we expect as argument
user_idrather than aUserobject. This is because we might have trouble serializing complex objects when sending the tasks to Celery. It's best to keep them simple.
Going back to main/models.py, the signal code turns into:
from django.db.models import signals
from main.tasks import send_verification_email
def user_post_save(sender, instance, signal, *args, **kwargs):
if not instance.is_verified:
# Send verification email
send_verification_email.delay(instance.pk)
signals.post_save.connect(user_post_save, sender=User)
Notice how we call the .delay method on the task object. This means we're sending the task off to Celery and we don't wait for the result. If we used send_verification_email(instance.pk) instead, we would still be sending it to Celery, but would be waiting for the task to finish, which is not what we want.
Before you start creating a new user, there's a catch. Celery is a service, and we need to start it. Open a new console, make sure you activate the appropriate virtualenv, and navigate to the project folder.
$ celery worker -A quick_publisher --loglevel=debug --concurrency=4
This starts four Celery process workers. Yes, now you can finally go and create another user. Notice how there's no delay, and make sure to watch the logs in the Celery console and see if the tasks are properly executed. This should look something like this:
[2017-04-28 15:00:09,190: DEBUG/MainProcess] Task accepted: main.tasks.send_verification_email[f1f41e1f-ca39-43d2-a37d-9de085dc99de] pid:62065 [2017-04-28 15:00:11,740: INFO/PoolWorker-2] Task main.tasks.send_verification_email[f1f41e1f-ca39-43d2-a37d-9de085dc99de] succeeded in 2.5500912349671125s: None
Periodic Tasks With Celery
Here's another common scenario. Most mature web applications send their users lifecycle emails in order to keep them engaged. Some common examples of lifecycle emails:
- monthly reports
- activity notifications (likes, friendship requests, etc.)
- reminders to accomplish certain actions ("Don't forget to activate your account")
Here's what we're going to do in our app. We're going to count how many times every post has been viewed and send a daily report to the author. Once every single day, we're going to go through all the users, fetch their posts, and send an email with a table containing the posts and view counts.
Let's change the Post model so that we can accommodate the view counts scenario.
class Post(models.Model):
author = models.ForeignKey(User)
created = models.DateTimeField('Created Date', default=timezone.now)
title = models.CharField('Title', max_length=200)
content = models.TextField('Content')
slug = models.SlugField('Slug')
view_count = models.IntegerField("View Count", default=0)
def __str__(self):
return '"%s" by %s' % (self.title, self.author)
As always, when we change a model, we need to migrate the database:
$ ./manage.py makemigrations $ ./manage.py migrate
Let's also modify the view_post Django view to count views:
def view_post(request, slug):
try:
post = Post.objects.get(slug=slug)
except Post.DoesNotExist:
raise Http404("Poll does not exist")
post.view_count += 1
post.save()
return render(request, 'post.html', context={'post': post})
It would be useful to display the view_count in the template. Add this <p>Viewed times</p> somewhere inside the publisher/templates/post.html file. Do a few views on a post now and see how the counter increases.
Let's create a Celery task. Since it is about posts, I'm going to place it in publisher/tasks.py:
from django.template import Template, Context
from django.core.mail import send_mail
from django.contrib.auth import get_user_model
from quick_publisher.celery import app
from publisher.models import Post
REPORT_TEMPLATE = """
Here's how you did till now:
"""
@app.task
def send_view_count_report():
for user in get_user_model().objects.all():
posts = Post.objects.filter(author=user)
if not posts:
continue
template = Template(REPORT_TEMPLATE)
send_mail(
'Your QuickPublisher Activity',
template.render(context=Context({'posts': posts})),
'from@quickpublisher.dev',
[user.email],
fail_silently=False,
)
Every time you make changes to the Celery tasks, remember to restart the Celery process. Celery needs to discover and reload tasks. Before creating a periodic task, we should test this out in the Django shell to make sure everything works as intended:
$ ./manage.py shell In [1]: from publisher.tasks import send_view_count_report In [2]: send_view_count_report.delay()
Hopefully, you received a nifty little report in your email.
Let's now create a periodic task. Open up quick_publisher/celery.py and register the periodic tasks:
# quick_publisher/celery.py
import os
from celery import Celery
from celery.schedules import crontab
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'quick_publisher.settings')
app = Celery('quick_publisher')
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
app.conf.beat_schedule = {
'send-report-every-single-minute': {
'task': 'publisher.tasks.send_view_count_report',
'schedule': crontab(), # change to `crontab(minute=0, hour=0)` if you want it to run daily at midnight
},
}
So far, we created a schedule that would run the task publisher.tasks.send_view_count_report every minute as indicated by the crontab() notation. You can also specify various Celery Crontab schedules.
Open up another console, activate the appropriate environment, and start the Celery Beat service.
$ celery -A quick_publisher beat
The Beat service's job is to push tasks in Celery according to the schedule. Take into account that the schedule makes the send_view_count_report task run every minute according to the setup. It's good for testing but not recommended for a real-world web application.
Making Tasks More Reliable
Tasks are often used to perform unreliable operations, operations that depend on external resources or that can easily fail due to various reasons. Here's a guideline for making them more reliable:
- Make tasks idempotent. An idempotent task is a task that, if stopped midway, doesn't change the state of the system in any way. The task either makes full changes to the system or none at all.
- Retry the tasks. If the task fails, it's a good idea to try it again and again until it's executed successfully. You can do this in Celery with Celery Retry. One other interesting thing to look at is the Exponential Backoff algorithm. This could come in handy when thinking about limiting unnecessary load on the server from retried tasks.
Conclusions
I hope this has been an interesting tutorial for you and a good introduction to using Celery with Django.
Here are a few conclusions we can draw:
- It's good practice to keep unreliable and time-consuming tasks outside the request time.
- Long-running tasks should be executed in the background by worker processes (or other paradigms).
- Background tasks can be used for various tasks that are not critical for the basic functioning of the application.
- Celery can also handle periodic tasks using the
celery beatservice. - Tasks can be more reliable if made idempotent and retried (maybe using exponential backoff).
Wednesday, June 21, 2017
Android Design Patterns: The Observer Pattern
What Is the Observer Pattern?
The Observer Pattern is a software design pattern that establishes a one-to-many dependency between objects. Anytime the state of one of the objects (the "subject" or "observable") changes, all of the other objects ("observers") that depend on it are notified.
Let's use the example of users that have subscribed to receive offers from Envato Market via email. The users in this case are observers. Anytime there is an offer from Envato Market, they get notified about it via email. Each user can then either buy into the offer or decide that they might not be really interested in it at that moment. A user (an observer) can also subscribe to receive offers from another e-commerce marketplace if they want and might later completely unsubscribe from receiving offers from any of them.
This pattern is very similar to the Publish-Subscribe pattern. The subject or observable publishes out a notification to the dependent observers without even knowing how many observers have subscribed to it, or who they are—the observable only knows that they should implement an interface (we'll get to that shortly), without worrying about what action the observers might perform.
Benefits of the Observer Pattern
- The subject knows little about its observers. The only thing it knows is that the observers implement or agree to a certain contract or interface.
- Subjects can be reused without involving their observers, and the same goes for observers too.
- No modification is done to the subject to accommodate a new observer. The new observer just needs to implement an interface that the subject is aware of and then register to the subject.
- An observer can be registered to more than one subject it's registered to.
All these benefits give you loose coupling between modules in your code, which enables you to build a flexible design for your application. In the rest of this post, we'll look at how to create our own Observer pattern implementation, and we'll also use the built-in Java Observer/Observable API as well as looking into third-party libraries that can offer such functionality.
Building Our Own Observer Pattern
1. Create the Subject Interface
We start by defining an interface that subjects (observables) will implement.
public interface Subject {
void registerObserver(RepositoryObserver repositoryObserver);
void removeObserver(RepositoryObserver repositoryObserver);
void notifyObservers();
}
In the code above, we created a Java Interface with three methods. The first method registerObserver(), as it says, will register an observer of type RepositoryObserver (we'll create that interface shortly) to the subject. removeObserver() will be called to remove an observer that wants to stop getting notifications from the subject, and finally, notifyObserver() will send a broadcast to all observers whenever there is a change. Now, let's create a concrete subject class that will implement the subject interface we have created:
import android.os.Handler;
import java.util.ArrayList;
public class UserDataRepository implements Subject {
private String mFullName;
private int mAge;
private static UserDataRepository INSTANCE = null;
private ArrayList<RepositoryObserver> mObservers;
private UserDataRepository() {
mObservers = new ArrayList<>();
getNewDataFromRemote();
}
// Simulate network
private void getNewDataFromRemote() {
final Handler handler = new Handler();
handler.postDelayed(new Runnable() {
@Override
public void run() {
setUserData("Chike Mgbemena", 101);
}
}, 10000);
}
// Creates a Singleton of the class
public static UserDataRepository getInstance() {
if(INSTANCE == null) {
INSTANCE = new UserDataRepository();
}
return INSTANCE;
}
@Override
public void registerObserver(RepositoryObserver repositoryObserver) {
if(!mObservers.contains(repositoryObserver)) {
mObservers.add(repositoryObserver);
}
}
@Override
public void removeObserver(RepositoryObserver repositoryObserver) {
if(mObservers.contains(repositoryObserver)) {
mObservers.remove(repositoryObserver);
}
}
@Override
public void notifyObservers() {
for (RepositoryObserver observer: mObservers) {
observer.onUserDataChanged(mFullName, mAge);
}
}
public void setUserData(String fullName, int age) {
mFullName = fullName;
mAge = age;
notifyObservers();
}
}
The class above implements the Subject interface. We have an ArrayList that holds the observers and then creates it in the private constructor. An observer registers by being added to the ArrayList and likewise, unregisters by being removed from the ArrayList.
Note that we are simulating a network request to retrieve the new data. Once the setUserData() method is called and given the new value for the full name and age, we call the notifyObservers() method which, as it says, notifies or sends a broadcast to all registered observers about the new data change. The new values for the full name and age are also passed along. This subject can have multiple observers but, in this tutorial, we'll create just one observer. But first, let's create the observer interface.
2. Create the Observer Interface
public interface RepositoryObserver {
void onUserDataChanged(String fullname, int age);
}
In the code above, we created the observer interface which concrete observers should implement. This allows our code to be more flexible because we are coding to an interface instead of a concrete implementation. A concrete Subject class does not need to be aware of the many concrete observers it may have; all it knows about them is that they implement the RepositoryObserver interface.
Let's now create a concrete class that implements this interface.
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.widget.TextView;
public class UserProfileActivity extends AppCompatActivity implements RepositoryObserver {
private Subject mUserDataRepository;
private TextView mTextViewUserFullName;
private TextView mTextViewUserAge;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_user_profile);
mTextViewUserAge = (TextView) findViewById(R.id.tv_age);
mTextViewUserFullName = (TextView) findViewById(R.id.tv_fullname);
mUserDataRepository = UserDataRepository.getInstance();
mUserDataRepository.registerObserver(this);
}
@Override
public void onUserDataChanged(String fullname, int age) {
mTextViewUserFullName.setText(fullname);
mTextViewUserAge.setText(age);
}
@Override
protected void onDestroy() {
super.onDestroy();
mUserDataRepository.removeObserver(this);
}
}
The first thing to notice in the code above is that UserProfileActivity implements the RepositoryObserver interface—so it must implement the method onUserDataChanged(). In the onCreate() method of the Activity, we got an instance of the UserDataRepository which we then initialized and finally registered this observer to.
In the onDestroy() method, we want to stop getting notifications, so we unregister from receiving notifications.
In the onUserDataChanged() method, we want to update the TextView widgets—mTextViewUserFullName and mTextViewUserAge—with the new set of data values.
Right now we just have one observer class, but it's possible and easy for us to create other classes that want to be observers of the UserDataRepository class. For example, we could easily have a SettingsActivity that wants to also be notified about the user data changes by becoming an observer.
Push and Pull Models
In the example above, we are using the push model of the observer pattern. In this model, the subject notifies the observers about the change by passing along the data that changed. But in the pull model, the subject will still notify the observers, but it does not actually pass the data that changed. The observers then pull the data they need once they receive the notification.
Utilising Java's Built-In Observer API
So far, we have created our own Observer pattern implementation, but Java has built-in Observer / Observable support in its API. In this section, we are going to use this. This API simplifies some of the implementation, as you'll see.
1. Create the Observable
Our UserDataRepository—which is our subject or observable—will now extend the java.util.Observable superclass to become an Observable. This is a class that wants to be observed by one or more observers.
import android.os.Handler;
import java.util.Observable;
public class UserDataRepository extends Observable {
private String mFullName;
private int mAge;
private static UserDataRepository INSTANCE = null;
private UserDataRepository() {
getNewDataFromRemote();
}
// Returns a single instance of this class, creating it if necessary.
public static UserDataRepository getInstance() {
if(INSTANCE == null) {
INSTANCE = new UserDataRepository();
}
return INSTANCE;
}
// Simulate network
private void getNewDataFromRemote() {
final Handler handler = new Handler();
handler.postDelayed(new Runnable() {
@Override
public void run() {
setUserData("Mgbemena Chike", 102);
}
}, 10000);
}
public void setUserData(String fullName, int age) {
mFullName = fullName;
mAge = age;
setChanged();
notifyObservers();
}
public String getFullName() {
return mFullName;
}
public int getAge() {
return mAge;
}
}
Now that we have refactored our UserDataRepository class to use the Java Observable API, let's see what has changed compared to the previous version. The first thing to notice is that we are extending a super class (this means that this class can't extend any other class) and not implementing an interface as we did in the previous section.
We are no longer holding an ArrayList of observers; this is handled in the super class. Similarly, we don't have to worry about registration, removal, or notification of observers—java.util.Observable is handling all of those for us.
Another difference is that in this class we are employing a pull style. We alert the observers that a change has happened with notifyObservers(), but the observers will need to pull the data using the field getters we have defined in this class. If you want to use the push style instead, then you can use the method notifyObservers(Object arg) and pass the changed data to the observers in the object argument.
The setChanged() method of the super class sets a flag to true, indicating that the data has changed. Then you can call the notifyObservers() method. Be aware that if you don't call setChanged() before calling notifyObsevers(), the observers won't be notified. You can check the value of this flag by using the method hasChanged() and clear it back to false with clearChanged(). Now that we have our observable class created, let's see how to set up an observer also.
2. Create the Observer
Our UserDataRepository observable class needs a corresponding Observer to be useful, so let's refactor our UserProfileActivity to implement the java.util.Observer interface.
import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.widget.TextView;
import com.chikeandroid.tutsplusobserverpattern.R;
import java.util.Observable;
import java.util.Observer;
public class UserProfileActivity extends AppCompatActivity implements Observer {
private Observable mUserDataRepositoryObservable;
private TextView mTextViewUserFullName;
private TextView mTextViewUserAge;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_user_profile);
mTextViewUserAge = (TextView) findViewById(R.id.tv_age);
mTextViewUserFullName = (TextView) findViewById(R.id.tv_fullname);
mUserDataRepositoryObservable = UserDataRepository.getInstance();
mUserDataRepositoryObservable.addObserver(this);
}
@Override
public void update(Observable observable, Object o) {
if (observable instanceof UserDataRepository) {
UserDataRepository userDataRepository = (UserDataRepository)observable;
mTextViewUserAge.setText(String.valueOf(userDataRepository.getAge()));
mTextViewUserFullName.setText(userDataRepository.getFullName());
}
}
@Override
protected void onDestroy() {
super.onDestroy();
mUserDataRepositoryObservable.deleteObserver(this);
}
}
In the onCreate() method, we add this class as an observer to the UserDataRepository observable by using the addObserver() method in the java.util.Observable super class.
In the update() method which the observer must implement, we check if the Observable we receive as a parameter is an instance of our UserDataRepository (note that an observer can subscribe to different observables), and then we cast it to that instance and retrieve the values we want using the field getters. Then we use those values to update the view widgets.
When the activity is destroyed, we don't need to get any updates from the observable, so we'll just remove the activity from the observer list by calling the method deleteObserver().
Libraries to Implement an Observer Pattern
If you don't want to build your own Observer pattern implementation from scratch or use the Java Observer API, you can use some free and open-source libraries that are available for Android such as Greenrobot's EventBus. To learn more about it, check out my tutorial here on Envato Tuts+.
- AndroidCommunication Within an Android App With EventBus
Or, you might like RxAndroid and RxJava. Learn more about them here:
- AndroidGetting Started With ReactiveX on Android
Conclusion
In this tutorial, you learned about the Observer pattern in Java: what is it, the benefits of using it, how to implement your own, using the Java Observer API, and also some third-party libraries for implementing this pattern.
In the meantime, check out some of our other courses and tutorials on the Java language and Android app development!
- Android SDKRxJava 2 for Android Apps: RxBinding and RxLifecycle
- Android SDKPractical Concurrency on Android With HaMeR
- AndroidEnsure High-Quality Android Code With Static Analysis Tools
- Android SDKCreate an Intelligent App With Google Cloud Speech and Natural Language APIs
Tuesday, June 20, 2017
Monday, June 19, 2017
Sunday, June 18, 2017
Saturday, June 17, 2017
Friday, June 16, 2017
How to Handle Exceptions in Python
It is very common to encounter errors during the execution of a program. Two common kinds of errors that you may have to deal with are syntax errors and exceptions. Syntax errors occur when you type the code incorrectly. In such cases, the erroneous line is repeated by the parser with an arrow pointing to the earliest location where the error was detected.
Exceptions are different from syntax errors. They occur during the execution of a program when something unexpected happens. For example, let's say you are asking the user to input a number in order to perform a division. Now, if the user enters a string instead of a number and you try to divide a number by the given input, the program will output a TypeError.
When you are not properly handling exceptions, the program will abruptly quit as it won't know what to do in such cases. The following code is such an example:
keep_asking = True
while keep_asking:
x = int(input("Enter a number: "))
print("Dividing 50 by", x,"will give you :", 50/x)
As long as you are entering an integral input value, the program will work correctly. However, as soon as you enter a string or even a decimal number as input, you will get a ValueError exception.
In this tutorial, you will learn how to properly handle and raise exceptions in Python.
Some Common Exceptions
Here are some basic exceptions that you might encounter when writing programs. You can read about many more built-in exceptions on the official website.
- NameError: This exception is raised when the program cannot find a local or global name. The name that could not be found is included in the error message.
- TypeError: This exception is raised when a function is passed an object of the inappropriate type as its argument. More details about the wrong type are provided in the error message.
- ValueError: This exception occurs when a function argument has the right type but an inappropriate value.
- NotImplementedError: This exception is raised when an object is supposed to support an operation but it has not been implemented yet. You should not use this error when the given function is not meant to support the type of input argument. In those situations, raising a
TypeErrorexception is more appropriate. - ZeroDivisionError: This exception is raised when you provide the second argument for a division or modulo operation as zero.
- FileNotFoundError: This exception is raised when the file or directory that the program requested does not exist.
Just like the names above, most exceptions have self-explanatory names.
Handling an Exception
The code at the beginning of the article asked users to enter an integer as input. If the user did not provide an integer input, the program stopped execution and raised a value error exception. In this section, we will write some code to tell the user that their input is not a valid integer value.
The first step of the process is to include the code that you think might raise an exception inside the try clause. The next step is to use the except keyword to handle the exception that occurred in the above code. The modified code for the user input will look like this:
keep_asking = True
while keep_asking:
try:
x = int(input("Please enter a number: "))
print("Dividing 50 by", x,"will give you :", 50/x)
except ValueError:
print("The input was not an integer. Please try again...")
What happens here is that the program tries to execute the code inside the try clause. If no exception was raised, the program skips the except clause and the rest of the code executes normally. If an exception is raised, the program skips the remaining code inside the try clause and the type of the exception is matched with the name of the exception after the except keyword. In case of a match, the code inside the except clause is executed first, and then the rest of the code after the try clause is executed normally.
When you enter an integer as an input, the program gives you the final result of the division. When a non-integral value is provided, the program prints a message asking you to try and enter an integer again. Note that this time, the program does not abruptly quit when you provide some invalid input.
You can have multiple except clauses to handle different exceptions. Please keep in mind that these handlers will only deal with exceptions that occurred in the corresponding try clause. They will not handle any exceptions raised within other exception handlers.
You can also handle multiple exceptions using a single except clause by passing these exceptions to the clause as a tuple.
except (ZeroDivisionError, ValueError, TypeError):
print("Something has gone wrong..")
# code to deal with the exception
Finally, you can also leave out the name of the exception after the except keyword. This is generally not recommended as the code will now be catching all the exceptions and handling them in the same way. This is not optimal as you will be handling a TypeError exception the same way as you would have handled a ZeroDivisionError exception. When handling exceptions, it is better to be as specific as possible and only catch what you can handle.
One possible use of catching all exceptions is to properly print out the exception error on screen like the following code:
import math
import sys
try:
result = math.factorial(2.4)
except:
print("Something Unexpected has happened.",sys.exc_info()[0])
else:
print("The factorial is", result)
Using the Else Clause
You can also use an else clause in a try ... except statement. The else clause is meant to contain code that needs to be executed if the try clause did not raise any exceptions. This can be useful to make sure that you don't add any code to the try block whose exceptions you don't intend to catch. One thing worth mentioning is that if you decide to use an else clause, you should include it after all the except clauses but before the finally block.
In our case, we could move the line that prints the result of our division inside the else block.
keep_asking = True
while keep_asking:
try:
x = int(input("Please enter a number: "))
except ValueError:
print("The input was not a valid integer. Please try again...")
else:
print("Dividing 50 by", x,"will give you :", 50/x)
Cleaning Up Using the Finally Clause
Let's say you have written some code inside the try block that is supposed to perform a task by utilizing a large amount of resources. It is important to release those resources back when you are done using them. This can be easily achieved by using the finally clause.
The code inside the finally clause is always executed irrespective of whether the try block raised an exception. You can verify this by running the following code:
keep_asking = True
while keep_asking:
try:
x = int(input("Please enter a number: "))
except ValueError:
print("The input was not a valid integer. Please try again...")
else:
print("Dividing 50 by", x,"will give you :", 50/x)
finally:
print("Already did everything necessary.")
If any of the except clauses that you specified do not handle the raised exception, the same exception is raised again after the execution of code inside the finally block.
A More Complex Example
In this section, we will write a program to deal with multiple exceptions. Just like the previous examples, we will be performing some mathematical operations. However, this time we will take the input from a list.
The following code checks for two exceptions, TypeError and ValueError. The else block is used to print the factorial. You can see in the output that this code is executed only when no exception is raised.
import math
number_list = [10,-5,1.2,'apple']
for number in number_list:
try:
number_factorial = math.factorial(number)
except TypeError:
print("Factorial is not supported for given input type.")
except ValueError:
print("Factorial only accepts positive integer values.", number," is not a positive integer.")
else:
print("The factorial of",number,"is", number_factorial)
finally:
print("Release any resources in use.")
The above code produces the following output:
The factorial of 10 is 3628800 Releasing any resources in use. Factorial only accepts positive integer values. -5 is not a positive integer. Releasing any resources in use. Factorial only accepts positive integer values. 1.2 is not a positive integer. Releasing any resources in use. Factorial is not supported for given input type. Releasing any resources in use.
Another thing worth noticing is that the code inside the finally clause is executed for each item in the list.
Final Thoughts
I hope this tutorial helped you understand exception handling in Python. Additionally, don’t hesitate to see what we have available for sale and for study in the marketplace, and don't hesitate to ask any questions and provide your valuable feedback using the feed below.
Properly handling exceptions can be very helpful in situations where exiting a program after it receives an unexpected input is not viable. If you have any questions related to exception handling in Python, please let me know in the comments.
