Wednesday, August 31, 2016

Learn MEAN Stack [93% Of] https://t.co/ENvm9pVysp #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment CSS : CSS Awesomeness [100% Off] https://t.co/tSvvf0pijx


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Pls share your opinion in comment: Which one do you choose - Yii2 vs Laravel Comparison? https://t.co/oLpp91zysm https://t.co/fwIvrl0CMl


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

CSS : CSS Awesomeness [100% Off] https://t.co/JkiGbJIwsB #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#html5 #css3 FlipHTML5 provides free eBook publisher to modernized ebooks #html5 #css3 https://t.co/fQYT7XldRZ


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Create gaming site with 10000 games [Free] https://t.co/sw2INGtGH7 #webdesign


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Learn MEAN Stack [93% Of] https://t.co/ENvm9pDXAR #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Create gaming site with 10000 games [Free] https://t.co/sw2INGc5Pz #webdesign


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

The Python Bible | Everything You Need to Program in Python! https://t.co/7z0bbN9ufi #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

What is Zapier? How to Automate Your Business Tasks Better

Posted by at No comments:
Labels:

Learn Web Scraping with Node.js [100% Off] https://t.co/fBwIQ94ypK #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Learn Web Scraping with Node.js [100% Off] https://t.co/cyhj4sbYuD


from Twitter https://twitter.com/Web_improve

Posted by at 1 comment:
Labels: ,

20 Best WordPress Directory Themes: For Business Listing Sites and More

Posted by at No comments:
Labels:

The Portable Guitarist—Hardware

The Portable Guitarist—Hardware

The my first tutorial, The Portable Guitarist—Using iOS as a Live Rig, I explained why an iOS-based live rig has many advantages. In this tutorial, I'll explain the core of the set-up: the hardware.

Before you start, you should consider the reason that you'll be using the device.

Consider Why You'll Be Using an iOS Device

Guitartists often use time-based apps like delays and reverbs. If, however, the device has insufficient memory, the sound will glitch and stutter terribly. Before choosing a device to use, bear this in mind: the more processing power and memory you can get, the better. 

Therefore, when looking at the specifications of a preferred device (which you can find on the relevant product page of the Apple website), pay attention to:

  • Which generation of processor is onboard, the more recent, the better
  • How much memory the device has: be aware that when a 16GB memory is quoted, the reality is nearer 12GB due to system requirements

Don’t spend everything on the device. It may be the centre of your sound set-up, but there’ll be peripherals to buy, so budget accordingly.

For further savings, check out the Refurbished section of the Apple Store. Not only are items cheaper, but they’ll be as-new, and come with a 1-year Apple warranty.

An iPad Is the Sensible Choice

Personally, I favour the iPad. If nothing else, a bigger screen is easier to see on a darkened stage (even if iPhones seem to get bigger with each iteration). Plus, on-screen adjustments are easier; no-one wants to hear you accidentally engaging your death metal tone during a sensitive ballad.

Then there’s battery life; having used both an iPad and iPhone live, the battery life of an iPhone can plummet alarmingly. True, if it’s fully charged to start with, you’re unlikely to play long enough to drain it down, and a break between sets lets you recharge it. However, I’d rather focus on my performance than the battery icon.

Remember to check the Apple store for refurbished iPads.

Don't Dismiss the iPhone or iPod Touch

Screen size and battery life aside, the iPhone and iPod Touch suffer little in specification; currently, the iPhone 6, 6 Plus, and iPod Touch all share the same processor as the iPad mini 4, and are all available up to 64GB. The iPhone 6S and 6S Plus offer improved processor and a maximum of 128GB memory.

It can also be cheaper; Apple currently charges £159 for the 16GB iPod Touch. If you can afford it, the top-of-the-range 128GB is £329; compare this to the iPhone 6S at £539, and the iPad Air 2 at £559.

So they can be cheaper, and they’re certainly even more portable. They generally run the same apps as the iPad, certainly the guitar-centric ones, and accept the same interfaces.

Interfaces

There are generally two types of interface: 

  1. headphone socket, and 
  2. dock 

Headphone Socket

These were the original iOS interface, and the iRig by IK Multimedia is probably the most famous. Beautifully straightforward, and attractively cheap, it was a trailblazer. 

In 2015 it was superseded by the iRig2, which is a little bigger, and has more features such as input gain control. Crucially, it’s still cheap; the iRig2 costs under £30. The original iRig is still available, and I’ve found it for as little as £15 new.

However, this wonderful cheapness comes at a price: noise. Lots and lots of noise.

The headphone socket is supposed to be for output, so introducing an input signal in such close proximity creates hiss, and can lead to feedback.

The Ampkit Link from Peavey tried to address this by being battery-powered, and it certainly meant a higher signal level could be achieved before feedback occurred. But it didn’t cure it, so guitarists relied heavily on noise gates. 

A better solution is that of a dock interface.

Dock Interfaces

Unlike the headphone socket, the dock doesn’t rely on just three connections, so input and output signals can be kept separate, significantly cleaning up the sound. It also frees the headphone socket to do what it was designed to do.

Of the models on the market, and ignoring those of a desktop nature, my choice is the Jam from Apogee. It’s extremely compact, and requires no external power source. 

Other features include the integral gain control which means it can also be used with certain microphones. It is compatible with both iOS and Mac for live performance and home recording. It also still serves the 30-pin format of older devices, alongside that of current, Lightning-equipped ones.

Quality of sound is what matters, and the Jam’s 24-bit, 48kHz digital converters don’t disappoint. If that isn’t good enough for you, there’s a 96kHz version.

All of this sonic goodness does come at a higher price, however; a 48kHz Jam is typically over £70, and the 96Khz is over £100. If you want to go slightly cheaper, check out the SonicPort from Line 6.

Hands Free

Using iOS live has meant having one's hands more on a screen than perhaps many guitarists would like; this becomes increasingly applicable if you use lots of different sounds.

Bluetooth-based foot controllers, however, exploit the fact that many apps accept MIDI commands. Consequently, switching pedals onscreen becomes no different to that of kicking old-school stompboxes on and off.

As for what’s available, the iRig BlueBoard from IK Multimedia has lots to recommend it; four backlit footpads, expression pedal inputs, fits in a gig bag, and works with most iDevices, as well as Mac.

If plastic seems flimsy, check out the all-metal BT-4 from Positive Grid; more expensive, but more robust.

Conclusion

In looking at the core of a live set-up, I have shown you that

  • An iPad is a preferable option
  • An iPhone or iPod Touch are be cheaper and aren’t necessarily a poorer choice
  • Go for the highest specification you can afford
  • Apple Refurbished models can save you money
  • A dock interface is the only serious choice for sound quality
  • A Bluetooth foot controller brings familiar analogue functionality

In the next tutorial, I’ll show you how to mount and connect equipment effectively for a live environment.


Posted by at No comments:
Labels:

Starting Game Development with Python https://t.co/SoWcp77Y90 #codeschool


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Starting Game Development with Python #codeschool


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

HTML Web Development Crash Course [100% Off] https://t.co/7PT9kzUWkH #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment HTML Web Development Crash Course [100% Off] https://t.co/bOheBFJTij


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

JavaScript Rockstar How to create Incredible Useful code [100% Off] https://t.co/s1kWDLML0b #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment JavaScript Rockstar How to create Incredible Useful code [100% Off] https://t.co/FQjTwAx4Gg


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Design a New Brand Identity for Your Business

Posted by at No comments:
Labels:

Let's Go: Command-Line Programs With Golang

Let's Go: Command-Line Programs With Golang

Overview

The Go Language is an exciting new language that gains a lot of popularity for a good reason. In this tutorial you'll learn how to write command-line programs with Go. The sample program is called multi-git, and it allows you to execute git commands on multiple repositories at the same time.

Quick Introduction to Go

Go is an open-source C-like language created at Google by some of the original C and Unix hackers, who were motivated by their dislike of C++. It shows in Go's design, which made several unorthodox choices such as eschewing implementation inheritance, templates, and exceptions. Go is simple, reliable, and efficient. Its most distinctive feature is its explicit support for concurrent programming via so-called goroutines and channels.

Before starting to dissect the sample program, follow the official guide to get ready for Go development.

The Multi-Git Program

The multi-git program is a simple but useful Go program. If you work on a team where the codebase is split across multiple git repositories then you often need to perform changes across multiple repositories. This is a problem because git has no concept of multiple repositories. Everything revolves around a single repository. 

This becomes especially troublesome if you use branches. If you work on a feature that touches three repositories then you will have to create a feature branch in each of these repositories and then remember to check out, pull, push, and merge all of them at the same time. This is not trivial. Multi-git manages a set of repositories and lets you operate on the whole set at once. Note that the current version of multi-git requires that you create the branches individually, but I may add this feature at a later date.

By exploring the way multi-git is implemented, you will learn a lot about writing command-line programs in Go.

Packages and Imports

Go programs are organized in packages. The multi-git program consists of a single file called main.go. At the top of the file, the package name 'main' is specified, followed by a list of imports. The imports are other packages that are used by multi-git.

package main



import (

    "flag"

    "fmt"

    "log"

    "os"

    "strings"

    "os/exec"

)

For example, the fmt package is used for formatted I/O similar to C's printf and scanf. Go supports installing packages from a variety of sources via the go get command. When you install packages, they end up in a namespace under the $GOPATH environment variable. You can install packages from a variety of sources such as GitHub, Bitbucket, Google code, Launchpad, and even IBM DevOps services via several common version control formats such as git, subversion, mercurial and bazaar.

Command-Line Arguments

Command-line arguments are one of the most common forms of providing input to programs. They are easy to use, allow you to run and configure the program in one line, and have great parsing support in many languages. Go calls them command-line "flags" and has the flag package for specifying and parsing command-line arguments (or flags). 

Typically, you parse command-line arguments at the beginning of your program, and multi-git follows this convention. The entry point is the main() function. The first two lines define two flags called "command" and "ignoreErrors". Each flag has a name, a data type, a default value, and a help string. The flag.Parse() call will parse the actual command-line passed to the program and will populate the defined flags.

func main() {

    command := flag.String("command", "", "The git command")

    ignoreErrors := flag.Bool(

        "ignore-errors",

        false,

        "Keep running after error if true")

    flag.Parse()

It is also possible to access undefined arguments via the flag.Args() function. So, flags stand for pre-defined arguments and "args" are unprocessed arguments. The unprocessed arguments are 0-based indexed.

Environment Variables

Another common form of program configuration is environment variables. When you use environment variables, you may run the same program multiple times in the same environment, and all runs will use the same environment variables. 

Multi-git uses two environment variables: "MG_ROOT" and "MG_REPOS". Multi-git is designed to manage a group of git repositories that have a common parent directory. That's "MG_ROOT". The repository names are specified in "MG_REPOS" as a comma-separated string. To read the value of an environment variable you can use the os.Getenv() function.

    // Get managed repos from environment variables

    root := os.Getenv("MG_ROOT")

    if root[len(root) - 1] != '/' {

        root += "/"

    }



    repo_names := strings.Split(os.Getenv("MG_REPOS"), ",")

Verifying the Repository List

Now that it found the root directory and the names of all the repositories, multi-git verifies that each repository exists under root and that it is really a git repository. The check is as simple as looking for a .git sub-directory for each repository directory.

First, an array of strings named "repos" is defined. Then it iterates over all the repo names and constructs a repository path by concatenating the root directory and the repo name. If the [os.Stat()]() call fails for the .git subdirectory, it logs the error and exits. Otherwise, the repository path is appended to the repos array.

    var repos []string

    // Verify all repos exist and are actually git repos (have .git sub-dir)

    for _, r := range repo_names {

        path := root + r

        _, err := os.Stat(path + "/.git")

        if err != nil {

            log.Fatal(err)

        }

        repos = append(repos, path)

    }

Go has a unique error-handling facility where functions often return both a return value and an error object. Check out how os.Stat() returns two values. In this case the "_" placeholder is used to hold the actual result because you only care about the error. Go is very strict and requires named variables to be used. If you don't plan to use a value, you should assign it to "_" to avoid compilation error.

Executing Shell Commands

At this point, you have your list of repository paths where we want to execute the git command. As you recall, we received the git command line as a single command-line argument (flag) called "command". This needs to be split into an array of components (git command, sub-command, and options). The whole command as a string is stored too for display purposes.

    // Break the git command into components (needed to execute)

    var git_components []string

    for _, component := range strings.Split(*command, " ") {

        git_components = append(git_components, component)

    }

    command_string := "git " + *command

Now, you're all set to iterate over each repository and execute the git command in each one. The "for ... range" loop construct is used again. First, multi-git changes its working directory to the current target repo "r" and prints the git command. Then it executes the command using the exec.Command() function and prints the combined output (both standard output and standard error). 

Finally, it checks if there was an error during execution. If there was an error and the ignoreErrors flag is false then multi-git bails out. The reason for optionally ignoring errors is that sometimes it's OK if commands fail on some repos. For example, if you want to check out a branch called "cool feature" on all the repositories that have this branch, you don't care if the checkout fails on repositories that don't have this branch.

    for _, r := range repos {

        // Go to the repo's directory

        os.Chdir(r);



        // Print the command

        fmt.Printf("[%s] %s\n", r, command_string)



        // Execute the command

        out, err := exec.Command("git", git_components...).CombinedOutput()



        // Print the result

        fmt.Println(string(out))



        // Bail out if there was an error and NOT ignoring errors

        if err != nil && !*ignoreErrors {

            os.Exit(1)

        }

    }



    fmt.Println("Done.")

Conclusion

Go is a simple yet powerful language. It's designed for large-scale system programming but works just fine for small command-line programs too. Go's minimal design is in stark contrast to other modern languages like Scale and Rust that are very powerful and well-designed too, but have a very steep learning curve. I encourage you to try Go and experiment. It's a lot of fun.


Posted by at No comments:
Labels:

How to Create a Coloring Book Style Illustration in Adobe Illustrator

Posted by at No comments:
Labels:

Angular 2 Crash Course with TypeScript [100% Off] https://t.co/xWYIgM7nfA #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Angular 2 Crash Course with TypeScript [100% Off] https://t.co/TE0kIFmgDi


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Perfectly Retouch Makeup for Beauty and Fashion Photography in 5 Steps

Posted by at No comments:
Labels:

Top 21 Popular Programming Languages Of 2016 https://t.co/7JGypBm1DO #codeschool


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How To Build A Buzzfeed Style Blog On WordPress [100% Off] https://t.co/BRvpq9GKoh #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment How To Build A Buzzfeed Style Blog On WordPress [100% Off] https://t.co/jqvaloFVgo


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

iOS 10 and Swift 3 for Beginners Best Tutorials https://t.co/UtLz1yzG7i #codeschool


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Here Is What to Look For When You Buy Photography Lenses

Posted by at No comments:
Labels:

Are you a developer? Build a Universal React and Node App Step-By-Step. Just for you! ☞ https://t.co/rsnXQvXsRI https://t.co/6h7YQ7rt7F


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Corporative Complete Family (64 different fonts) – only $19! https://t.co/ZXnGYfj4AP #developerdeals


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Tuesday, August 30, 2016

Creating a Low Poly Aeroplane Set for Games: Part 2

Posted by at No comments:
Labels:

How To Creating an MP3 Player with HTML5 Step-By-Step Try It Now! ☞ https://t.co/nDjKx1a9mi https://t.co/tmXnmXtcRt


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Run an Effective Brainstorming Session

Posted by at No comments:
Labels:

How to Find a Great Job and Get Hired (In the Next 30 Days)

Posted by at No comments:
Labels:

How to Create WordPress Pages With Hierarchy and Templates

Posted by at No comments:
Labels:

How to Build a Responsive UI Component Using Element Queries

Posted by at No comments:
Labels:

#html5 #css3 Opening For Sr. Web Application UI Developer #html5 #css3 https://t.co/e0nvzUCw7o


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Create a Multi-Image Twitter Header Image With Adobe Photoshop

Posted by at No comments:
Labels:

#Webdevelopment Creating Animations using HTML5 Canvas [60% Off] https://t.co/5FakfLt9dN


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Creating Animations using HTML5 Canvas [60% Off] https://t.co/L2nunOQUoX #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Creating an MP3 Player with HTML5 [50% Off] https://t.co/lok2pywX2u #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Creating an MP3 Player with HTML5 [50% Off] https://t.co/gAGr3ttbUa


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Installing AMP in WordPress

Posted by at No comments:
Labels:

#webdevelop #webdesigns Blog: Sponsored: Tenorshare Any Data Recovery for Mac: An in-depth review of Tenorshare's … https://t.co/aekXsfJiXj


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#webdevelop #webdesigns Sponsored: Tenorshare Any Data Recovery for Mac: Everyone loses files at some point and ba… https://t.co/ggVyuTYIOW


from Twitter https://twitter.com/Web_improve

Posted by at 1 comment:
Labels: ,

How to Draw a Car From Scratch

Posted by at No comments:
Labels:

#Webdevelopment Bootstrap Basics: Program Responsive Websites [87% Off] https://t.co/gUiH88irKg


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Bootstrap Basics: Program Responsive Websites [87% Off] https://t.co/didGhWNW92 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Add Custom Callouts to Screencast Videos in Screenflow

Posted by at No comments:
Labels:

Mighty Deals Exclusive! 500+ Linear Vector Elements & Icons - only $14! https://t.co/iimrLzw5ec #developerdeals


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Monday, August 29, 2016

How to Stabilize Video in After Effects with ReelSteady

How to Stabilize Video in After Effects with ReelSteady

In this tutorial you will learn how to get started using the plug-in ReelSteady to stabilize footage in Adobe After Effects. We will take a look at a few common scenarios shot on various formats, such as a DSLR and GoPro cameras. You will also learn about the best settings to use on your camera before filming shots you plan on stabilizing in post.

What You Need

Besides After Effects, in order to follow along with this lesson you will need to download the plug-in ReelSteady. A free-trial version is available to download as well, so you can easily follow along with this tutorial and experiment using ReelSteady on your own footage.

Tips before you Shoot

  • Shoot with a high shutter speed (I usually shoot around 200 to 320.)
  • Use a wide-angle lens. A wide-angle or fish-eye lens work best, but if you don't have one of those, just shoot with the widest lens you have available.
  • Shoot in 4K, or the highest resolution you can. More pixels means more tracking data and less of a chance for pixelization to occur.
  • Film using the 'Tripod Trick'. If you have a tripod, attached it to the bottom of your camera and hold it while moving or walking. This will help prevent any high frequency vibration that can cause excessive rolling shutter artifacts.
  • Be aware of parallax between objects in your scene.

How to Use ReelSteady to Stabilize Footage in Adobe After Effects

Links Mentioned





Posted by at No comments:
Labels:

The best twitter tool on the market! auto-follow, unfollow, follow back, tweet, retwee ..etc https://t.co/XbpjBgQBOs https://t.co/OWUmL5H08a


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

5 Inspirational Baby Photographs and How to Make Your Own

Posted by at No comments:
Labels:

How to Set Up a Gmail (Out of Office) Vacation Responder Email

Posted by at No comments:
Labels:

#code #HTML5 #webdesigns #css #ruby My Love letter to Weapons of Mass Creation https://t.co/SmMeoKV59P


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#code #HTML5 #webdesigns #css #ruby Four ways to convert visitors to leads https://t.co/35csFz86Kb


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Serialization and Deserialization of Python Objects: Part 2

Serialization and Deserialization of Python Objects: Part 2

This is part two of a tutorial on serializing and deserializing Python objects. In part one, you learned the basics and then dove into the ins and outs of Pickle and JSON. 

In this part you'll explore YAML (make sure to have the running example from part one), discuss performance and security considerations, get a review of additional serialization formats, and finally learn how to choose the right scheme.

YAML

YAML is my favorite format. It is a human-friendly data serialization format. Unlike Pickle and JSON, it is not part of the Python standard library, so you need to install it:

pip install yaml

The yaml module has only load() and dump() functions. By default they work with strings like loads() and dumps(), but can take a second argument, which is an open stream and then can dump/load to/from files.

import yaml



print yaml.dump(simple)



boolean: true

int_list: [1, 2, 3]

none: null

number: 3.44

text: string

Note how readable YAML is compared to Pickle or even JSON. And now for the coolest part about YAML: it understands Python objects! No need for custom encoders and decoders. Here is the complex serialization/deserialization using YAML:

> serialized = yaml.dump(complex)

> print serialized



a: !!python/object:__main__.A

  simple:

    boolean: true

    int_list: [1, 2, 3]

    none: null

    number: 3.44

    text: string

when: 2016-03-07 00:00:00



> deserialized = yaml.load(serialized)

> deserialized == complex

True

As you can see, YAML has its own notation to tag Python objects. The output is still very human readable. The datetime object doesn't require any special tagging because YAML inherently supports datetime objects. 

Performance

Before you start thinking of performance, you need to think if performance is a concern at all. If you serialize/deserialize a small amount of data relatively infrequently (e.g. reading a config file at the beginning of a program) then performance is not really a concern and you can move on.

But, assuming you profiled your system and discovered that serialization and/or deserialization are causing performance issues, here are the things to address.

The are two aspects for performance: how fast can you serialize/deserialize, and how big is the serialized representation?

To test the performance of the various serialization formats, I'll create a largish data structure and serialize/deserialize it using Pickle, YAML, and JSON. The big_data list contains 5,000 complex objects.

big_data = [dict(a=simple, when=datetime.now().replace(microsecond=0)) for i in range(5000)]

Pickle

I'll use IPython here for its convenient %timeit magic function that measures execution times.

import cPickle as pickle



In [190]: %timeit serialized = pickle.dumps(big_data)

10 loops, best of 3: 51 ms per loop



In [191]: %timeit deserialized = pickle.loads(serialized)

10 loops, best of 3: 24.2 ms per loop



In [192]: deserialized == big_data

Out[192]: True



In [193]: len(serialized)

Out[193]: 747328

The default pickle takes 83.1 milliseconds to serialize and 29.2 milliseconds to deserialize, and the serialized size is 747,328 bytes.

Let's try with the highest protocol.

In [195]: %timeit serialized = pickle.dumps(big_data, protocol=pickle.HIGHEST_PROTOCOL)

10 loops, best of 3: 21.2 ms per loop



In [196]: %timeit deserialized = pickle.loads(serialized)

10 loops, best of 3: 25.2 ms per loop



In [197]: len(serialized)

Out[197]: 394350

Interesting results. The serialization time shrank to only 21.2 milliseconds, but the deserialization time increased a little to 25.2 milliseconds. The serialized size shrank significantly to 394,350 bytes (52%).

JSON

In [253] %timeit serialized = json.dumps(big_data, cls=CustomEncoder)

10 loops, best of 3: 34.7 ms per loop



In [253] %timeit deserialized = json.loads(serialized, object_hook=decode_object)

10 loops, best of 3: 148 ms per loop



In [255]: len(serialized)

Out[255]: 730000

Ok. Performance seems to be a little worse than Pickle for encoding, but much, much worse for decoding: 6 times slower. What's going on? This is an artifact of the object_hook function that needs to run for every dictionary to check if it needs to convert it to an object. Running without the object hook is much faster.

%timeit deserialized = json.loads(serialized)

10 loops, best of 3: 36.2 ms per loop

The lesson here is that when serializing and deserializing to JSON, consider very carefully any custom encodings because they may have a major impact on the overall performance.

YAML

In [293]: %timeit serialized = yaml.dump(big_data)

1 loops, best of 3: 1.22 s per loop



In[294]: %timeit deserialized = yaml.load(serialized)

1 loops, best of 3: 2.03 s per loop



In [295]: len(serialized)

Out[295]: 200091

Ok. YAML is really, really slow. But, note something interesting: the serialized size is just 200,091 bytes. Much better than both Pickle and JSON. Let's look inside real quick:

In [300]: print serialized[:211]

- a: &id001

    boolean: true

    int_list: [1, 2, 3]

    none: null

    number: 3.44

    text: string

  when: 2016-03-13 00:11:44

- a: *id001

  when: 2016-03-13 00:11:44

- a: *id001

  when: 2016-03-13 00:11:44

YAML is being very clever here. It identified that all 5,000 dicts share the same value for the 'a' key, so it stores it only once and references it using *id001 for all objects.

Security

Security is an often a critical concern. Pickle and YAML, by virtue of constructing Python objects, are vulnerable to code execution attacks. A cleverly formatted file can contain arbitrary code that will be executed by Pickle or YAML. There is no need to be alarmed. This is by design and is documented in Pickle's documentation:

Warning: The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

As well as in YAML's documentation:

Warning: It is not safe to call yaml.load with any data received from an untrusted source! yaml.load is as powerful as pickle.load and so may call any Python function.

You just need to understand that you shouldn't load serialized data received from untrusted sources using Pickle or YAML. JSON is OK, but again if you have custom encoders/decoders than you may be exposed, too.

The yaml module provides the yaml.safe_load() function that will load only simple objects, but then you lose a lot of YAML's power and maybe opt to just use JSON.

Other Formats

There are many other serialization formats available. Here are a few of them.

Protobuf

Protobuf, or protocol buffers, is Google's data interchange format. It is implemented in C++ but has Python bindings. It has a sophisticated schema and packs data efficiently. Very powerful, but not very easy to use.

MessagePack

MessagePack is another popular serialization format. It is also binary and efficient, but unlike Protobuf it doesn't require a schema. It has a type system that's similar to JSON, but a little richer. Keys can be any type, and not just strings and non-UTF8 strings are supported.

CBOR

CBOR stands for Concise Binary Object Representation. Again, it supports the JSON data model. CBOR is not as well-known as Protobuf or MessagePack but is interesting for two reasons: 

  1. It is an official Internet standard: RFC 7049.
  2. It was designed specifically for the Internet of Things (IoT).

How to Choose?

This is the big question. With so many options, how do you choose? Let's consider the various factors that should be taken into account:

  1. Should the serialized format be human-readable and/or human-editable?
  2. Is serialized content going to be received from untrusted sources?
  3. Is serialization/deserialization a performance bottleneck?
  4. Does serialized data need to be exchanged with non-Python environments?

I'll make it very easy for you and cover several common scenarios and which format I recommend for each one:

Auto-Saving Local State of a Python Program

Use pickle (cPickle) here with the HIGHEST_PROTOCOL. It's fast, efficient and can store and load most Python objects without any special code. It can be used as a local persistent cache also.

Configuration Files

Definitely YAML. Nothing beats its simplicity for anything humans need to read or edit. It's used successfully by Ansible and many other projects. In some situations, you may prefer to use straight Python modules as configuration files. This may be the right choice, but then it's not serialization, and it's really part of the program and not a separate configuration file.

Web APIs

JSON is the clear winner here. These days, Web APIs are consumed most often by JavaScript web applications that speak JSON natively. Some Web APIs may return other formats (e.g. csv for dense tabular result sets), but I would argue that you can package csv data into JSON with minimal overhead (no need to repeat each row as an object with all the column names). 

High-Volume / Low-Latency Large-Scale Communication

Use one of the binary protocols: Protobuf (if you need a schema), MessagePack, or CBOR. Run your own tests to verify the performance and the representative power of each option.

Conclusion

Serialization and deserialization of Python objects is an important aspect of distributed systems. You can't send Python objects directly over the wire. You often need to interoperate with other systems implemented in other languages, and sometimes you just want to store the state of your program in persistent storage. 

Python comes with several serialization schemes in its standard library, and many more are available as third-party modules. Being aware of all the options and the pros and cons of each one will let you choose the best method for your situation.


Posted by at No comments:
Labels:

Building a WordPress-Powered Front End With the WP REST API and AngularJS: The Posts, Categories, and Users Controllers

Posted by at No comments:
Labels:

How to Create a Fantasy Fairy Photo Manipulation With Adobe Photoshop

Posted by at No comments:
Labels:

Android From Scratch: Hardware Sensors

Posted by at No comments:
Labels:

How to Create a Pokémon Themed Icon Pack in Adobe Illustrator

Posted by at No comments:
Labels:

An Introduction to Remote Usability Testing

Posted by at No comments:
Labels:

#Webdevelopment The Docker for DevOps course: From development to production [60% Off] https://t.co/z7e0KYgH1e


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

The Docker for DevOps course: From development to production [60% Off] https://t.co/fwx7Akto2P #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Mockup Scene Generator Bundle: I Am Creator Perspective Edition - only $14! https://t.co/fZR1dySwcQ #developerdeals


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Sunday, August 28, 2016

How to Use a Mac as a Media Centre With Plex

Posted by at No comments:
Labels:

Typography Mastery for UI Designers and Developers [80% Off] https://t.co/fEMNzDlIC7 #webdesign


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Angular 2 – Superheroic Framework [50% Off] https://t.co/77YYF1nsIm


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Angular 2 – Superheroic Framework [50% Off] https://t.co/GGwS0AoHek #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Learn By Example: Angular JS [50% Off] https://t.co/Ytbmgy6fPw #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Learn By Example: Angular JS [50% Off] https://t.co/mw2YK2sTME


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Learn Rails: Quickly Code, Style and Launch 4 Web Apps [50% Off] https://t.co/NHjiBOS8po


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Learn Rails: Quickly Code, Style and Launch 4 Web Apps [50% Off] https://t.co/VnT6FWzGFv #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

The Complete Web Development Tutorial Using React and Redux [50% Off] https://t.co/29yrhJGLZ8 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment The Complete Web Development Tutorial Using React and Redux [50% Off] https://t.co/YG2ppLDjLW


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Build a Local Message Board in Ruby on Rails [Free] https://t.co/vuRIfE371j


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Build a Local Message Board in Ruby on Rails [Free] https://t.co/3z3fIhHCHq #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Responsive Website Template from Scratch HTML CSS [Free] https://t.co/KtxTWr6S60 #webdesign


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Saturday, August 27, 2016

React Redux React-Router: From Beginner to Paid Professional [$20 Only] https://t.co/1ft9cff0db #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment React Redux React-Router: From Beginner to Paid Professional [$20 Only] https://t.co/zlpvl9JuJM


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment HTML5 & CSS3 : Landing Pages for Entrepreneurs 2016 [75% Off] https://t.co/obwLQhKUfd


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

HTML5 & CSS3 : Landing Pages for Entrepreneurs 2016 [75% Off] https://t.co/3ByOoN4Mol #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Beginners Guide to JavaScript Dynamic HTML interaction [71% Off] https://t.co/VY9hIBKniI #webdesign


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

You can create Incredible Websites with HTML CSS learn how [100% Off] https://t.co/HehkDYNC2V #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment You can create Incredible Websites with HTML CSS learn how [100% Off] https://t.co/QVDnWPTVKO


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Build Responsive Website Using HTML5, CSS3, JS And Bootstrap [100% Off] https://t.co/xRwJxU1Qmo


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Build Responsive Website Using HTML5, CSS3, JS And Bootstrap [100% Off] https://t.co/lzePAH2nAB #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Build a Responsive Website with HTML5, CSS3 and Bootstrap 4 [100% Off] https://t.co/P5FPT73rOt #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Build a Responsive Website with HTML5, CSS3 and Bootstrap 4 [100% Off] https://t.co/uEs1D4gEs0


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Learn How To Build A Corporate Website Using Joomla 2.5 [Free] https://t.co/0m9yyFDv48 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Learn How To Build A Corporate Website Using Joomla 2.5 [Free] https://t.co/LHDq8ZWdUu


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

eBook: Marketing for Developers Guide - only $14! https://t.co/l1rM93AFNC #developerdeals


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Friday, August 26, 2016

Modern Web Development with Laravel 5.2 (PHP Framework) [67% Off] https://t.co/3duWv4LEB8 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Modern Web Development with Laravel 5.2 (PHP Framework) [67% Off] https://t.co/26kmoCUyhN


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

amazing extension. Definitely the best twitter automation tool I have ever seen #RoboTwity https://t.co/ED6L8NmxK7


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Introduction to Machine Learning & Face Detection in Python [50% Off] https://t.co/r7JXBGy55K #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Create A Custom Responsive WordPress Website For A Client [89% Off] https://t.co/D27vg4xVUf


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Create A Custom Responsive WordPress Website For A Client [89% Off] https://t.co/hsocMKvs3Z #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

How to Get the Most Dynamic Range from Your iPhone Camera

Posted by at No comments:
Labels:

Local Development Environments for Web Design [Free] https://t.co/w2I1uT9NXN #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Local Development Environments for Web Design [Free] https://t.co/IDAATtm4lc


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Envato Tuts+ Community Challenge: Created By You, August 2016 Edition

Posted by at No comments:
Labels:

#html5 #css3 PRAIRIE PEOPLE: For the week of Aug. 26 #html5 #css3 https://t.co/b6qzbmWKs5


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#webdevelop #webdesigns Blog: Share what makes Linux amazing; win an actual penguin*: To celebrate the 25th annive… https://t.co/qvSshKxU16


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

New Course: Essential WordPress Plugins

Posted by at No comments:
Labels:

How to Create a Hand-Lettering Textured Poster in Adobe Illustrator

Posted by at No comments:
Labels:

https://t.co/Ro6tc8PSh9 MVC Quick Start [Free] https://t.co/ip3jpNPpk7 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment https://t.co/Ro6tc8PSh9 MVC Quick Start [Free] https://t.co/CFQ6u7JFxX


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Corpo Font Family: 24 Sans & Serif Typefaces - only $15! https://t.co/rRND8OQ2CX #developerdeals


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Thursday, August 25, 2016

Learn Python Through Exercises [50% Off] https://t.co/ppkUYX6D8y #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

C# Developers: Double Your Coding Speed with Visual Studio [71% Off] https://t.co/GstQyICCSY #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Understanding Java easily! through case studies of real life [Free] https://t.co/beePXY59xN #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment Basic Structure of HTML5-For Beginners [88% Off] https://t.co/laiMZF5C9N


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Basic Structure of HTML5-For Beginners [88% Off] https://t.co/1Bx51erMh1 #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

The Ultimate Go (golang) Programming Tutorial [Free] https://t.co/c2ssCexWOz #programming


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#Webdevelopment 1 Hour HTML [90% Off – $10 Only] https://t.co/LNuD6Oxm3N


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

1 Hour HTML [90% Off – $10 Only] https://t.co/JIm0IYQmef #webdevelopment


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

#code #HTML5 #webdesigns #css #ruby Download of the Day: Vintage Sunbeam Vectors https://t.co/tyZl33Mxhe


from Twitter https://twitter.com/Web_improve

Posted by at No comments:
Labels: ,

Serialization and Deserialization of Python Objects: Part 1

Serialization and Deserialization of Python Objects: Part 1

Python object serialization and deserialization is an important aspect of any non-trivial program. If in Python you save something to a file, if you read a configuration file, or if you respond to an HTTP request, you do object serialization and deserialization. 

In one sense, serialization and deserialization are the most boring things in the world. Who cares about all the formats and protocols? You just want to persist or stream some Python objects and get them back later intact. 

This is a very healthy way to look at the world at the conceptual level. But, at the pragmatic level, which serialization scheme, format or protocol you choose may determine how fast your program runs, how secure it is, how much freedom you have to maintain your state, and how well you're going to interoperate with other systems. 

The reason there are so many options is that different circumstances call for different solutions. There is no "one size fits all". In this two-part tutorial I'll go over the pros and cons of the most successful serialization and deserialization schemes, show how to use them, and provide guidelines for choosing between them when faced with a specific use case.

Running Example

In the following sections I'll serialize and deserialize the same Python object graphs using different serializers. To avoid repetition, I'll define these object graphs here.

Simple Object Graph

The simple object graph is a dictionary that contains a list of integers, a string, a float, a boolean, and a None.

simple = dict(int_list=[1, 2, 3],

              text='string',

              number=3.44,

              boolean=True,

              none=None)

Complex Object Graph

The complex object graph is also a dictionary, but it contains a datetime object and user-defined class instance that has a self.simple attribute, which is set to the simple object graph.

from datetime import datetime



class A(object):

    def __init__(self, simple):

        self.simple = simple        

    def __eq__(self, other):

        if not hasattr(other, 'simple'):

            return False

        return self.simple == other.simple

    def __ne__(self, other):

        if not hasattr(other, 'simple'):

            return True

        return self.simple != other.simple



complex = dict(a=A(simple), when=datetime(2016, 3, 7))

Pickle

Pickle is a staple. It is a native Python object serialization format. The pickle interface provides four methods: dump, dumps, load, and loads. The dump() method serializes to an open file (file-like object). The dumps() method serializes to a string. The load() method deserializes from an open file-like object. The loads() method deserializes from a string.

Pickle supports by default a textual protocol, but has also a binary protocol, which is more efficient, but not human-readable (helpful when debugging).

Here is how you pickle a Python object graph to a string and to a file using both protocols.

import cPickle as pickle



pickle.dumps(simple)

"(dp1\nS'text'\np2\nS'string'\np3\nsS'none'\np4\nNsS'boolean'\np5\nI01\nsS'number'\np6\nF3.4399999999999999\nsS'int_list'\np7\n(lp8\nI1\naI2\naI3\nas."



pickle.dumps(simple, protocol=pickle.HIGHEST_PROTOCOL)

'\x80\x02}q\x01(U\x04textq\x02U\x06stringq\x03U\x04noneq\x04NU\x07boolean\x88U\x06numberq\x05G@\x0b\x85\x1e\xb8Q\xeb\x85U\x08int_list]q\x06(K\x01K\x02K\x03eu.'

The binary representation may seem larger, but this is an illusion due to its presentation. When dumping to a file, the textual protocol is 130 bytes, while the binary protocol is only 85 bytes.

pickle.dump(simple, open('simple1.pkl', 'w'))

pickle.dump(simple, open('simple2.pkl', 'wb'), protocol=pickle.HIGHEST_PROTOCOL)



ls -la sim*.*

-rw-r--r--  1 gigi  staff  130 Mar  9 02:42 simple1.pkl

-rw-r--r--  1 gigi  staff   85 Mar  9 02:43 simple2.pkl

Unpickling from a string is as simple as:

x = pickle.loads("(dp1\nS'text'\np2\nS'string'\np3\nsS'none'\np4\nNsS'boolean'\np5\nI01\nsS'number'\np6\nF3.4399999999999999\nsS'int_list'\np7\n(lp8\nI1\naI2\naI3\nas.")

assert x == simple



x = pickle.loads('\x80\x02}q\x01(U\x04textq\x02U\x06stringq\x03U\x04noneq\x04NU\x07boolean\x88U\x06numberq\x05G@\x0b\x85\x1e\xb8Q\xeb\x85U\x08int_list]q\x06(K\x01K\x02K\x03eu.')

assert x == simple

Note that pickle can figure out the protocol automatically. There is no need to specify a protocol even for the binary one.

Unpickling from a file is just as easy. You just need to provide an open file.

x = pickle.load(open('simple1.pkl'))

assert x == simple



x = pickle.load(open('simple2.pkl'))

assert x == simple



x = pickle.load(open('simple2.pkl', 'rb'))

assert x == simple

According to the documentation, you're supposed to open binary pickles using the 'rb' mode, but as you can see it works either way.

Let's see how pickle deals with the complex object graph.

pickle.dumps(complex)

"(dp1\nS'a'\nccopy_reg\n_reconstructor\np2\n(c__main__\nA\np3\nc__builtin__\nobject\np4\nNtRp5\n(dp6\nS'simple'\np7\n(dp8\nS'text'\np9\nS'string'\np10\nsS'none'\np11\nNsS'boolean'\np12\nI01\nsS'number'\np13\nF3.4399999999999999\nsS'int_list'\np14\n(lp15\nI1\naI2\naI3\nassbsS'when'\np16\ncdatetime\ndatetime\np17\n(S'\\x07\\xe0\\x03\\x07\\x00\\x00\\x00\\x00\\x00\\x00'\ntRp18\ns."



pickle.dumps(complex, protocol=pickle.HIGHEST_PROTOCOL)

'\x80\x02}q\x01(U\x01ac__main__\nA\nq\x02)\x81q\x03}q\x04U\x06simpleq\x05}q\x06(U\x04textq\x07U\x06stringq\x08U\x04noneq\tNU\x07boolean\x88U\x06numberq\nG@\x0b\x85\x1e\xb8Q\xeb\x85U\x08int_list]q\x0b(K\x01K\x02K\x03eusbU\x04whenq\x0ccdatetime\ndatetime\nq\rU\n\x07\xe0\x03\x07\x00\x00\x00\x00\x00\x00\x85Rq\x0eu.'



pickle.dump(complex, open('complex1.pkl', 'w'))

pickle.dump(complex, open('complex2.pkl', 'wb'), protocol=pickle.HIGHEST_PROTOCOL)



ls -la comp*.*

-rw-r--r--  1 gigi  staff  327 Mar  9 02:58 complex1.pkl

-rw-r--r--  1 gigi  staff  171 Mar  9 02:58 complex2.pkl

The efficiency of the binary protocol is even greater with complex object graphs.

JSON

JSON (JavaScript Object Notation) has been part of the Python standard library since Python 2.5. I'll consider it a native format at this point. It is a text-based format and is the unofficial king of the web as far as object serialization goes. Its type system naturally models JavaScript, so it is pretty limited. 

Let's serialize and deserialize the simple and complex objects graphs and see what happens. The interface is almost identical to the pickle interface. You have dump(), dumps(), load(), and loads() functions. But, there are no protocols to select, and there are many optional arguments to control the process. Let's start simple by dumping the simple object graph without any special arguments:

import json

print json.dumps(simple)

{"text": "string", "none": null, "boolean": true, "number": 3.44, "int_list": [1, 2, 3]}

The output looks pretty readable, but there is no indentation. For a larger object graph, this can be a problem. Let's indent the output:

print json.dumps(simple, indent=4)

{

    "text": "string",

    "none": null,

    "boolean": true,

    "number": 3.44,

    "int_list": [

        1,

        2,

        3

    ]

}

That looks much better. Let's move on to the complex object graph.

json.dumps(complex)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-19-1be2d89d5d0d> in <module>()

----> 1 json.dumps(complex)



/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)

    241         cls is None and indent is None and separators is None and

    242         encoding == 'utf-8' and default is None and not sort_keys and not kw):

--> 243         return _default_encoder.encode(obj)

    244     if cls is None:

    245         cls = JSONEncoder



/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in encode(self, o)

    205         # exceptions aren't as detailed.  The list call should be roughly

    206         # equivalent to the PySequence_Fast that ''.join() would do.

--> 207         chunks = self.iterencode(o, _one_shot=True)

    208         if not isinstance(chunks, (list, tuple)):

    209             chunks = list(chunks)



/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in iterencode(self, o, _one_shot)

    268                 self.key_separator, self.item_separator, self.sort_keys,

    269                 self.skipkeys, _one_shot)

--> 270         return _iterencode(o, 0)

    271

    272 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,



/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in default(self, o)

    182

    183         """

--> 184         raise TypeError(repr(o) + " is not JSON serializable")

    185

    186     def encode(self, o):



TypeError: <__main__.A object at 0x10f367cd0> is not JSON serializable

Whoa! That doesn't look good at all. What happened? The error message is that the A object is not JSON serializable. Remember that JSON has a very limited type system and it can't serialize user defined classes automatically. The way to address it is to subclass the JSONEncoder class used by the json module and implement the default() that is called whenever the JSON encoder runs into an object it can't serialize. 

The job of the custom encoder is to convert it to a Python object graph that the JSON encoder is able to encode. In this case we have two objects that require special encoding: the datetime object and the A class. The following encoder does the job. Each special object is converted to a dict where the key is the name of the type surrounded by dunders (double underscores). This will be important for decoding. 

from datetime import datetime

import json





class CustomEncoder(json.JSONEncoder):

     def default(self, o):

         if isinstance(o, datetime):

             return {'__datetime__': o.replace(microsecond=0).isoformat()}

         return {'__{}__'.format(o.__class__.__name__): o.__dict__}

Let's try again with our custom encoder:

serialized = json.dumps(complex, indent=4, cls=CustomEncoder)

print serialized



{

    "a": {

        "__A__": {

            "simple": {

                "text": "string",

                "none": null,

                "boolean": true,

                "number": 3.44,

                "int_list": [

                    1,

                    2,

                    3

                ]

            }

        }

    },

    "when": {

        "__datetime__": "2016-03-07T00:00:00"

    }

}

This is beautiful. The complex object graph was serialized properly, and the original type information of the components was retained via the keys: "__A__" and "__datetime__". If you use dunders for your names, then you need to come up with a different convention to denote special types.

Let's decode the complex object graph.

> deserialized = json.loads(serialized)

> deserialized == complex

False

Hmmm, the deserialization worked (no errors), but it is different than the original complex object graph we serialized. Something is wrong. Let's take a look at the deserialized object graph. I'll use the pprint function of the pprint module for pretty printing.

> from pprint import pprint

> pprint(deserialized)

{u'a': {u'__A__': {u'simple': {u'boolean': True,

                               u'int_list': [1, 2, 3],

                               u'none': None,

                               u'number': 3.44,

                               u'text': u'string'}}},

 u'when': {u'__datetime__': u'2016-03-07T00:00:00'}}

Ok. The problem is that the json module doesn't know anything about the A class or even the standard datetime object. It just deserializes everything by default to the Python object that matches its type system. In order to get back to a rich Python object graph, you need custom decoding. 

There is no need for a custom decoder subclass. The load() and loads() functions provide the "object_hook" parameter that lets you provide a custom function that converts dicts to objects. 

def decode_object(o):

    if '__A__' in o:

        a = A()

        a.__dict__.update(o['__A__'])

        return a

    elif '__datetime__' in o:

        return datetime.strptime(o['__datetime__'], '%Y-%m-%dT%H:%M:%S')        

    return o

Let's decode using the decode_object() function as a parameter to the loads() object_hook parameter.

> deserialized = json.loads(serialized, object_hook=decode_object)

> print deserialized

{u'a': <__main__.A object at 0x10d984790>, u'when': datetime.datetime(2016, 3, 7, 0, 0)}



> deserialized == complex

True

Conclusion

In part one of this tutorial, you've learned about the general concept of serialization and deserialization of Python objects and explored the ins and out of serializing Python objects using Pickle and JSON. 

In part two, you'll learn about YAML, performance and security concerns, and a quick review of additional serialization schemes.


Posted by at No comments:
Labels:
Subscribe to: Posts (Atom)