Trip to Europe

Had business trip to Europe for 2 weeks, visited Barcelona, Berlin and Amsterdam.

Barcelona

Berlin

Read more...

Acceptance Testing for Web App

I'd like share an approach we use at Qubell to ensure quality of our product. A set of practices that allows us to use Acceptance Testing in simple and efficient manner.

Note: this article is my personal opinion and can't be associated with Qubell's official view on the matter in any way.

Test Case

Let's try to express the test case with plain english first. A little about the product itself

  • the Qubell Platform simplifies deployment. You write requirements for your Application, hit the launch button - and your Application get delivered into production.

On the screenshot below you can see list of sample applications.

Read more...

Making Crawler Fast and Reliable

So, as I told in the previous article - the basic version of the Crawler worked well and proved to be usable. The problem - it was slow and unstable.

To make it fast we need to run it on multiple machines (about 5 - 20). And to make it stable we need to figure out how to make reliable system from unreliable components.

The unreliable component - Crawler uses Browser Emulator (Selenium) with enabled JavaScript to properly render content of sites. And it consumes lots of resources and frequently crash and hangs (it is not very stable by itself and what's worse - there maybe invalid HTML or JS on different sites that can crush or hang it).

Going distributed

Multiple machines instead of just one make things a bit complex because couple issues arise:

  • Provisioning and deployment couple of tens of machines. I don't want to do it by hands.
  • Handle crashes of machines and heal after it. Crawler should be robust and continue to work if one of its nodes crashes, and pick it up again when this node get fixed or new node get added.
  • Detect if one of its nodes hanged and need to be rebooted.

Read more...

Sochi Svetlana

Read more...

Crawling JavaScript Sites

Recently finished project - Web Crawler for JavaScript sites. It crawl eCommerce sites and collects information about the products. You can see parsed products for one of such eCommerce sites below.

Usually crawlers browse site pages, collect HTML from it, parse and extract some data.

But, in our case we can't use such approach - because sites we are interested in aren't usual sites. There are no any useful data in HTML sent by server, because they rendered using JavaScript in the Browser. And to parse such sites you need full Browser with working JavaScript Engine.

Browser Emulator

Read more...

Sochi Ahun

Read more...

Sochi Zapolyarye

Read more...

MongoDB Performance Tuning

Best feature of MongoDB is not it's performance but simple and flexible data model. So, let's say you build prototype - you concentrate on the big picture - the product itself and ignore little things like performance and db indexes.

Later you deploy your product into the wild users came and it starting to get slow. You need to add indexes, to do so you need to know data usage patterns. Doing it manually by searching codebase is boring and not very productive. Thankfully MongoDB has Profiler - all you need is to enable it and it will give you all details about slow queries and what indexes you need to add.

I like this approach very much, because it fits iterative & lean development very well - you always concentrate on the most important things at the moment. At the first step most important thing is to experiment with the product and features without being distracted by performance issues. And flexible data model of MongoDB comes very handy to that. Later you deploy product into production and can use its Profiler to zoom to more fine grained performance details.

Example

Read more...

Ruby on Rails vs Node.js

I wanted to see benchmark that more or less close to real life, not just measuring how fast it can stream data via http.

So I created application that query some text from remote HTTP service (service delays each request for 200ms) and render HTML page using that text.

It simulates how Web Framework performs when it needs to wait for response from DB or other services, how fast its templating engine is and also how fast it is itself.

And hit it with wrk -t2 -c100 -d10s http://localhost:3000 you can see results on the picture.

Benchmark Ruby on Rails vs. Node.js

3782 vs. 2914 hits, average response time is also very close.

Read more...

Cart Widget

Created simple Cart Widget http://salejs.com that can be included into any HTML page and turn it into Shop.

There's also tutorial and screencast explaining development step by step http://jslang.info/projects/salejs

Cart Widget

Read more...

California

Read more...

Prototyping Mobile Web Application

We decided to spend one evening prototyping very simple and small but unusual thing for our product.

I choose to build very basic mobile prototype. Actually our app is already responsive and can work on mobile devices, but it doesn't looks very good. I wanted it looks like a native application.

It took me about 4 hours to build this prototype (Backbone, Ratchet and CoffeeScript), you can see it live here code on GitHub.

Prototyping Mobile Web Application

Read more...

Client-Side QA

Video (russian language) and slides.

Presentation was short and I didn't have a time to cover many interesting things like testing strategy, named scopes and matchers.

I'll try to add additional materials and code samples in next months, slides are below.

Read more...

Countries with the best medicine

How can we decide if medicine is good or bad? About couple of years ago I found very interesting statistics, but only recently had time to investigate it in details.

Countries with people having the longest life

So, we can roughly guess that if lifespan is big, then the medicine should be probably good too. Also, as soon as lifespan is also affected by lots of other variables like - climate, food, ecology, lifestyle, income and so on - we can decide that they are probably good too.

But, the fact that lifespan is a sum of many variables has a drawbacks:

Read more...

A little about CouchDB

I've read CouchDB Guide for the second time recently. Very interesting book, it's interesting to understand how CouchDB works internally, one of those rare books that creates a mind shift and expand it to the new territory. It's definitely worth the time spent even if I never will be using CouchDB.

So, what's good about CouchDB:

  • Reliable, can accept thousands of connections and behave elastically and predictably under high load (MongoDB is also pretty fast).
  • Has ability to efficiently (without blocking and copying) make snapshot of database state using MVCC (MongoDB can't to that).
  • High availability - thanks to MVCC it's never blocked (MongoDB block on write, although usually it's not a problem, unless you have write-heavy application).
  • Incremental Map/Reduce (MongoDB can do something similar).
  • Incremental, non-blocking, consistent replication (MongoDB also can do it).
  • Has notifications / change-log (MongoDB doesn't have it).
  • Has versioning, although pretty basic (MongoDB doesn't have it).

Now, about disadvantages:

Read more...

Tovarkovo

Read more...

Sochi

Read more...

Node.js not suitable for general web development

Had a quick look at Node.js usage in production.

In most cases Node.js isn't used as a web framework for the core web product. It's used for additional specialized API or real-time extensions.

From my subjective experience with Node.js web frameworks - it's about the level of Ruby Sinatra. None of node.js web frameworks even closely compared to Ruby on Rails. Nor in terms of functionality nor in development productivity.

So, it seems for me that Node.js isn't suitable for general web development. It's a good fit for API, but not for web site or web applications (online project management and so on.).

And that's sad news, because I believe it has all features needed to build rich and high productive web framework like Ruby on Rails (Fibers can be used to mitigate callbacks).

Still, even if productivity with Node.js is less than with Ruby on Rails it's better than with Java web frameworks.

UPDATE 2014/2/4

As I said before:

I believe Node.js has all features needed to build rich and high productive web framework like Ruby on Rails

So, here it is - http://monojs.org

Read more...

Quality of Product

In order to be able to sell the product it should do something useful and be in a working state. The only practically way to check if product works or not - is to check it according to the specification.

That's why product specification is important - it defines the goal, what the product should be and how to check if it works or not (it also helps to measure progress - how close current state to the goal).

Process of measuring how close the product to the specification and if it's working called testing, there are manual and automatic tests.

There's very huge difference between manual and automatic testing:

  • Manual tests require smaller amount of time - but it requires it every time you apply it.
  • Automatic tests require bigger amount of time - but it requires it only once, all next runs will be virtually free (some time will be spend on support, but it's very small).

That difference dictates when and how apply manual or automatic testing.

Read more...

Open source games

Simple open source games that are nice and playable.

Monk Story

DotA, video, play, sources, JavaScript & CraftyJS.

Monk Story

Eat them all

Tower Defense / RTS, play (it's not available online, you need to download and run it locally), video, sources, JavaScript & CraftyJS.

Read more...

Game Art

Liberated Pixel Cup

Free, video, download, documentation, demo, source of demo.

Lost Garden

Free, tutorial

Read more...

SQL for Data Analyst

These days lots of data available in digital form, ability to analyze and get meaning from that data become more important. Usually such job is called Data Analysis or Data Mining and the person who does that is called Data Analyst.

Actually, I wrote that article for my brother (who's Analyst and ask me about SQL) and decided to publish it because it may be also useful for others.

Main skills of Analyst are Mathematics, Statistics and Domain expertise. But in order to apply those skills he should be able to get the data itself. Widely supported way to get access to data is called SQL, and Analyst can benefit greatly if it knows basics of it.

SQL is a declarative language for querying and transforming data stored in relational database.

  • Declarative - it means that you declare what you want without explicitly telling how to do that. And database figures it out by herself how to fulfill your request in the best way. It's a good thing, because describe what you want usually much simpler than to explain how to do that.
  • Relational database - special type of database (it's the most widely used type of database) that stores data as rows (also called records) in tables.

Read more...

Prototyping 2D Game

Had one free evening and tried to create simple game prototype with plasticine.

Still don't know how to create ground tiles to allow easy combine it and quickly create diferend grounds. Square and random shapes looks bad.

Read more...

Interesting 2D Open Source Games

Flare - RPG, clone of Diablo 1, http://flarerpg.org

Read more...

Functional Mixins in JavaScript

Functional Mixins is a flexible way to share functionality between objects.

From my experience it's more powerful than Class or Prototype based Inheritance, Mixins and even complex machinery like Ruby Object Model. But I believe the biggest advantage of it isn't the power but the simplicity, it's so simple that I wonder why it's so rarely used.

In this example we create Game Unit and add some basic functionality to it (source).

Creating simple Model

Defining Model that have only two methods - set to set attributes, and initialize - a constructor supporting setting attributes.

Read more...